Solidfire - Add a new storage class (QOS profile) to running backend

[ghost]

i tried to modify:
{
"version": 1,
"storageDriverName": "solidfire-san",
"Endpoint": "https://user_trid:uuuuuu@address.com/json-rpc/7.0",
"SVIP": "10.10.10.10:3260",
"TenantName": "trident",
"InitiatorIFace": "default",
"Types": [{"Type": "Bronze", "Qos": {"minIOPS": 1000, "maxIOPS": 2000, "burstIOPS": 4000}},
{"Type": "Silver", "Qos": {"minIOPS": 4000, "maxIOPS": 6000, "burstIOPS": 8000}},
{"Type": "Gold", "Qos": {"minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000}}]
}

into:

{
"version": 1,
"storageDriverName": "solidfire-san",
"Endpoint": "https://user_trid:uuuuuu@address.com/json-rpc/7.0",
"SVIP": "10.10.10.10:3260",
"TenantName": "trident",
"InitiatorIFace": "default",
"Types": [{"Type": "Bronze", "Qos": {"minIOPS": 1000, "maxIOPS": 2000, "burstIOPS": 4000}},
{"Type": "Silver", "Qos": {"minIOPS": 4000, "maxIOPS": 6000, "burstIOPS": 8000}},
{"Type": "Gold", "Qos": {"minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000}},
{"Type": "Platinum", "Qos": {"minIOPS": 8000, "maxIOPS": 16000, "burstIOPS": 20000}}]
}

in order to have Platinum storage class available with the following command:
oc exec -i trident-12345678-xkm6x -- post.sh backend < trident-bk-iscsi-prod.txt
But it does not work and trident logs:
time="2017-06-01T12:50:08Z" level=info msg="Added a new backend." backend= handler=AddBackend
2017/06/01 12:50:08 http: panic serving [::1]:43536: runtime error: invalid memory address or nil pointer dereference
goroutine 776970 [running]:
.....

Adding a storage class works however for specific solidfire case it seems to be necessary tomodify the back-end too.

[kangarlou]

Trident allows updating a backend's storage pools subject to some conditions. More specifically, the storage pools in the updated backend should be a superset of storage pools in the old backend so that all storage classes and volumes mapped to the old backend's storage pools are still supported by the updated backend's storage pools. Your example satisfies this requirement. I tried your updated definition on a SolidFire backend and it worked fine for me.
Could you please share all log messages that correspond to the panic? Also, please share the output of the following command:
oc exec -i trident-12345678-xkm6x -- json-get.sh backend nameOfYourBackend

[ghost]

thanks for your reply, This is the output
Defaulting container name to trident-main. Use 'oc describe pod/trident-4215269415-xkm6x' to see all of the containers in this pod. { "backend": { "name": "solidfire_10.10.10.10", "config": { "version": 1, "storageDriverName": "solidfire-san", "storagePrefix": {}, "TenantName": "trident", "EndPoint": "https://sf101.google.com/json-rpc/7.0", "SVIP": "10.10.10.10:3260", "InitiatorIFace": "default", "Types": [ { "Type": "Bronze", "QOS": { "minIOPS": 1000, "maxIOPS": 2000, "burstIOPS": 4000 } }, { "Type": "Silver", "QOS": { "minIOPS": 4000, "maxIOPS": 6000, "burstIOPS": 8000 } }, { "Type": "Gold", "QOS": { "minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000 } } ] }, "storage": { "Bronze": { "name": "Bronze", "storageClasses": [ "netapp-trident-ams4prod-bronze" ], "storageAttributes": { "IOPS": { "min": 1000, "max": 2000 }, "backendType": { "offer": [ "solidfire-san" ] }, "media": { "offer": [ "ssd" ] }, "provisioningType": { "offer": [ "thin" ] }, "snapshots": { "offer": true } }, "volumes": [ "depstr-dobbitest-pvc-5b4eb", "gfranklin-gfranklin-iscsi-2-8b2f6", "openshift-infra-metrics-cassandra--1-43aa0", "openshift-infra-metrics-cassandra--2-302f6", "openshift-infra-metrics-cassandra--3-3bbba", "test02-postgresql-65eac", "test20000-jenkins-4ba3a" ] }, "Gold": { "name": "Gold", "storageClasses": [ "netapp-trident-ams4prod-gold" ], "storageAttributes": { "IOPS": { "min": 6000, "max": 8000 }, "backendType": { "offer": [ "solidfire-san" ] }, "media": { "offer": [ "ssd" ] }, "provisioningType": { "offer": [ "thin" ] }, "snapshots": { "offer": true } }, "volumes": [ "depstr-rares-highbw-05-11-02-job-1-pvc-5429b", "prometheus-kubernetes-prometheus-storage-iscsi-0473d" ] }, "Silver": { "name": "Silver", "storageClasses": [ "netapp-trident-ams4prod-silver" ], "storageAttributes": { "IOPS": { "min": 4000, "max": 6000 }, "backendType": { "offer": [ "solidfire-san" ] }, "media": { "offer": [ "ssd" ] }, "provisioningType": { "offer": [ "thin" ] }, "snapshots": { "offer": true } }, "volumes": [] } }, "online": true, "volumes": [ "openshift-infra-metrics-cassandra--3-3bbba", "openshift-infra-metrics-cassandra--1-43aa0", "test02-postgresql-65eac", "test20000-jenkins-4ba3a", "gfranklin-gfranklin-iscsi-2-8b2f6", "depstr-dobbitest-pvc-5b4eb", "openshift-infra-metrics-cassandra--2-302f6", "depstr-rares-highbw-05-11-02-job-1-pvc-5429b", "prometheus-kubernetes-prometheus-storage-iscsi-0473d" ] } }
And these are the logs:
time="2017-06-01T12:50:08Z" level=info msg="Added a new backend." backend= handler=AddBackend 2017/06/01 12:50:08 http: panic serving [::1]:43536: runtime error: invalid memory address or nil pointer dereference goroutine 776970 [running]: net/http.(*conn).serve.func1(0xc820574d00) /usr/local/go/src/net/http/server.go:1389 +0xc1 panic(0x1a680c0, 0xc820016080) /usr/local/go/src/runtime/panic.go:443 +0x4e9 github.com/netapp/trident/storage_class.(*StorageClass).Matches(0x0, 0xc820816380, 0xc8202c3b18) /go/src/github.com/netapp/trident/storage_class/storage_class.go:41 +0x35 github.com/netapp/trident/core.(*tridentOrchestrator).validateBackendUpdate(0xc8201cfac0, 0xc8202b7320, 0xc82145c5d0, 0x0, 0x0) /go/src/github.com/netapp/trident/core/orchestrator_core.go:357 +0xdef github.com/netapp/trident/core.(*tridentOrchestrator).AddStorageBackend(0xc8201cfac0, 0xc820324280, 0x270, 0x0, 0x0, 0x0) /go/src/github.com/netapp/trident/core/orchestrator_core.go:393 +0x1c8 github.com/netapp/trident/frontend/rest.AddBackend.func1(0xc820e42600, 0x270, 0x600) /go/src/github.com/netapp/trident/frontend/rest/handlers.go:204 +0x7b github.com/netapp/trident/frontend/rest.AddGeneric(0x7f15243e9428, 0xc821632000, 0xc8211821c0, 0x7f15243ec8a8, 0xc82154b4c0, 0xc820da5908) /go/src/github.com/netapp/trident/frontend/rest/handlers.go:112 +0x2f5 github.com/netapp/trident/frontend/rest.AddBackend(0x7f15243e9428, 0xc821632000, 0xc8211821c0) /go/src/github.com/netapp/trident/frontend/rest/handlers.go:210 +0xad net/http.HandlerFunc.ServeHTTP(0x20fd2d0, 0x7f15243e9428, 0xc821632000, 0xc8211821c0) /usr/local/go/src/net/http/server.go:1618 +0x3a github.com/netapp/trident/frontend/rest.Logger.func1(0x7f15243e9428, 0xc821632000, 0xc8211821c0) /go/src/github.com/netapp/trident/frontend/rest/logger.go:15 +0xad net/http.HandlerFunc.ServeHTTP(0xc8203c10b0, 0x7f15243e9428, 0xc821632000, 0xc8211821c0) /usr/local/go/src/net/http/server.go:1618 +0x3a github.com/netapp/trident/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc82044a4b0, 0x7f15243e9428, 0xc821632000, 0xc8211821c0) /go/src/github.com/netapp/trident/vendor/github.com/gorilla/mux/mux.go:98 +0x29e net/http.serverHandler.ServeHTTP(0xc820574500, 0x7f15243e9428, 0xc821632000, 0xc8211821c0) /usr/local/go/src/net/http/server.go:2081 +0x19e net/http.(*conn).serve(0xc820574d00) /usr/local/go/src/net/http/server.go:1472 +0xf2e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2137 +0x44e

[kangarlou]

Thanks for the additional information. I suspect that you have at least a volume that points to a deleted storage class. Please run the following command for the 9 volumes in your environment:
oc exec -i trident-12345678-xkm6x -- json-get.sh volume nameOfYourVolume
For each volume, please check the "storageClass" field. It seems for at least one of your volumes, the storage class is not one of "netapp-trident-ams4prod-gold", "netapp-trident-ams4prod-silver", or "netapp-trident-ams4prod-bronze". Could you please verify that and report the results?

[ghost]

thanks for the reply, so we have the following storage classes:
"netapp-trident-ams4prod-gold",
"netapp-trident-ams4prod-silver",
"netapp-nfs",
"netapp-trident-ams4prod-bronze"

but we have a orphan class "netapp-trident" where most of the volumes are from. This class refers to old trident deployment, the backend has been removed but not yet the volumes. I guess or we need to wait to decomm those volumes or there is another way?

[kangarlou]

A deleted storage class explains why you're experiencing this problem. You can't really "delete" a backend while it has volumes associated with it, but you can certainly update the backend, which I think is what you have done. Having said that, the problem is not so much the deleted backend but rather the deleted storage class. I'll work on a patch that wouldn't require you to decommission the volumes before you can update the backend definition.

[ghost]

ok thanks

[kangarlou]

@labazza v17.07.0-beta.1 includes the fix.

[ghost]

thanks