Cinder: add volume metadata/properties as labels (i.e. cluster)

[MPV]

We have multiple clusters using the same OpenStack tenant, and as such they share volumes. Trying to use alerts such as CinderDiskWithoutPV then gives false-negatives, as it thinks reports disks created by PV/PVCs from another cluster as matching its alert query:
kos_cinder_volume_status{name=~"(pvc-.+|kubernetes-dynamic-pvc.+)",pv_name=""} == 1

Would adding volume properties help? Such as CSI Cinder's CLUSTER_NAME setting, added support for in:

• [OCCM] Add --cluster-name=$(CLUSTER_NAME) support kubernetes/cloud-provider-openstack#1809
• [cinder-csi-plugin] Add clusterID Helm value kubernetes/cloud-provider-openstack#1741

Here's an example of a volume with a "cluster" attribute set (by Cinder CSI):

$ openstack volume show -f json bc7c0b38-72b2-481e-bb9c-a0a41a38ad65  | jq '.properties'
{
  "cinder.csi.openstack.org/cluster": "kubernetes"
}

And here's another example with more properties being set:

$ openstack volume show -f json pvc-3e542918-a4cb-4268-bdc1-375fb39d6c71 
{
  "attachments": [
    {
      "id": "11534b61-352e-41fe-a53c-49cd5c634375",
      "attachment_id": "3f665eee-94bb-4596-a141-2ef26f68abb3",
      "volume_id": "11534b61-352e-41fe-a53c-49cd5c634375",
      "server_id": "da4e0e7a-fa62-4b8b-b5b8-eaf4571cdef3",
      "host_name": "h56.az1.obfuscated",
      "device": "/dev/vdd",
      "attached_at": "2022-05-02T14:26:38.000000"
    }
  ],
  "availability_zone": "kvm-linux",
  "bootable": "false",
  "consistencygroup_id": null,
  "created_at": "2022-04-29T15:18:38.000000",
  "description": "Created by OpenStack Cinder CSI driver",
  "encrypted": false,
  "id": "11534b61-352e-41fe-a53c-49cd5c634375",
  "multiattach": false,
  "name": "pvc-3e542918-a4cb-4268-bdc1-375fb39d6c71",
  "os-vol-tenant-attr:tenant_id": "obfuscated",
  "properties": {
    "cinder.csi.openstack.org/cluster": "kubernetes",
    "csi.storage.k8s.io/pv/name": "pvc-3e542918-a4cb-4268-bdc1-375fb39d6c71",
    "csi.storage.k8s.io/pvc/name": "prometheus-monitoring-prometheus-oper-prometheus-db-prometheus-monitoring-prometheus-oper-prometheus-0",
    "csi.storage.k8s.io/pvc/namespace": "obfuscated"
  },
  "replication_status": "disabled",
  "size": 20,
  "snapshot_id": null,
  "source_volid": null,
  "status": "in-use",
  "type": "__DEFAULT__",
  "updated_at": "2022-05-02T14:26:56.000000",
  "user_id": "obfuscated"
}

I suspect this would be the place to add it?

kosmoo/pkg/metrics/cinder.go

Line 166 in 72c46a7

labels := []string{v.ID, v.Description, v.Name, v.Status, v.AvailabilityZone, v.VolumeType}

Would it be the Metadata property on the GopherCloud Volume result to add, perhaps?
https://github.com/gophercloud/gophercloud/blob/19ba3dc8b6df9f0c12aec667e77b4cee634d4245/openstack/blockstorage/v2/volumes/results.go#L64-L65

[MPV]

Related:

• Per-cluster Kosmoo vs multiple-clusters-per-OpenStack #106