Broker Pod stuck in Init state - PVC Problem

[marcelloraffaele]

Hi Team,
I have a three pod broker, one of my pod is stuck in Init Status:

NAME                                            READY   STATUS              RESTARTS   AGE
broker-server-0                                 0/1     Init:0/1            0          35m
broker-server-1                                 1/1     Running             0          3h40m
broker-server-2                                 1/1     Running             0          7h17m

The problem seems to be a problem with PVC/PV.

Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Warning  FailedScheduling    35m                 default-scheduler        0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled           35m                 default-scheduler        Successfully assigned 9204-q-broker/broker-server-0 to moc-l4791op0109
  Warning  FailedScheduling    35m                 default-scheduler        0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedMount         33m                 kubelet                  Unable to attach or mount volumes: unmounted volumes=[persistence], unattached volumes=[cert-noprod rabbitmq-erlang-cookie persistence pod-info rabbitmq-plugins rabbitmq-confd test-ca plugins-conf kube-api-access-xr42c rabbitmq-tls erlang-cookie-secret]: timed out waiting for the condition
  Warning  FailedMount         102s (x6 over 13m)  kubelet                  (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[persistence], unattached volumes=[pod-info persistence cert-noprod test-ca plugins-conf erlang-cookie-secret rabbitmq-confd kube-api-access-xr42c rabbitmq-tls rabbitmq-erlang-cookie rabbitmq-plugins]: timed out waiting for the condition
  Warning  FailedAttachVolume  41s (x24 over 35m)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-9e6af637-839c-47d2-b4af-e180f3177ad9" : rpc error: code = Internal desc = No device path information was returned. Attach failed

Is there a way to force the restart of this pod?
Can you help me?

Thank you,
RM

[mkuratczyk]

This sounds like a problem with your Kubernetes / storage you use in Kubernetes. The other two pods started fine so the definition of the statefulset/pod is correct, it's just that your Kubernetes cluster seems unable to provision another volume.

If you want to fore a restart/retry, just delete the pod kubectl delete pod broker-server-0.

[marcelloraffaele]

Thank you,
I already tried to:

• kill the prod
• delete pvc and pv

But nothing changed. I think this problem could be a bit more difficult how can I solve this strange problem?

[mkuratczyk]

I'm afraid this is not a question for us. This is a Kubernetes problem, we (RabbitMQ) just request a volume to be provisioned and mounted. A quick google search links to repositories for Kubernetes storage drivers (CSIs) which would confirm that this is a lower-level problem.