Handling disconnections in a cluster

[arsham]

Hi,

Thanks for providing this awesome library!

We're currently facing an issue with handling disconnections based on the example provided in this repository. The problem is that the connection and the channel are not notifying us about disconnections via channels provided to the NotifyClose methods of channel and connection objects.

Our Setup

We have a RabbitMQ cluster of three nodes running inside Kubernetes. The queue in question, let's call it my_queue, is currently bound to node 2 of the cluster.

Disconnection Behaviour

If the client is connected to node 2 and trying to consume messages, everything works as expected. At this stage, there is a channel bound to the my_queue queue. When node 2 is disconnected or restarted, we receive the notification successfully, and the client attempts to reconnect.

However, depending on the situation, the next time the client might connect to a different node. If that happens, and node 2 goes away, we don't get any notification. At this point, the client has a connection to another node, the channel is not bound to any queue, and we don't consume any messages indefinitely.

Other Observations

We have noticed that at this point we receive a message from the consumer where its ConsumerTag is an empty string. We are planning to use this behaviour to indicate that we are disconnected from the server. We know this is not the correct way to handle disconnections, but it's the only option we have at this stage. So far, we have been killing our pods regularly to keep our services running.

I understand that handling disconnections is not the primary goal of this library, but I would appreciate it if you could comment on this issue.

Best regards,

Arsham

[lukebakken]

Hello, Team RabbitMQ prefers to use discussions until actionable work is discovered.

In this case, please provide the following:

• An export of your k8s configuration
• An export of your RabbitMQ definitions
• A client application that I can clone, compile, and run to see the issue
• Detailed instructions for reproducing the issue

Team RabbitMQ does not have the time or resources to set up an environment, write code, and try to reproduce the issue, plus, time doing that is time not spent diagnosing the issue. Please, make it as easy as possible for us to see your issues and we are happy to assist.

[arsham]

Dear Team RabbitMQ.

Thank you for your prompt response. We understand that you are busy and to aid with debugging this issue, I have prepared a step-by-step setup and how to trigger the issue and an observation regarding the ConsumerTag value.

Installing Kind

On Arch Linux you can install it with any package manager:

paru -S kind

Kind config

There is no specific reason for having three worker nodes in this setup, however we wanted to have kubernetes put the pods in different nodes. Keep this in a file called kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

Create the cluster

kind create cluster --name rabbitmq-cluster --config kind-config.yaml

Helm

Assuming you have helm installed:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Setting up RabbitMQ

We are opting for a three node cluster to resemble our own setup, and also illustrate the reconnection issue in a cluster.
Contents of the rabbitmq.yaml values:

Contents of the rabbitmq.yaml file:

replicaCount: 3

auth:
  username: guest
  password: guest
  erlangCookie: guest

clustering:
  ## Rebalance master for queues in cluster when new replica is created
  rebalance: true

podLabels:
  db.blokur.com/type: rabbitmq
  db.blokur.com/name: rabbitmq
  app: rabbitmq

resources:
  requests:
    cpu: 600m
    memory: 512Mi

persistence:
  size: 5Gi

metrics:
  enabled: false

pdb:
  create: true
  minAvailable: 1

extraConfiguration: |-
  log.default.level = warning

Using helm:

kubectl create ns rabbitmq
helm upgrade -i -n rabbitmq -f rabbitmq.yaml rabbitmq bitnami/rabbitmq

Access

To Access the RabbitMQ AMQP port:

kubectl port-forward --namespace rabbitmq svc/rabbitmq 5672:5672

To Access the RabbitMQ Management interface:

kubectl port-forward --namespace rabbitmq svc/rabbitmq 15672:15672

Example Program

Please find the attached file to this message.
main.go.tar.gz

Check the node the queue is bound to

Browse to the http://localhost:15672/#/queues/%2F/updates page. There is a Details section that shows the value for Node.

Breaking the connection

Assuming the queue is bound to rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local node.

First change the namespace to RabbitMQ:

kubectl ns rabbitmq

We are trying to simulate this scenario:

• The queue is bounded to rabbitmq-0 node
• The client is connected to rabbitmq-1 or rabbitmq-2 nodes
• The rabbitmq-0 node dies.
• The channel or connection doesn't receive the disconnection notification

If the client is initially connected to rabbitmq-0 node, simply killing the rabbitmq-0 node should force the client to connect to another node.
Once we are connected to another node, we kill the pod that the queue is bounded to:

kubectl delete pod rabbitmq-0

At this point and after the pod is started and the cluster is healthy, we get
the following messages:

2024/09/23 15:43:34 Received message: Messages #9 (consumer_tag=ctag-/tmp/go-build368701305/b001/exe/main-1)
2024/09/23 15:43:36 Received message: Messages #10 (consumer_tag=ctag-/tmp/go-build368701305/b001/exe/main-1)
2024/09/23 15:43:38 Received message: Messages #11 (consumer_tag=ctag-/tmp/go-build368701305/b001/exe/main-1)
2024/09/23 15:43:40 Received message:  (consumer_tag=)
2024/09/23 15:43:40 Error acknowledging message: delivery not initialized (consumer_tag=)
2024/09/23 15:43:42 Received message:  (consumer_tag=)
2024/09/23 15:43:42 Error acknowledging message: delivery not initialized (consumer_tag=)

And reconnection is unsuccessful because the channels never received the
notification.

I would like to mention that the re-connection strategy works if the client is connected to the RabbitMQ node that is killed.

Another thing to look out for the port-forward also might be disconnected if the port-forward was on the pod you kill. In that case please port-forward again.

Observation

As mentioned in the first message we have noticed that the ConsumerTag is empty, which we are planning to use as a backup plan for detecting disconnections. However we think this is not the intended use to indicate that the client was disconnected from the server.

Please let me know if you have any questions regarding this setup.

Kind regards.

Arsham