Multiple Rabbitmq cluster in k8s

[janiu-001]

Hi Expert，

I plan to deploy multiple Rabbitmq cluster in different k8s namespace, But I also expect to be able to access the mulitple cluster through an address（one URI). I do this because we can horizontal scale based on business.

Very appreciated for your response and suggestions!

[mkuratczyk]

Cluster Operator fully supports deploying multiple clusters to the same and/or different namespaces. However, it won't help you deploy a single service to expose them all (it only provision things per cluster). Can you elaborate on what you are trying to achieve?

[janiu-001]

I plan to use Rabbitmq in our k8s environment to make an mqtt broker for the Internet of Things. Vertical scale is our first choice, but as the number of devices increases, horizontal scale is required. However, if it is a horizontal scale, it is ideal for the device to connect to the mqtt broker with the same domain name and load a balance to different mqtt brokers.

[michaelklishin]

Scaling w.r.t. the number of connections is relatively easy as as long as the connections are roughly evenly distributed between nodes. RabbitMQ 3.12 will handle 1M+ MQTT connections per node with reasonable hardware, and having 5 or 7 or 9 nodes in a cluster is perfectly fine (as long as you don't replicate all queues to all nodes, which is overly excessive).

In most cases distributed connections across clusters would be much harder: if publishers connect to cluster A and consumers to cluster B, you have to figure out how to transfer relevant messages from A to B. Queue federation in theory can do this but the plugin does not have any data about client connections and data locality.

Therefore, unless your workload can naturally be sharded into N clusters (say, one country or region = one cluster), growing the number of cluster nodes would be much easier and can take you quite far.

VMware is investigating a specialized proxy for messaging protocols that would be able to handle even more connections per node but it won't ship until at least late 2023.

[janiu-001]

@michaelklishin Thanks very much for your response! To make sure we are on the same page, i draw two diagram:

1. My origin solution:

But it can not deploy a single service to expose them all , and it is hard to figure out how to transfer relevant messages from NameSpace1 Cluster to NameSpace2 Cluster

2. Your suggested solution

The node you mentioned is the pod in k8s world, right? your suggestions is one cluster with multiple pods and adding one pod will add one million connections, is my understanding right?

And what's your suggested config(cpu and memory) for each pod?

[mkuratczyk]

It sounds like you are planning to have many millions of connection, is that right?

1. We do not guarantee you can just add a node to increase the capacity of your cluster by 1 million additional connections. There are many factors that need to be considered.
2. You can find our test configuration and some results here: https://github.com/rabbitmq/mqtt-testing and in the blog post mentioned
3. You should definitely perform your own tests. You will have different hardware, you will have specific QoS levels, you will have a certain publish rate that's different and so on.

[janiu-001]

ok, understand the capacity based the hardware. Double confirm- is my understating right?

The node you mentioned is the pod in k8s world, it can increase the capacity bu adding pods number, but the quota is based on hardwares and other factors, like QOS...etc

[mkuratczyk]

Can you please let us know what you are planning? How many connections, what QoS? What message rate? This will both help us answer your questions and just develop RabbitMQ in general - it's hard without understanding how RabbitMQ is used.

Of course performance depends on the hardware and yes, on Kubernetes RabbitMQ nodes run in pods. However:

1. you can't assume perfect linear scalability of RabbitMQ and everything around it. Just because we you can handle 1 million connections on 1 node, doesn't mean you can easily handle 10 million with 10 nodes (that would be true in our original idea where the nodes are completely independent and there is no communication between the nodes (because they are independent 1-node clusters) but that design has other challenges as you already noticed.

2. there are many factors that impact the performance. For example a QoS0 subscription, starting with RabbitMQ 3.12, is very cheap - it's just a connection. But a QoS1 subscription requires a queue to be declared. Suddenly with just QoS change we not only have 1 million connections, but also 1 millions queues. I can already tell you that I'd be surprised if you could run 1 million queues on a single node so the original assumption of a node being able to handle a million connections is no longer valid with QoS1, because a node can't handle 1 million queues.

In short, if you are designing a large deployment, you need to experiment and learn where the bottlenecks are in your specific case.

[janiu-001]

My planning:

1. QoS: 0
2. to support 1 million~5 million devices which mean 1 million～5 million connections
3. message rate: 12 messages per second

[mkuratczyk]

Sounds super interesting but I'm fairly sure you'll run into other issues/challenges working on this. We'd love to learn from your experience and will be happy to assist to the extent possible, but please keep in mind that you are doing something that was basically impossible until now, so we don't have experience doing this. When you report issues, please provide as much details as possible, make sure you have monitoring set up, go through the perf setup (https://blog.rabbitmq.com/posts/2022/05/flame-graphs/) so that we can investigate issues using perf in the future and so on.

If you can provide further details, that would be very helpful. 12 msg/s, do you mean per connection, so 12-60 million messages/s total? What's the message size? Is it exclusively a fan-in scenario (devices send their updates, all of them consumed by one central application) or do you also have fan-out (central app sending messages to all devices) or anything else?

[mkuratczyk]

oh, I'd also recommend installing https://parca.dev/. Another tool to help you (and us) find performance problems when you have it deployed.

[janiu-001]

We have not started deploying yet, as we are studying the feasibility of the solution as Google's IoT has expired and we need to use Rabbitmq as our alternative solution!

That's why I inquired about the feasibility of horizontal and vertical scaling. If a cluster can meet the best requirements for millions of devices, all devices can use one mqtt address

We also consider to scable horizontally by grouping devices, and the worst-case scenario is for each device in a group to have an address of rabbitmq

[michaelklishin]

Then consider using 3.12.0 previews from the start. Note that the test uses 1M connections that do not consume, having that many queues and consumers is a completely different design problem. But publishers may outnumber consumers by orders of magnitude in device fleet deployments.

I'm not sure what you mean by "grouping devices" but the way to sustain a large number of connections is always to add more cluster nodes, and spread those connections reasonably evenly between them. The same goes for queue or stream replicas.

Streams specifically can be used by N consumers non-destructively, so usually there are tens, hundreds of thousands of those, not millions. Quorum queues (and classic queues alike) do not allow for non-destructive consumption, so it is much more likely to see a lot more of them than streams.

[janiu-001]

thanks, we will deploy the 3.12.0 and run the performance testing

Another option is HA + Rabbitmq(https://github.com/haproxy/haproxy/tree/master/doc) if we have more than 1M connection, would you help share your suggestion?

[HowardTwine]

Hi @janiu-001
I work on the product team for RabbitMQ (so not very technical) and wondered if you had done any performance testing with 3.12.0? Happy to set up a call to learn more about this feasibility study.

[janiu-001]

@HowardTwine Thanks very much! we just start the testing, will update you when we finish the testing

[janiu-001]

@michaelklishin I tried the retained feature today, looks like it is not very stable, the highlighted sub don't get the message. Would you help to take a look? our testing based on rabbitmq:3.12.0-beta.6-management with 3 pods

[michaelklishin]

We can take a look if you put together a repository that can be used to reproduce.

We do not guess in this community. Please use code (repositories, GitHub gists, code blocks in comments) instead of screenshots, or the chances that someone will re-type your code from a screenshot and investigate approach zero.

This is open source software you very likely get entirely for free. So please help others help you.

[janiu-001]

@michaelklishin ok, thanks for your suggestion ! a quick question, will the retained messages sync in different pod/node? I saw your response on the same question:(https://groups.google.com/g/rabbitmq-users/c/CEyb6xfKbHA), do we support replicate the retain message?

[michaelklishin]

Retained message replication was not implemented for 3.12.0 and right now is not planned for 3.13.0 (where the focus is on MQTTv5 support).

[janiu-001]

@michaelklishin is there any alternative solution for retained message? we tired the bitnami/rabbitmq, it can work well, is there any difference?

[michaelklishin]

How can a different image help with a (fairly obscure) protocol feature that RabbitMQ currently does not implement?

I always recommend the community Docker image because in the past, some decisions in the Bitnami one were very questionable. That has changed somewhat but I'd always start with docker-library/rabbitmq unless you need Bitnami's SBOM/secure toolchain guarantees.

[janiu-001]

@mkuratczyk

“You can find our test configuration and some results here: https://github.com/rabbitmq/mqtt-testing and in the blog post mentioned”

we will start our load testing, in our previous discussion, you provided a link about the configuration and some results, but i cannot open it, would you help providing a right link?

[mkuratczyk]

I didn't realize that repo was private. You can find a subset of that in https://github.com/ansd/rabbitmq-mqtt/tree/blog-post-native-mqtt

[janiu-001]

Yes, i found that one