DLX data safety guarantees with node restarts with classic queues on RabbitMQ 3.9

[usernameisnull]

Describe the bug

In the process of a large number of messages being moved to the dead-letter queue due to TTL expiration, the RabbitMQ cluster crashed, resulting in the loss of some messages.

Reproduction steps

1. 2 queue vs 1 dead letter queue
2. each queue has 100000 messages
3. each message is durable and set random ttl, and 2 queues are durable
   
4. When you see the message that it has been transferred to DLQ due to TTL, restart the entire cluster
   
5. When the cluster returns to normal, you can see that some messages are lost.
   
6. No consumers are involved in the whole process

Expected behavior

Hope the message wasn't lost

Additional context

rabbitmq version: 3.9.25
environ: kubernets
3 replicas

[mkuratczyk]

RabbitMQ 3.9 has been out of support for many years. Try to reproduce this with RabbitMQ 4.0 (and therefore classic queues v2, since only v2 is available in 4.0).

There's a pretty good chance things will work better just by upgrading. However, there's indeed no credit flow mechanism when dead lettering between classic queues. You can switch to quorum queues (even in a single-node environment potentially), since quorum queues do have more reliable dead-lettering in newer versions.
https://www.rabbitmq.com/blog/2022/03/29/at-least-once-dead-lettering

We are not going to investigate issues for such old versions.

[michaelklishin]

@usernameisnull DLX is not magic, it is a special kind of publisher. Publishing can and will fail when nodes are restarted. This is why Publisher Confirms exist, so that clients could track such cases and retry (or do something else).

However, DLX is not a regular client and it cannot execute arbitrary code. So there are only two options to choose from in modern versions, and only for quorum queues who have sufficient data safety guarantees and focus to implement a reasonably safe retry for DLX. Specifically the source queue must be a quorum one, although using QQs with non-mirrored classic queues for dead lettered messages will negate most data safety characteristics of QQs.

As @mkuratczyk has already explained, the "at least once" DLX strategy was introduced in 3.10. All versions in 3.x are out of community support and will receive no future open source releases.

It's time to upgrade, possibly using a Blue/Green deployment strategy given that you are five release series behind.

[michaelklishin]

In addition, classic queues do not offer much in terms of data safety. Classic queue mirroring was removed completely from RabbitMQ, and single replicate queues will have the obvious limitation of losing availability for a period of time during a cluster restart.

Use quorum queues with 3+ replicas as explained in the docs, and a DLX strategy that will re-publish if that's what you want. Keep a majority of nodes online, as QQs, streams, and any other Raft-based feature of modern RabbitMQ requires.

[usernameisnull]

I upgraded to 3.10 and haven't experienced data loss in this situation. However, sometimes when an OOM (Out of Memory) occurs, it seems that the dead letter queue receives more data than expected. Could this be because, with the quorum ensuring at least once delivery in 3.10, there might be multiple deliveries to the dead letter queue?

[michaelklishin]

@usernameisnull 3.10 has been entirely out of support — even for paying users — for over a year.

Please upgrade to 4.0.x where classic queue v2 storage is not only an option but is the only option. CQv2s will have lower memory footprint. Or simply use a stream for dead-lettering, that should work just fine, and you don't even need to immediately introduce a stream protocol client library to consume those messages.