check_if_node_is_quorum_critical with a noproc follower

[gomoripeti]

Given a 3-node cluster with a 3-member quorum queue. One of the followers is dead (eg. crashed and reach max intensity). In this case check_if_node_is_quorum_critical does not report any queue with minimum quorum for any of the 3 nodes.
However shutting down the node hosting the leader (or delete_member for the leader node) will leave the queue with 1 running follower and 1 dead follower. It wont have quorum and wont be operational (cluster_change_not_permitted).

% rabbitmq-queues quorum_status qq1 --node rabbit-2
Status of quorum queue qq1 on node rabbit-2@localhost ...
┌────────────────────┬────────────┬────────────┬───────────────┬──────────────┬──────────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name          │ Raft State │ Membership │ LastLog Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├────────────────────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit-1@localhost │ noproc     │            │               │              │              │              │                │      │                 │
├────────────────────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit-2@localhost │ follower   │ voter      │ 22            │ 22           │ 22           │ 22           │ 22             │ 5    │ 3               │
├─────────────-──────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit-3@localhost │ leader     │ voter      │ 22            │ 22           │ 22           │ 22           │ 14             │ 5    │ 3               │
└────────────────────┴────────────┴────────────┴───────────────┴──────────────┴──────────────┴──────────────┴────────────────┴──────┴─────────────────┘

% rabbitmq-queues check_if_node_is_quorum_critical --node rabbit-2
Checking if node rabbit-2@localhost is critical for quorum of any queues/streams ...
Node rabbit-2@localhost reported no queues/streams with minimum quorum

% rabbitmq-queues check_if_node_is_quorum_critical --node rabbit-3
Checking if node rabbit-3@localhost is critical for quorum of any queues/streams ...
Node rabbit-3@localhost reported no queues/streams with minimum quorum

In case the whole node 01 is not available, then the queue qq1 is reported.
It would be helpful if also in the dead follower process case the command would report the same.

I wonder if this is intentional? Or is this a necessary optimisation as it would be very expensive to check each queue member process on each node in case of a lot of queues. (I understand the reason is because the check only looks at the ra_state table and does not contact any process to fetch raft state.) Would #9518 or #10394 improve on this situation?

My question is similar to #9518 but the opposite. There the "wrong" member should not be reported as critical while in my case the "working" member should be.

My use case would be to do a rolling restart and check before each node shutdown if it is ok to continue without loosing availability of any queue. (Another use case is to do a rolling replace node which is roughly: shrink QQs, delete rabbitmq cluster node, add new cluster node, grow QQs)

[kjnilsson]

the Ra process should delete its record from the ra_state table on terminate (since 2018) so I'm not sure why the check would not pick up the terminated process. Can you check if in your case the queue has a record in the ra_state table on the node with the down member?

[kjnilsson]

did you use exit(Pid, kill) to simulate the crash?

[gomoripeti]

first of all I want to apologise that I did not put a RabbitMQ version in the original description. That would be helpful for the current myself as well.

to reproduce I definitely used exit(Pid, kill) but this was seen in production where the queue process died in another way.

Digging up some historical notes I think in some cases on 3.11 and 3.12 we did a "rolling replace node" (that I mentioned above: shrink QQs, delete rabbitmq cluster node, add new cluster node, grow QQs) and after replica had been added on the first node, a QQ crashed like this #12635. Then running check_if_node_is_quorum_critical on the second node said everything was OK. So there were definitely cases when the ra_state table was not updated. But a lot of things have changed since (we don't do shrinking and growing that much any more, non-voter startup since 3.13 could also have helped, and many things changed in quorum queues and ra since)

We have definitely also seen cases when a queue process "disappeared" (ie crashed) but there was no crash logged. That made it harder to know how it exactly crashed. But this was not reproducible on 4.0

To sum up: unfortunately I don't have any more helpful information.
If we see this again I will check the ra_state table.

[michaelklishin]

This should be addressed by #12727 (contributed by @Ayanda-D) and available for testing in alpha 4.0.4-alpha.0caa0664.