rabbit_peer_discovery: Retry RPC calls #12801
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Why] This was the first solution put in place to prevent that the temporary hidden node connects to the node that started it to write any printed messages. Because of this, the nodes that the temporary hidden node queried found out about the parent node and they opened an Erlang distribution connection to it. This polluted the known nodes list. However later, the temporary hidden node was started with the `standard_io` connection option. This prevented the temporary hidden node from knowing about the node that started it, solving the problem in a cleaner way. [How] This commit garbage-collects that piece of code that is now useless. It makes the query code way simpler to understand.
[Why] This impacts what is reported by the catch because it caught exceptions emitted by code supposedly called later. An example is the assert in `query_node_props2/3` last clause.
[Why] In CI, we observe some timeouts in the Erlang distribution connections between the temporary hidden node and the nodes it queries. This affects peer discovery obviously. [How] We introduce some query retries to reduce the risk of an incomplete query. While here, we move the sorting of queried nodes from the `query_node_props2/3` last clause (executed in the temporary hidden node) to the function setting the temporary hidden node and asking for these queries. This way the debug messages from that sorting are logged by RabbitMQ out of the box.
dumbbell
force-pushed
the
make-peer-disc-nodes-querying-more-resiliant
branch
from
8a4a684 to
f6314d0
Compare
michaelklishin
added a commit
that referenced
this pull request
Nov 25, 2024
rabbit_peer_discovery: Retry RPC calls (backport #12801)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why
In CI, we observe some timeouts in the Erlang distribution connections between the temporary hidden node and the nodes it queries. This affects peer discovery obviously.
How
We introduce some query retries to reduce the risk of an incomplete query.
While here, we move the sorting of queried nodes from the
query_node_props2/3last clause (executed in the temporary hidden node) to the function setting the temporary hidden node and asking for these queries. This way the debug messages from that sorting are logged by RabbitMQ out of the box.The branch contains two additional related commits:
standard_ioconnection option to start the temporary hidden node.query_node_props2/3.