[DPE-5125] Remove peer relation lock #392
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR separates the lock-related changes from the DPE-4575.
The main goal is to ensure we are not unnecessarily requesting for the lock for a single unit.
Removing the peer lock
There are two situations that could justify the use of peer lock:
Use Case 1: moving from 1 unit up while that single unit is powered off
Use Case 2: a cluster with X elegible managers, where a subset is powered down already
In this case, we may have two situations: (i) we have a minimal quorum of nodes powered up - hence the other nodes can pick the .charm-node-lock one by one and no need for the peer relation here; or (ii) we powered down nodes to less than the minimal quorum: in this case, the cluster is essentially stopped and unresponsive.
For case (ii), recovering units will only matter when the minimal quorum of voting seats have been achieved once again. Before that, the cluster will not be able to define shard allocation. On the other hand, if we have to bring several nodes at once up, then we may end up with a lot of recovery traffic happening at once as well. OpenSearch does provide a protection for these flooding, using
cluster.routing.allocation.node_concurrent_{|outgoing|incoming}_recoveries or node_initial_primaries_recoveries
settings. That sets a limit of how many concurrent shard relocation tasks may happen at once.Therefore, a case of (ii) will eventually become a case (i), and can then start to use the charm lock in OpenSearch for restarting. This PR will not address how to decide when we are in case (ii) and what do. This is going to be subject of further investigation.