Assigner waits to get existing assignments from indexers #1116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If an indexer is offline when the assigner service starts, the assigner cannot read the existing assignments or preferences from that indexer. The assigner will not know what publishers are assigned to that indexer, resulting in assigning those publishers to other indexers as announcements from them arrive. The assigner service needs to retry getting indexer assignments before assigning a publisher to an indexer, since that publisher may already be assigned to an offline indexer. This prevents over-assigning publishers to the indexer pool. If the assigner starts without getting assignment information for all indexers in the pool, then it will recheck each time an announce message is received from an unassigned peer. If there are still indexers offline, the assigner assumes that the peer may be assigned to an offline indexer and reduces the required number of assignments by the number of indexers offline.
Codecov ReportBase: 54.67% // Head: 54.95% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #1116 +/- ##
==========================================
+ Coverage 54.67% 54.95% +0.28%
==========================================
Files 134 134
Lines 12877 12928 +51
==========================================
+ Hits 7040 7105 +65
+ Misses 5090 5080 -10
+ Partials 747 743 -4
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
masih
approved these changes
Dec 10, 2022
|
Of course. I will deploy ther first. We should not see anything as this is really a measure to prevent problems if the assigner starts when indexers are offline. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
If an indexer is offline when the assigner service starts, the assigner cannot read the existing assignments or preferences from that indexer. The assigner will not know what publishers are assigned to that indexer, resulting in assigning those publishers to other indexers as announcements from them arrive.
The assigner service needs to retry getting indexer assignments before assigning a publisher to an indexer, since that publisher may already be assigned to an offline indexer. This prevents over-assigning publishers to the indexer pool.
Fixes #1118
Proposed Changes
If the assigner starts without getting assignment information for all indexers in the pool, then it will recheck each time an announce message is received from an unassigned peer. If there are still indexers offline, the assigner assumes that the peer may be assigned to an offline indexer and reduces the required number of assignments by the number of indexers offline.
This may mean that an unassigned peer will not get assigned until indexer(s) come back online. It is necessary to prevent over-assigning publishers to the indexer pool.
Tests
Tests include starting with an indexer offline and testing that:
Revert Strategy
Reverting will return to the previous situation and may require resetting indexer assignments.