Improve scalability of BroadcastReplicationActions #92902
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BroadcastReplicationAction derivatives (`POST /<indices>/_refresh` and `POST /<indices>/_flush`) are pretty inefficient when targeting high shard counts due to how `TransportBroadcastReplicationAction` works: - It computes the list of all target shards up-front on the calling (transport) thread. - It accumulates responses in a `CopyOnWriteArrayList` which takes quadratic work to populate, even though nothing reads this list until it's fully populated. - It then mostly discards the accumulated responses, keeping only the total number of shards, the number of successful shards, and a list of any failures. - Each failure is wrapped up in a `ReplicationResponse.ShardInfo.Failure` but then unwrapped at the end to be re-wrapped in a `DefaultShardOperationFailedException`. This commit fixes all this: - The computation of the list of shards, and the sending of the per-shard requests, now happens on the relevant threadpool (`REFRESH` or `FLUSH`) rather than a transport thread. - The failures are tracked in a regular `ArrayList`, avoiding the accidentally-quadratic complexity. - Rather than accumulating the full responses for later processing we track the counts and failures directly. - The failures are tracked in their final form, skipping the unwrap-and-rewrap step at the end. Relates elastic#77466 Relates elastic#92729
DaveCTurner
added
:Distributed Indexing/Distributed
|
Pinging @elastic/es-distributed (Team:Distributed) |
elasticsearchmachine
added
the
Team:Distributed (Obsolete)
97 tasks
tlrx
approved these changes
Jan 13, 2023
| addShardResponse(numCopies, 0, createSyntheticFailures(numCopies, e)); | ||
| } | ||
|
|
||
| private List<DefaultShardOperationFailedException> createSyntheticFailures(int numCopies, Exception e) { |
There was a problem hiding this comment.
I'm not sure if it really deserves a dedicated method, we can probably include this in onFailure()
There was a problem hiding this comment.
Yes, this was better at some point in the process but no longer needed indeed.
…13-TransportBroadcastReplicationAction
DaveCTurner
added
the
auto-merge-without-approval
|
@elasticmachine please run elasticsearch-ci/bwc |
DaveCTurner
deleted the
2023-01-13-TransportBroadcastReplicationAction
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BroadcastReplicationAction derivatives (
POST /<indices>/_refreshandPOST /<indices>/_flush) are pretty inefficient when targeting high shard counts due to howTransportBroadcastReplicationActionworks:It computes the list of all target shards up-front on the calling (transport) thread.
It accumulates responses in a
CopyOnWriteArrayListwhich takes quadratic work to populate, even though nothing reads this list until it's fully populated.It then mostly discards the accumulated responses, keeping only the total number of shards, the number of successful shards, and a list of any failures.
Each failure is wrapped up in a
ReplicationResponse.ShardInfo.Failurebut then unwrapped at the end to be re-wrapped in aDefaultShardOperationFailedException.This commit fixes all this:
The computation of the list of shards, and the sending of the per-shard requests, now happens on the relevant threadpool (
REFRESHorFLUSH) rather than a transport thread.The failures are tracked in a regular
ArrayList, avoiding the accidentally-quadratic complexity.Rather than accumulating the full responses for later processing we track the counts and failures directly.
The failures are tracked in their final form, skipping the unwrap-and-rewrap step at the end.
Relates #77466
Relates #92729