Fix hang in bulk helper semaphore when server responses are slower than flushInterval #2027
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before it was waiting until after semaphore resolved, then sending with a reference to bulkBody. If flushInterval is reached after `await semaphore()` but before `send(bulkBody)`, onFlushTimeout is "stealing" bulkBody so that there is nothing left in bulkBody for the flushBytes block to send, causing an indefinite hang for a promise that does not resolve.
|
I've published this to npm as |
pquentin
previously approved these changes
Oct 4, 2023
|
@pquentin Thanks. 🖤 Mostly tagged dynamic clients team members for visibility because this bug came up in conversation in our Monday team meeting. It still bends my mind a bit, too, and I can't find a clear explanation for why an empty |
|
This pull request is stale because it has been open 90 days with no activity. Remove the |
alkozayMoein
approved these changes
Jan 16, 2024
pquentin
approved these changes
Feb 6, 2024
github-actions bot
pushed a commit
that referenced
this pull request
Feb 6, 2024
…an flushInterval (#2027) * Set version to 8.10.1 * Add tests for bulk helper with various flush and server timeouts * Copy and empty bulkBody when flushBytes is reached Before it was waiting until after semaphore resolved, then sending with a reference to bulkBody. If flushInterval is reached after `await semaphore()` but before `send(bulkBody)`, onFlushTimeout is "stealing" bulkBody so that there is nothing left in bulkBody for the flushBytes block to send, causing an indefinite hang for a promise that does not resolve. * comment typo fixes --------- Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> (cherry picked from commit 1607a0d)
pquentin
pushed a commit
that referenced
this pull request
Feb 6, 2024
…an flushInterval (#2027) (#2129) * Set version to 8.10.1 * Add tests for bulk helper with various flush and server timeouts * Copy and empty bulkBody when flushBytes is reached Before it was waiting until after semaphore resolved, then sending with a reference to bulkBody. If flushInterval is reached after `await semaphore()` but before `send(bulkBody)`, onFlushTimeout is "stealing" bulkBody so that there is nothing left in bulkBody for the flushBytes block to send, causing an indefinite hang for a promise that does not resolve. * comment typo fixes --------- Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> (cherry picked from commit 1607a0d) Co-authored-by: Josh Mock <joshua.mock@elastic.co>
JoshMock
added a commit
to elastic/elasticsearch-serverless-js
that referenced
this pull request
Apr 3, 2024
JoshMock
added a commit
to elastic/elasticsearch-serverless-js
that referenced
this pull request
Apr 3, 2024
Applies patches from elastic/elasticsearch-js#2199 and elastic/elasticsearch-js#2027, adding support for an onSuccess callback and fixing a bug that would cause the helper to hang when the flushInterval was lower than the request timeout. --------- Co-authored-by: JoshMock <160161+JoshMock@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A possible fix for #1562.
The failing state in the above issue is reached when a server's response times are slower than
flushInterval. What happens in this situation, in this order:flushBytesblock awaitssemaphore()onFlushTimeoutis triggeredonFlushTimeout"steals" the bulk body rows that theflushBytesblock is waiting to send by emptyingbulkBodybefore awaiting a semaphoreflushBytesblock's semaphore resolves, it tries tosend(bulkBody)butbulkBodyis emptyBy having
flushBytesset its blocks aside before awaiting the semaphore, likeonFlushTimeoutalready does, it appears to solve the situation where a Promise sits unresolved indefinitely.@delvedor I'm particularly interested in your thoughts on this solution, since you wrote the helper and chose the semaphore strategy.