[configgrpc] create zstd.Decoder's using WithDecoderConcurrency(1) #10360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The zstd.Decoder documentation states: > It is important to use the "Close" function when you no longer need > the Reader to stop running goroutines, when running with default > settings. Goroutines will exit once an error has been returned, > including io.EOF at the end of a stream. > > Streams are decoded concurrently in 4 asynchronous stages to give > the best possible throughput. However, if you prefer synchronous > decompression, use WithDecoderConcurrency(1) which will decompress > data as it is being requested only. Since we store zstd.Decoder's in a sync.Pool, we should probably use WithDecoderConcurrency(1), so that we don't need to call Close() when they are dropped from the pool. Relates to #10323
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10360 +/- ##
=======================================
Coverage 92.54% 92.54%
=======================================
Files 387 387
Lines 18254 18254
=======================================
Hits 16894 16894
Misses 1015 1015
Partials 345 345 ☔ View full report in Codecov by Sentry. |
songy23
requested changes
Jun 12, 2024
There was a problem hiding this comment.
Looks like #10394 supersedes this PR since the internal fork of zstd is removed.
cc @jpkrohling
|
Despite this being closed as not merged, I wanted to thank you for your time to open this PR here, @mostynb, and we appreciate the work you put also on the library we are using for gRPC payload compression. |
|
No worries- you're welcome. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The zstd.Decoder documentation states:
Since we store zstd.Decoder's in a sync.Pool, we should probably use WithDecoderConcurrency(1), so that we don't need to call Close() when they are dropped from the pool.
Link to tracking issue
Relates to #10323
Testing
TODO