Reuse compression objects #1185
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
October 4, 2018 17:46
Use sync.Pool to reuse lz4 and gzip reader objects across decompressions. lz4 in particular makes a large allocation per-reader, so you spend all your time in GC if make a new reader per-message. Benchmarking reading 500 messages/s with 3 consumers and 32 partitions, lz4 consumer CPU fell from ~120% to ~5%. gzip went from ~20% to ~5%.
Similar to decompressing, you can reuse *lz4.Writer and *gzip.Writer objects by calling Reset() between uses. Add two more sync.Pools so we can reuse writers easily. The gzip writer is only reused when using the default compression. We could maintain a separate sync.Pool for each gzip compression level, but that adds more complexity and gzip wasn't that slow without this optimization. When producing 100 msgs/second across 10 producers, CPU usage dropped from 70% to 4% on my machine (lz4). Gzip dropped from 15% to 5%.
|
Yes,I used 1.17.0 and gzip, will lead to gc use cpu very high. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reusing lz4/gzip reader and writer objects via sync.Pool greatly reduces CPU usage (saves allocations and creates much less garbage).