Fix required decompression memory usage reported by -vv + --long #3042
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cyan4973
reviewed
Feb 1, 2022
programs/fileio.c
Outdated
| if (windowLog == 0) { | ||
| if (prefs->ldmFlag) { | ||
| /* If long mode is set libzstd will set this window size internally */ | ||
| windowLog = ZSTD_WINDOWLOG_LIMIT_DEFAULT; |
There was a problem hiding this comment.
--long mode can be activated with a parameter.
For example, --long=30 means "use a 1 GB window size".
The shorter --long is equivalent to --long=27.
This means we can't guarantee that the window size is necessarily ZSTD_WINDOWLOG_LIMIT_DEFAULT when the --long mode is activated, it could be a different value if an explicit parameter was passed.
There was a problem hiding this comment.
Thanks, yes, there's an override via ZSTD_overrideCParams that I missed, so the ldmFlag check should only be performed if the windowLog ends up as 0.
The use of --long alters the window size internally in the underlying library (lib/compress/zstd_compress.c:ZSTD_getCParamsFromCCtxParams), which changes the memory required for decompression. This means that the reported requirement from the zstd binary when -vv is specified is incorrect. A full fix for this would be to add an API call to be able to retrieve the required decompression memory from the library, but as a lighterweight fix we can just take account of the fact we've enabled long mode and update our verbose output appropriately. Fixes facebook#2968
Cyan4973
approved these changes
Feb 2, 2022
|
Thanks @u1f35c ! |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The use of --long alters the window size internally in the underlying
library (lib/compress/zstd_compress.c:ZSTD_getCParamsFromCCtxParams),
which changes the memory required for decompression. This means that the
reported requirement from the zstd binary when -vv is specified is
incorrect.
A full fix for this would be to add an API call to be able to retrieve
the required decompression memory from the library, but as a
lighterweight fix we can just take account of the fact we've enabled
long mode and update our verbose output appropriately.
Fixes #2968