Bugfixes for the Block-Level Sequence Producer API #3433
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes four bugs which we were very lucky to catch before the 1.5.4 release, mostly thanks to testing by @GarenJian-Intel (who caught three of the bugs).
Some of these bugs, it turns out, can be caught by our fuzzers with only a small modification to
FUZZ_setRandomParameters()
. I got fuzzing working on my machine, was able to reproduce bugs 1, 2, and 4, and will put that code up as a separate PR. I ran the fuzzers for a couple hours and they haven't found any additional bugs (we'll see if OSS-fuzz gets a different result). Will discuss with @terrelln whether it makes sense to modify our fuzzers so they can catch bug 3.Bug 1
The external matchfinder API is not compatible with
nbWorkers > 0
right now. This PR fails compression in that case, rather than failing an assert (debug builds) or crashing the application (release builds).ZSTD_CCtx_init_compressStream2()
zstreamtest.c
zstd.h
Bug 2
Prior to this PR, some input sizes would lead to an incorrect estimate of memory required for the
ZSTD_cwksp
. This would cause compression to fail, since the workspace would run out of space for sub-allocations after making the initial shared allocation.ZSTD_estimateCCtxSize_usingCCtxParams_internal()
Bug 3
The external matchfinder doesn't interact with older compression APIs, which means that this assert isn't always true. libzstd is behaving correctly, it's just the assert which is incorrect.
zstreamtest.c
zstd.h
Bug 4
When I was working on the fuzzers, I realized that I need a way to clear the external matchfinder. After discussing with @Cyan4973, I realized that there was a bug with calls of the form
ZSTD_registerExternalMatchFinder(cctx, NULL, NULL)
, which should clear the matchfinder.ZSTD_registerExternalMatchFinder()
such that it clears any previously registered external matchfinder when called with aNULL
function pointer.zstreamtest.c
zstd.h