-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix sequence compression API in Explicit Delimiter mode #3023
Conversation
0f2816a
to
55f5946
Compare
Looks like theres an assert in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM once the tests pass
lib/common/error_private.c
Outdated
@@ -27,7 +27,7 @@ const char* ERR_getErrorString(ERR_enum code) | |||
case PREFIX(version_unsupported): return "Version not supported"; | |||
case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter"; | |||
case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding"; | |||
case PREFIX(corruption_detected): return "Corrupted block detected"; | |||
case PREFIX(corruption_detected): return "Input corruption detected"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Maybe "Data corruption detected"? It feels more natural than "Input corruption"
55f5946
to
87fb8a5
Compare
lib/compress/zstd_compress.c
Outdated
cctx->blockSize, remaining, | ||
inSeqs, inSeqsSize, seqPos); | ||
U32 const lastBlock = (blockSize == remaining); | ||
assert(blockSize <= remaining); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this assert is failing.
Maybe it just needs to be moved below the FORWARD_IF_ERROR()
check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's exactly the issue,
blockSize
can be an error code, so this assert()
is only valid after the FORWARD_IF_ERROR()
.
c89e7c8
to
6096d33
Compare
add explicit delimiter mode to libfuzzer test
6096d33
to
fc2ea97
Compare
The misplaced |
In Explicit Delimiter mode, the Sequence compression API was only working when
the frame has only a single block
or when all blocks are full (except the last one).
This fix makes is possible to select block boundaries wherever the caller wants them.
It still requires that data conforms to
zstd
specification, aka block sizes still have a maximum.The fuzzer capability has been extended to test the explicit delimiter mode too.