You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This assert which was added as part of short cache has uncovered two bugs:
The short cache PR only validates that hashLog is <= 24 bits. We need to do the same for chainLog, this has showed up in some CI failures.
The row based matchfinder needs to have hashLog capped at 28 bits (24 + 4). The assert linked above can currently be triggered by ./zstd -o /dev/null -7 < ~/silesia.tar --zstd=hlog=29
These bugs can't lead to data corruption, but they do have some bad effects:
Can regress compression ratio by corrupting the hashtable
Every time the assert fails, that is undefined behavior (shift by larger than width of type)
* Cap shortCache chainLog to 24
* Cap row match finder hashLog so that rowLog <= 24
* Add unit tests to expose all cases. The row match finder unit tests
are only run in 64-bit mode, because they allocate ~1GB.
Fixesfacebook#3336
* Cap shortCache chainLog to 24
* Cap row match finder hashLog so that rowLog <= 24
* Add unit tests to expose all cases. The row match finder unit tests
are only run in 64-bit mode, because they allocate ~1GB.
Fixesfacebook#3336
terrelln
added a commit
to terrelln/zstd
that referenced
this issue
Jan 20, 2023
* Cap shortCache chainLog to 24
* Cap row match finder hashLog so that rowLog <= 24
* Add unit tests to expose all cases. The row match finder unit tests
are only run in 64-bit mode, because they allocate ~1GB.
Fixesfacebook#3336
This assert which was added as part of short cache has uncovered two bugs:
./zstd -o /dev/null -7 < ~/silesia.tar --zstd=hlog=29
These bugs can't lead to data corruption, but they do have some bad effects:
Code pointer:
zstd/lib/compress/zstd_compress.c
Line 1443 in 4f7183d
The text was updated successfully, but these errors were encountered: