Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decoding error (36) : Corrupted block detected #170

Closed
fproulx-dfuse opened this issue Oct 18, 2019 · 6 comments · Fixed by #173
Closed

Decoding error (36) : Corrupted block detected #170

fproulx-dfuse opened this issue Oct 18, 2019 · 6 comments · Fixed by #173

Comments

@fproulx-dfuse
Copy link

Hello,

I am unable to decompress some (like one out of several thousands) of files compressed using your library in *.tar.zst (I use mholt/archiver, which uses your lib as backend). We have forked mholt to update to v1.8.6, still the same isssue.

The error we get with unzstd is Decoding error (36) : Corrupted block detected
@klauspost
Copy link
Owner

klauspost commented Oct 18, 2019 •
edited
Loading

Is it possible share a sample?

Maybe email a link to ----- if you want to keep it private.

@fproulx-dfuse
Copy link
Author

fproulx-dfuse commented Oct 18, 2019 •
edited
Loading

I have provided sample files via e-mail - Hopefully that helps to track down the issue.

@fproulx-dfuse
Copy link
Author

@abourget ^

@klauspost
Copy link
Owner

klauspost commented Oct 19, 2019 •
edited
Loading

Thanks for the test case!

This seems to go back to #146 and possibly outputting too much data if tables are reused.

If may be fixable by going back to v1.7.4. I am pretty much on my way out the door for a few days, so it may take me a little longer than expected to get a fix ready.

@abourget abourget mentioned this issue Oct 23, 2019
Closed
klauspost added a commit that referenced this issue Oct 24, 2019
@klauspost
zstd: Fix reuse of huff0 when data hard to compress
2988758 
zstd would reject huff0 compressed literals if the improvement was too small.

However, this would update the huff0 state to contain a new table which could be reused. In that case a wrong table could be used for the next block.

We move the rejection code to huff0, so the state can be properly maintained.

Fixes #170
@klauspost klauspost mentioned this issue Oct 24, 2019
Merged
@klauspost
Copy link
Owner

Found the issue. See #173 I will merge it to master and have fuzz tests run before I make a release (probably v1.9.0)

klauspost added a commit that referenced this issue Oct 24, 2019
@klauspost
zstd: Fix reuse of huff0 when data hard to compress (#173)
54198fb 
* zstd: Fix reuse of huff0 when data hard to compress

zstd would reject huff0 compressed literals if the improvement was too small.

However, this would update the huff0 state to contain a new table which could be reused. In that case a wrong table could be used for the next block.

We move the rejection code to huff0, so the state can be properly maintained.

Fixes #170
@klauspost
Copy link
Owner

@fproulx-eoscanada v1.9.0 released

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

zstd: Fix reuse of huff0 when data hard to compress klauspost/compress
2 participants
@klauspost @fproulx-dfuse