Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZSTD CLI: Use buffered output #2985

Merged
merged 1 commit into from
Jan 21, 2022
Merged

Conversation

yoniko
Copy link
Contributor

@yoniko yoniko commented Jan 8, 2022

When opening an output file the zstd CLI will try to set it to buffered mode with a 1MB buffer.
Testing shown possible performance improvement on linux based machine and a major performance improvement on macOS:

comment                    dev  setvbuf  improvement
input_file      machine
enwik8.zst      desktop  0.210    0.211    -0.476190
                macbook  0.468    0.217    53.632479
                server   0.229    0.221     3.493450
silesia.tar.zst desktop  0.387    0.386     0.258398
                macbook  0.870    0.331    61.954023
                server   0.419    0.409     2.386635

Note: as per code comment, this usage of setvbuf might not be supported on legacy systems, but even in this case it shouldn't impact overall success of the execution.

@Cyan4973
Copy link
Contributor

Cyan4973 commented Jan 8, 2022

Looks good to me !

@yoniko yoniko merged commit 8ab95f2 into facebook:dev Jan 21, 2022
@Cyan4973 Cyan4973 mentioned this pull request Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants