-
Notifications
You must be signed in to change notification settings - Fork 994
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace gzip
with zstd
for speed + size benefits
#7950
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I have been meaning for a while to experiment with different compression formats, so took the time to play around with it for a few mins this morning for fun. @Nishnha this is likely a better solution to the |
jeffwidman
force-pushed
the
replace-gzip-with-zstd
branch
from
September 1, 2023 17:59
d6b8dc5
to
9dd0dae
Compare
Replace `gzip` with `zstd` for both speed + size benefits. I added `zstd` to the base image rather than the python-specific image because: 1. it's only 1,695 KB 2. We'll likely use it in other places down the road, such as unpacking deps for jobs 3. The Python image would have required additional steps like `apt-get update` This results in faster compression + space savings: ```shell root@ca684869d4af:/usr/local/.pyenv/versions# time tar -acf 3.11.5.tar.gz 3.11.5 real 0m5.564s user 0m5.458s sys 0m0.626s root@ca684869d4af:/usr/local/.pyenv/versions# time tar -acf 3.11.5.tar.zst 3.11.5 real 0m0.850s user 0m1.003s sys 0m0.354s root@ca684869d4af:/usr/local/.pyenv/versions# ls -lah -rw-r--r-- 1 root root 32M Sep 1 17:19 3.11.5.tar.gz -rw-r--r-- 1 root root 30M Sep 1 17:20 3.11.5.tar.zst ``` As well as faster decompression too: ```shell root@ca684869d4af:/usr/local/.pyenv/versions# ls -lah -rw-r--r-- 1 root root 32M Sep 1 17:38 3.11.5.tar.gz -rw-r--r-- 1 root root 30M Sep 1 17:36 3.11.5.tar.zst root@ca684869d4af:/usr/local/.pyenv/versions# time tar -axf $PYENV_ROOT/versions/3.11.5.tar.gz -C $PYENV_ROOT/versions real 0m1.113s user 0m0.986s sys 0m0.811s root@ca684869d4af:/usr/local/.pyenv/versions# rm -rf 3.11.5 root@ca684869d4af:/usr/local/.pyenv/versions# time tar -axf $PYENV_ROOT/versions/3.11.5.tar.zst -C $PYENV_ROOT/versions real 0m0.774s user 0m0.501s sys 0m0.695s ``` The `-a` flag to `tar` tells it to autoselect the compression format based on the file extension, so it's flipping between `gzip` and `zstd`.
jeffwidman
force-pushed
the
replace-gzip-with-zstd
branch
from
September 1, 2023 18:06
9dd0dae
to
bdf8bde
Compare
jakecoffman
approved these changes
Sep 1, 2023
brettfo
pushed a commit
to brettfo/dependabot-core
that referenced
this pull request
Oct 11, 2023
Replace `gzip` with `zstd` for both speed + size benefits. I added `zstd` to the base image rather than the python-specific image because: 1. it's only 1,695 KB 2. We'll likely use it in other places down the road, such as unpacking deps for jobs 3. The Python image would have required additional steps like `apt-get update` This results in faster compression + space savings: ```shell root@ca684869d4af:/usr/local/.pyenv/versions# time tar -acf 3.11.5.tar.gz 3.11.5 real 0m5.564s user 0m5.458s sys 0m0.626s root@ca684869d4af:/usr/local/.pyenv/versions# time tar -acf 3.11.5.tar.zst 3.11.5 real 0m0.850s user 0m1.003s sys 0m0.354s root@ca684869d4af:/usr/local/.pyenv/versions# ls -lah -rw-r--r-- 1 root root 32M Sep 1 17:19 3.11.5.tar.gz -rw-r--r-- 1 root root 30M Sep 1 17:20 3.11.5.tar.zst ``` As well as faster decompression too: ```shell root@ca684869d4af:/usr/local/.pyenv/versions# ls -lah -rw-r--r-- 1 root root 32M Sep 1 17:38 3.11.5.tar.gz -rw-r--r-- 1 root root 30M Sep 1 17:36 3.11.5.tar.zst root@ca684869d4af:/usr/local/.pyenv/versions# time tar -axf $PYENV_ROOT/versions/3.11.5.tar.gz -C $PYENV_ROOT/versions real 0m1.113s user 0m0.986s sys 0m0.811s root@ca684869d4af:/usr/local/.pyenv/versions# rm -rf 3.11.5 root@ca684869d4af:/usr/local/.pyenv/versions# time tar -axf $PYENV_ROOT/versions/3.11.5.tar.zst -C $PYENV_ROOT/versions real 0m0.774s user 0m0.501s sys 0m0.695s ``` The `-a` flag to `tar` tells it to autoselect the compression format based on the file extension, so it's flipping between `gzip` and `zstd`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace
gzip
withzstd
for both speed + size benefits.I added
zstd
to the base image rather than the python-specific image because:apt-get update
zstd
would take more time than is saved with the compression speedup--and that time would apply to all images, but looking at the raw CI logs and grep'ing forzst
shows that it takes < 0.5s to install it.This results in faster compression + space savings:
As well as faster decompression too:
The
-a
flag totar
tells it to autoselect the compression format based on the file extension, so it's flipping betweengzip
andzstd
.