image-builder: Use zstd for a workspace image compression instead of gzip

[utam0k]

Is your feature request related to a problem? Please describe

Support for zstd¹²³ compression, now we are using gzip, which is default for buildkit
zstd is said to be much faster than gzip.

dazzle has already implemented gitpod-io/dazzle#58

AFAI, the implementation is simply to add compression=zstd to the following line. So it is worth a try.

gitpod/components/image-builder-bob/pkg/builder/builder.go

Line 141 in 2c451bc

"--output=type=image,name=" + target + ",push=true,oci-mediatypes=true",

Describe the behaviour you'd like

Pull and push a workspace image with zstd.
You should be able to check the manifest using oci-tool.

Describe alternatives you've considered

Additional context

docker (or rather moby) has not yet released a version that supports zstd. Implementation is complete.

Footnotes

1. https://www.infoq.com/news/2022/09/amazon-gzip-zstd/ ↩

2. https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/ ↩

3. https://www.slideshare.net/KoheiTokunaga/starting-up-containers-super-fast-with-lazy-pulling-of-images ↩

[kylos101]

@utam0k can you update the issue description to better describe how this will benefit users? For example, will users see time savings when pulling container images? If yes, what is the expected time savings?

[kylos101]

For now I've classified this change as "user quality of life". In other words, it does not seem like it is an essential change that we need "now". Is that right, @aledbf and @utam0k , that it's a nice to have? I'm just trying to make sure I understand the related motivation or "the why" for this issue.

[utam0k]

@kylos101 Thanks for your comment.
We could take detailed data, but that would take some time. For example, how small the size of the image will be and the time it will take to affect the startup of the container. We didn't dare to do that this time because I need time. Do you want me to decide on a time box and take data?

[utam0k]

For now I've classified this change as "user quality of life". In other words, it does not seem like it is an essential change that we need "now".

yes, However, I expect implementation to be fairly straightforward.

However, it depends on the containerd version(>=1.5), so it may be necessary to separate the case for self-hosted.

[kylos101]

Do you want me to decide on a time box and take data?

👋 @utam0k no, thank you for asking, though. 😄 Please ignore this issue for now, let's leave in the inbox. That way, you can focus on groundwork that is either in-progress or scheduled.

[utam0k]

Do you want me to decide on a time box and take data?

👋 @utam0k no, thank you for asking, though. 😄 Please ignore this issue for now, let's leave in the inbox. That way, you can focus on groundwork that is either in-progress or scheduled.

I agree with you👍 I was just going to write the issue as an idea as I don't think we should spend time on it now either.

[atduarte]

For now I've classified this change as "user quality of life".

@kylos101 how does this benefit the end user? Shorter workspace startup times?

[kylos101]

how does this benefit the end user? Shorter workspace startup times?

Yes @atduarte , shorter workspace startup times on image build and pull, specifically.

We'd have to change parts of our stack, though. Dazzle, image-builder-mk3, and registry-facade. I'm not sure about IPFS. There may be others.

Related internal conversations [1][2]

edit: 👇

General zstd goodness:
https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/
https://github.com/facebook/zstd
https://lemire.me/blog/2021/06/30/compressing-json-gzip-vs-zstd/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.