Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable zstd:chunked support in containers/image #775

Merged
merged 4 commits into from
May 14, 2021
Merged

Enable zstd:chunked support in containers/image #775

merged 4 commits into from
May 14, 2021

Commits on May 7, 2021

  1. drivers: add interface to directly write diff 

    Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
    @giuseppe
    giuseppe committed May 7, 2021
    Configuration menu
    Copy the full SHA
    c2a5fd1 View commit details
    Browse the repository at this point in the history

  2. overlay: implement direct diff access 

    Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
    @giuseppe
    giuseppe committed May 7, 2021
    Configuration menu
    Copy the full SHA
    c2c9ca9 View commit details
    Browse the repository at this point in the history

  3. types: add map PullOptions to the conf file 

    Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
    @giuseppe
    giuseppe committed May 7, 2021
    Configuration menu
    Copy the full SHA
    7400647 View commit details
    Browse the repository at this point in the history

  4. pkg/chunked: new zstd variant zstd:chunked 

    add a new custom variant of the zstd compression that permits to
retrieve each file separately.

The idea is based on CRFS and its stargz format for having seekable
and indexable tarballs.

One disadvantage of the stargz format is that a custom file is added
to the tarball to store the layer metadata, and the metadata file is
part of the image itself.  Clients that are not aware of the stargz
format will propagate the metadata file inside of the containers.

The zstd compression supports embeddeding additional data as part of
the stream that the zstd decompressor will ignore (skippable frame),
so the issue above with CRFS can be solved directly within the zstd
compression format.

Beside this minor advantage, zstd is much faster and compresses better
than gzip, so take this opportunity to push the zstd format further.

The zstd compression is supported by the OCI image specs since August
2019: opencontainers/image-spec#788 and has
been supported by containers/image since then.

Clients that are not aware of the zstd:chunked format, won't notice
any difference when handling a blob that uses the variant.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
    @giuseppe
    giuseppe committed May 7, 2021
    Configuration menu
    Copy the full SHA
    a0ff89e View commit details
    Browse the repository at this point in the history