Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy images using OCI manifests #951

Merged
merged 1 commit into from
Oct 13, 2022
Merged

Copy images using OCI manifests #951

merged 1 commit into from
Oct 13, 2022

Conversation

aledbf
Copy link
Member

@aledbf aledbf commented Oct 13, 2022

Description

Use OCI manifests for the publication of the images instead of docker.

The goal here is to enable the research of lazy snapshotters.

How to test

  • Use oci-tool to verify the published images use OCI instead docker media types

@gitpod-io
Copy link

gitpod-io bot commented Oct 13, 2022

@gitpod-staging
Copy link

@aledbf aledbf marked this pull request as ready for review October 13, 2022 22:49
@aledbf aledbf requested a review from a team October 13, 2022 22:49
@@ -72,7 +72,7 @@ jobs:
IMAGES=$(cat .github/promote-images.yml | yq '."europe-docker.pkg.dev/gitpod-artifacts/docker-dev"."images-by-tag-regex"|keys[]' -r)
for IMAGE in $IMAGES;
do
sudo skopeo copy \
sudo skopeo copy --format=oci --dest-oci-accept-uncompressed-layers \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question. Why is it worth uncompressing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for research?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it worth uncompressing?

We are just copying something. If we want to compress it, let's do that during the build, not the copy.
At this point, this should be dumb.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for research?

This is the first step to anything interesting in the future, like SOCI

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see! Thanks for your explanation 🙏
By the way, can we think of using zstd:chunked?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, can we think of using zstd:chunked?

Yes? But I think this is missing in buildkit opencontainers/image-spec#803

@aledbf
Copy link
Member Author

aledbf commented Oct 13, 2022

Merging. I want to see if there's any failure during the push to dockerhub and also check the change in mediaType.

@aledbf aledbf merged commit 62f8553 into main Oct 13, 2022
@aledbf
Copy link
Member Author

aledbf commented Oct 13, 2022

@utam0k please open a PR in dazzle to add compression=zstd

@utam0k
Copy link
Contributor

utam0k commented Oct 13, 2022

@utam0k please open a PR in dazzle to add compression=zstd

Sure!

@aledbf
Copy link
Member Author

aledbf commented Oct 14, 2022

❯ oci-tool fetch manifest docker.io/gitpod/workspace-full:2022-10-14-01-57-06
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:4c8d4320ba4f5ccbd700ed75841dac1768b25379f44cd84927f4c8e5cd599e02",
    "size": 35755
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:675920708c8bf10fbd02693dc8f43ee7dbe0a99cdfd55e06e6f1a8b43fd08e3f",
      "size": 28572685
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:aceaa9f66a60a3695a6348be449df629aed61f57a9cbd55a388bdcfa40cb27b7",
      "size": 7747479
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e5d69e69900ca0c9eb8762e001879c1e84e29d94a2397e389440b1d8689445e5",
      "size": 3624451
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:035d79e04aad5acedcb6ebd870819e030f0f1cb534d55136ac5f6087b31b39f9",
      "size": 60738066
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:42f65cb058b8e19392f740a77af749897af039a43730af19d5b986a7de8cc696",
      "size": 145052588
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:20769770acc9f09b7feb5781525d3869bb2ad18e3de597ef3de821fa4cf0cb14",
      "size": 847
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:418ad0ba0e8c4c1788f10e2a5a4bdceb6a7f3d02a19139dc430b3bacf73a221e",
      "size": 121835719
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b02aec010b52dc61028012896521789e4cecc54ca3bd545727b4e6942db308be",
      "size": 114300
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e359f970aba7fc7a3486fd3f3185b5294a69f55978c44a2fdfe27f7344e62a61",
      "size": 26998079
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:90fa85db082f89fd823c254c4668dbc4d6ba6d91272a7e78c253c7b5b56c4fdf",
      "size": 25299901
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:68321a4914e74046421a07e130b3d9b4ac851cc5f1fe4c00c82d908fd7fd04fc",
      "size": 4118
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
      "size": 32
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:10faf03690a473f82fbf934932d1304426a6730e2e3a980f0fee4e86ee5c0ef5",
      "size": 1947
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4fb029693210f9c0dff76d3d65091fdda3dd956c830bd311a2466e3cd60ebc9e",
      "size": 170
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:6c8e8d14b408a40241bc2f21ee7287a6ca0c14767f8eff344b5c94a19cc9ec5f",
      "size": 202
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:13280044394901039f7d90c1394e6f423639cad3550fc06e5203cf43661cb103",
      "size": 235
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:951c59a3b09dce235b5a08e418cf2065bd34a05527217f4b936d12860f1d3f1e",
      "size": 2125
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c8ce97850f3de098388884d6c0d6cacadeab000f703098a5c8113702614d6f5c",
      "size": 209
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2e9d8f17208062a1aa5737b70c5e5ec7a8d9e89f96b74cf8eff9d3024eb98a1c",
      "size": 90651104
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:27ca5965c5eb526dc6446f3e57da107c22fa9d623dd5e924e78f0a64baf8dd3e",
      "size": 730657
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:87d94ba957f1df6b28502c717f4d22a3083d1555bc419162328bb63ea06630da",
      "size": 14277255
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3004c1c790fa973723ffc4514d20ce789a7d5aaf2d16b3263d35682605539253",
      "size": 220703201
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2a8f98778db78863ed71e23ab61d32ff9033f60f1caed0cecb5d17022f828ddc",
      "size": 707962329
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:0c6b1fc19f9211134289bfdca05e8cb2a9165ff9fe6d72037f37e29582135687",
      "size": 95900813
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:6d6ee475191a2123321433b63399969adab5ff1914431dc05813d76406b44190",
      "size": 455
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:afa65ce204f53d6de543a2d55c9f92bed18245533539b0e5fa1293f05dbecaec",
      "size": 194437584
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:12567cdd52f4b26ed4b7ddbb4dd85299b68268ad558355ab7bb55ca9516eac3b",
      "size": 58206297
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:57c3dd97d4a262d303ad2a89828de0a18351f422205e211ee6665a84d9d9a471",
      "size": 221
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b31e6e06d2f9b615efd905b148aa30ebb0012270d60d4813c7a5ff2529b7491a",
      "size": 2001
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ec0fb79e1aadb0a2478d061f01ad1eccde16744ebac5de225a3617651faefb82",
      "size": 761
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:9f1976005f59c6c2b025913cedfca39d6398362db951aaaf1c2838d6fa0be78f",
      "size": 1663
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c55b9a200b51d469b7a7135d6dc9b9233234c8c55599d1fc66e246d7ead3e536",
      "size": 535
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:745311948e11309fb3ff5909bc54705a8e07ff9bfbfe22ee7c400f65a0a70ad6",
      "size": 122005152
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4bf9acce0cd93e166a81568f4f064d76a202e048097d3036f276150073bad0d1",
      "size": 199
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ab63812fb1b694e8d303f4fecd2ee536eef2723171330a13376c601744d701e5",
      "size": 196114643
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5efd8f5bdb92f860c3fd63c9d59d2d5040f1b0466013691b2a7137ed93ebf744",
      "size": 653765883
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8a165cb20ad858554142a1f42331492dcfc34789da476ab1a1a2f116b03aa3e3",
      "size": 53409997
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:0dc69e0969c7fa01d7d2e130355cb2be09c56a831f469131acab50f9b7c4c7d4",
      "size": 124606374
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:767f03219b5ba311930c5b5410acc74f04beb6d1d642e85adb0d78605c418a3f",
      "size": 1402125
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a8d73ba62449d8cb90df7ab146c50c9bab98121aafe8ac4d49c8e7df550177d3",
      "size": 9103325
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:5d47a44a4d8854b0d63406635e2a772bf6b0b2af5f53f8593592e89ffc38edda",
      "size": 4580926
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ae6281e4b560687b844751fa7dd2e029079bacfc5ecd19a44e1302cbaa3de69e",
      "size": 54001957
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3238cd60e9060eda1cf19c425e2c5adf71886985ad8bcb4b9e2f00811821edb2",
      "size": 841
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8db72d3c672343760f4973cb22b8adbc2f1b2855bc20b7898eefb4c82ba99e4a",
      "size": 641
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:17bfeff8a5f4ee51ab1947a0d3c6bf091c45132983e2729006f8617aed4ed7c0",
      "size": 19336
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
      "size": 32
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a6e7f2ea14450d4651486ec00c398a4ab34604d921d755bfb4c17797d494b1b5",
      "size": 85640826
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:aed313278de56a3078c041c47aa7d8e1a3ee3cb5a13f6a0a50a788c95f3532e8",
      "size": 227
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:320e449c68fcd052d248549e475243f6ca915a000ccbb06f79b9f404c28e76fb",
      "size": 239
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:890b84e1b0b5b25864048a7a9150fc681a08046b348f41b74e6d7f7551df7012",
      "size": 244
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:83d4af21f40188c240c67d8ca17883ac4d7d19c2101b9db904fe4040ae072cba",
      "size": 37390491
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c80874d2ae78b2c1bc00752e8257297db589a30e6311d90b4614b1ee27d12b97",
      "size": 19683378
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d5e8e79f0090f6d9ba17ef11141d0619da8d81ce991327bdb888af29760b5045",
      "size": 50104643
    }
  ],
  "annotations": {
    "dazzle.gitpod.io/base-ref": "europe-docker.pkg.dev/gitpod-artifacts/docker-dev/workspace-base-images:base--f44ced66c8c62d09fb07b94ffe78cdb5834519028ee2c102880d3c14573613ca@sha256:3576141d9b695b2a03737e0c9955c818cb1b95feb200cce126446550cc92ef1f",
    "dazzle.gitpod.io/env-HOME": "use-last",
    "dazzle.gitpod.io/env-PATH": "merge-unique"
  }
}

@aledbf aledbf deleted the aledbf/oci branch October 8, 2023 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants