Fact finding: Registry & client support for Image Manifest type artifacts

[brackendawson]

To hopefully help with #999 and #1015 I've tested various flavors of Image Manifest with various registries and client software. Please to chip in if you can try other registries & clients. And definitely let me know if the working group has documented something similar.

NB: The ✅ and ❌ do not indicate conformance, only of what we could do now would work in existing implementations.

Registries

Test manifest PUT and GET

Registry	Arbitrary Config mediaType	No Config	Empty Layers	No Layers	No Subject	Zero Layer	Scratch Layer
distribution	✅	❌ PUT: "blob unknown to registry"	✅	✅	✅	✅	✅
ICR	✅	❌ PUT: "blob unknown to registry"	✅	✅	✅	❌ 500 error uploading zero blob	✅
Docker Hub	✅*	❌ PUT: "manifest config: mediaType and digest can not be empty"	✅*	✅*	✅	✅*	✅*
Quay.io	✅	❌† PUT: "manifest invalid"	✅	❌ PUT: "manifest invalid"	✅	❌ 500 error uploading zero blob	✅
GCR	✅	❌ PUT: "Manifest is missing required field "config"."	❌ "Manifest is missing required field "layers"."	❌ "Manifest is missing required field "layers"."	✅	✅	✅
AR	✅	❌ PUT: "failed to read config blob"	✅	✅	✅	✅	✅
Azure	✅	❌ PUT: "manifest invalid"	✅	✅	✅	✅	✅
ECR	✅	❌ "Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'"	❌ "unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Image has to have at least 1 layer'"	❌ "unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Image has to have at least 1 layer'"	✅	✅	✅
Artifactory	✅	❌ PUT: "manifest invalid"	✅	❌ PUT: "manifest invalid"	✅	✅	✅
Harbor	✅	❌ PUT: "blob unknown to registry"	✅	✅	✅	❌ PUT: "blob unknown to registry"	✅

* Docker Hub responds with "not found" to any OCI Image Manifest with a subject, even if the subject exists in the same repository. Subjects have been temporarily disabled.

Clients

Test pull and push

Client	Arbitrary Config mediaType	No Config	Empty Layers	No Layers	No Subject	Zero Layer	Scratch Layer
Docker 20.10.21	❌* pull: "invalid rootfs in image configuration"	-	✅†	✅†	✅†	✅†	✅†
Skopeo 1.10.0 (github.com/containers/image)	✅	-	✅	✅	✅	✅	✅

* Runtimes unpacking artifacts are almost certain to fail.

† If performed with a compatible config blob.

Manifests

The subject in these images is: dev.icr.io/oci-artifacts/test:image

Arbitrary Config mediaType

dev.icr.io/oci-artifacts/test:artifact

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "image/gif",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "layers": [
        {
            "mediaType": "image/gif",
            "digest": "sha256:725c49c527a83669901d00392768df9f653b1964a056c54232bc4c93003ddb48",
            "size": 3540101
        }
    ]
}

No config

I can't upload this anywhere, but it references the same blobs as the one above.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "layers": [
        {
            "mediaType": "image/gif",
            "digest": "sha256:725c49c527a83669901d00392768df9f653b1964a056c54232bc4c93003ddb48",
            "size": 3540101
        }
    ]
}

Empty Layers

dev.icr.io/oci-artifacts/test:artifact-empty-layers

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "image/gif",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "layers": [],
    "annotations": {
        "com.example.key1": "value1"
    }
}

No Layers

dev.icr.io/oci-artifacts/test:artifact-no-layers

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "image/gif",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "annotations": {
        "com.example.key1": "value1"
    }
}

No Subject

dev.icr.io/oci-artifacts/test:artifact-no-subject

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "image/gif",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "layers": [
        {
            "mediaType": "image/gif",
            "digest": "sha256:725c49c527a83669901d00392768df9f653b1964a056c54232bc4c93003ddb48",
            "size": 3540101
        }
    ]
}

Zero Layer

alandaws/oci-artifacts:artifact-0-layer (Subject is missing because Docker Hub).

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "image/gif",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "layers": [
        {
            "mediaType": "application/octet-stream",
            "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "size": 0
        }
    ]
}

Scratch Layer

dev.icr.io/oci-artifacts/test:artifact-scratch-layer

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "text/comment",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
    },
    "subject": {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:71dbae9d7e6445fb5e0b11328e941b8e8937fdd52465079f536ce44bb78796ed",
        "size": 406
    },
    "layers": [
        {
            "mediaType": "application/octet-stream",
            "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
            "size": 2
        }
    ],
    "annotations": {
        "com.example.comment": "Hello world"
    }
}

[brackendawson]

So from what I've gathered so far a config blob is always needed in practice and if you have no binary blob to store (annotation only artifact) then a small blob, empty layers array or missing layers field all seem equivalent.

As the spec is written now, I think most would say that allowing layers to be empty or absent breaks compatibility because one reading of the Image Manifest spec says that layer 0 is mandatory. In practice nobody that I tested has implemented it that way. So what shall we prescribe as the way to write an Image Manifest for an annotations only artifact? This is something I think should be clarified regardless of whether the Artifact Manifest is dropped or not.

[vbatts]

this is the confusion with the current spec wording: it was thought to say "the layer at index 0 is intended to be the base layer", and not that it must exist.

[brackendawson]

I want to hear more opinions but I'd be open to making layers optional based on this. Again, I don't have a vote.

[tianon]

FWIW, Docker Hub did add support for subject but backed it out temporarily.

It does support both empty layers and no layers without a subject field, though:

• https://explore.ggcr.dev/?image=tianon/scratch:oci
• https://explore.ggcr.dev/?image=tianon/scratch:nolayers

The Docker Engine/CLI also has no problem with either image:

$ docker pull tianon/scratch:oci
oci: Pulling from tianon/scratch
Digest: sha256:3d455195f1141b05cb8cc6deb07ea96b617e5436c7a259d9f3e7b1985f4d3f51
Status: Downloaded newer image for tianon/scratch:oci
docker.io/tianon/scratch:oci

$ docker pull tianon/scratch:nolayers
nolayers: Pulling from tianon/scratch
Digest: sha256:aad957e043852ec4cd630cabb56da94c024a0bb3a72bd4d530a13b8f6b752ca7
Status: Downloaded newer image for tianon/scratch:nolayers
docker.io/tianon/scratch:nolayers

[jonjohnsonjr]

Can you point us at public versions of these images so we can reuse your test cases?

[brackendawson]

Can you point us at public versions of these images so we can reuse your test cases?

Yes, references added above.

[jonjohnsonjr]

Credit card needed, can anyone help?

Registry	Artifact	No Config	Empty Layers	No Layers	No Subject
GCR	✅	❌ PUT: "Manifest is missing required field "config"."	❌ "Manifest is missing required field "layers"."	❌ "Manifest is missing required field "layers"."	✅
AR	✅	❌ PUT: "failed to read config blob"	✅	✅	✅

[AaronFriel]

Anyone have an active Artifactory config to check their support?

[sudo-bmitch]

I believe we ignored the option to have a single layer with a 0 size blob because of a registry support. Can we add that scenario to the list of tests? I want to be sure it's an issue with multiple registries and not just a one-off.

[brackendawson]

Anyone have an active Artifactory config to check their support?

Yes, added.

I believe we ignored the option to have a single layer with a 0 size blob because of a registry support. Can we add that scenario to the list of tests? I want to be sure it's an issue with multiple registries and not just a one-off.

Added, I don't think this is a great option for two reasons:

1. Older versions of distribution using at least the S3 storage driver fail to store zero length blobs.
2. It requires the client to push a second blob, scratch for the config and zero for the layer. Why not just re-use the scratch layer? The result is smaller content.

[brackendawson]

As of this afternoon Quay.io are allowing any config media type, chart updated. Thanks siblings, I guess..

[AaronFriel]

Never underestimate the power of a ❌ in a feature table to motivate change ;)

[jlbutler]

Working on providing ECR results for the table above.

Registry	Artifact	No Config	Empty Layers	No Layers	No Subject	Zero Layer	Scratch Layer
ECR	✅	❌ "Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'"	❌ "unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Image has to have at least 1 layer'"	❌ "unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Image has to have at least 1 layer'"	✅	✅	✅

[imjasonh]

Can I suggest we rename the heading Artifact in favor of Arbitrary config mediaType or something?

The term "artifact" is very overloaded, and I think it could be misconstrued to mean "support for the v1.1-RC artifact manifest", which is out of scope of this fact finding mission.

[rchincha]

Just curious, how many customers are actually pushing OCI images to registries?

[jkjell]

I did some testing on Harbor:

Registry	Arbitrary Config mediaType	No Config	Empty Layers	No Layers	No Subject	Zero Layer	Scratch Layer
Harbor	✅	❌ PUT: "blob unknown to registry"	✅	✅	✅	❌ PUT: "blob unknown to registry"	✅

[jonjohnsonjr]

GCR and AR both appear to work with both zero layer and scratch layer.

[sajayantony]

This thread is just awesome. It helped in driving change building clarity. Kudos to @brackendawson.

[brackendawson]

As v1.1.0 has been released I think this has served its purpose. Thank you everyone for helping me push this cat gif to the various registries.