MANIFEST_INVALID while pushing signature/attestation

[sbose78]

Expected Behavior

Should have been able to push signature/attestations after being able to push the image.
( Image ref: quay.io/shbose/kaniko-chains:0.2 )

Actual Behavior

Push to registry fails.

MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]

Is this the issue with Quay.io not supporting a specific version of the schema?

Here's how my docker manifest inspect... looks like:

docker manifest inspect quay.io/shbose/kaniko-chains:0.2
{
	"schemaVersion": 2,
	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
	"config": {
		"mediaType": "application/vnd.docker.container.image.v1+json",
		"size": 774,
		"digest": "sha256:19f5201aa46fcc76531d0e6219efaeb3c0143935f6abd4889adb04b85f74d5b5"
	},
	"layers": [
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 2811969,
			"digest": "sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
		}
	]
}

Logs

{"level":"info","ts":"2021-10-07T04:24:36.425Z","logger":"watcher","caller":"oci/oci.go:133","msg":"Starting to upload attestations to OCI ...","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"a20ac015-4148-462a-9689-11f39a43a794","knative.dev/key":"foo/kaniko-chains-run-8j48h"}
{"level":"info","ts":"2021-10-07T04:24:36.425Z","logger":"watcher","caller":"oci/oci.go:136","msg":"Starting attestation upload to OCI for quay.io/shbose/kaniko-chains@sha256:cb86f5e505413f593fb2a6d4997252b0d1ef3153f8263e67f23cd8d13f80cd06...","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"a20ac015-4148-462a-9689-11f39a43a794","knative.dev/key":"foo/kaniko-chains-run-8j48h"}
{"level":"error","ts":"2021-10-07T04:24:39.467Z","logger":"watcher","caller":"chains/signing.go:204","msg":"uploading: PUT https://quay.io/v2/shbose/kaniko-chains/manifests/sha256-cb86f5e505413f593fb2a6d4997252b0d1ef3153f8263e67f23cd8d13f80cd06.att: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"a20ac015-4148-462a-9689-11f39a43a794","knative.dev/key":"foo/kaniko-chains-run-8j48h","stacktrace":"github.com/tektoncd/chains/pkg/chains.(*TaskRunSigner).SignTaskRun\n\tgithub.com/tektoncd/chains/pkg/chains/signing.go:204\ngithub.com/tektoncd/chains/pkg/reconciler/taskrun.(*Reconciler).FinalizeKind\n\tgithub.com/tektoncd/chains/pkg/reconciler/taskrun/taskrun.go:61\ngithub.com/tektoncd/chains/pkg/reconciler/taskrun.(*Reconciler).ReconcileKind\n\tgithub.com/tektoncd/chains/pkg/reconciler/taskrun/taskrun.go:42...
{"level":"info","ts":"2021-10-07T04:24:39.805Z","logger":"watcher","caller":"chains/signing.go:214","msg":"Uploaded entry to https://rekor.sigstore.dev with index 740388","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"a20ac015-4148-462a-9689-11f39a43a794","knative.dev/key":"foo/kaniko-chains-run-8j48h"}
{"level":"error","ts":"2021-10-07T04:24:39.829Z","logger":"watcher","caller":"taskrun/reconciler.go:308","msg":"Returned an error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","knative.dev/traceid":"a20ac015-4148-462a-9689-11f39a43a794","knative.dev/key":"foo/kaniko-chains-run-8j48h","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* uploading: PUT https://quay.io/v2/shbose/kaniko-chains/manifests/sha256-cb86f5e505413f593fb2a6d4997252b0d1ef3153f8263e67f23cd8d13f80cd06.att: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]\n\n","stacktrace":"github.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:308\ngithub.com/tektoncd/chains/vendor...
{"level":"error","ts":"2021-10-07T04:24:39.829Z","logger":"watcher","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9bd7fb8","knative.dev/controller":"github.com.tektoncd.chains.pkg.reconciler.taskrun.Reconciler","knative.dev/kind":"tekton.dev.TaskRun","duration":3.566535488,"error":"1 error occurred:\n\t* uploading: PUT https://quay.io/v2/shbose/kaniko-chains/manifests/sha256-cb86f5e505413f593fb2a6d4997252b0d1ef3153f8263e67f23cd8d13f80cd06.att: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]\n\n","stacktrace":"github.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:566\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tgithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:543\ngithub.com/tektoncd/chains/vendor/knative.dev/pkg/controller.(*Impl).RunContext.func3\n\tgithub.com/tekto...
{"level

Steps to Reproduce the Problem

1. Setup & Configured Tekton Chains along with the signing key in tekton-chains/signing-secrets
2. Install the kaniko-chains Task
3. Created the secret out of the dockerconfig json as registration-credentials and linked it under secrets & pullSecrets in the pipeline sa which is used to run my TaskRuns.
4. Create the TaskRun
5. Observed that the image was built and pushed successfully https://quay.io/repository/shbose/kaniko-chains?tab=tags
6. Observed that the "chains.tekton.dev/signed:failed"

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  annotations:
    chains.tekton.dev/retries: "3"
    chains.tekton.dev/signed: failed
    chains.tekton.dev/transparency: https://rekor.sigstore.dev/740388

Additional Info

• Kubernetes version: v1.21.1+a620f50
• Tekton Pipeline version: version: v0.24.3

[sbose78]

... And now, tried the same with docker.io after setting up "v1" dockerconfigjson, push was successful for the signatures/attestations.

{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "..."
                }
        }
}

[priyawadhwa]

Hey @sbose78 thanks for opening this issue. I browsed the cosign repo and it looks like Quay will not be supported until Quay 3.6 is released. Check out this issue for more details: sigstore/cosign#40 (comment)

[sbose78]

Thank you @priyawadhwa for looking it up for me - should've done some looking around :)

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[priyawadhwa]

/remove-lifecycle stale

@sbose78 looks like the cosign issue was closed and support was added, does this work for you now?

[sbose78]

👋 @priyawadhwa , I will test this and let you know!

[sbose78]

Closing this for now, if this persists, I will re-open.

[davidkarlsen]

@sbose78 It still fails for me: https://github.com/evryfs/base-ubuntu/runs/5173488457?check_suite_focus=true

[concaf]

FWIW, pushing signatures and attestations to quay.io still fails with the error:
{"level":"info","ts":"2022-03-22T07:50:18.486Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"kaniko-chains-run-cmzrs\", UID:\"c4e0072d-13a2-4387-8079-6363f4b2062e\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"10180\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* PUT https://quay.io/v2/concaf/kaniko-chains/manifests/sha256-97bbad2fbfda26b2974a34d1a1f06514dc55ca114a6942022b3bb670f62e16d2.att: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'application/vnd.dsse.envelope.v1+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']\n\nFailed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:\n {'description': 'The MIME type of the referenced manifest',\n 'enum': ['application/vnd.oci.image.layer.v1.tar',\n 'application/vnd.oci.image.layer.v1.tar+gzip',\n 'application/vnd.oci.image.layer.v1.tar+zstd',\n 'application/vnd.oci.image.layer.nondistributable.v1.tar',\n 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',\n 'application/vnd.dev.cosign.simplesigning.v1+json',\n 'application/tar+gzip',\n 'application/vnd.cncf.helm.chart.content.v1.tar+gzip',\n 'application/vnd.oci.image.layer.v1.tar+gzip'],\n 'type': 'string'}\n\nOn instance['layers'][0]['mediaType']:\n 'application/vnd.dsse.envelope.v1+json']\n\n","commit":"843c6b3"}

The primary reason being that the media type application/vnd.dsse.envelope.v1+json is not yet supported by quay.io.

Issue reported here - https://issues.redhat.com/browse/PROJQUAY-3386