400 error when serving store containing multiple images

[nikkelma]

When collecting and serving multiple container images, sometimes an error of unexpected status: 400 Bad Request is thrown. Unsure if root cause is related to shared layers between images.

[nikkelma]

Works as expected:

hauler store add image quay.io/jetstack/cert-manager-controller:v1.6.1
hauler store serve

Throws error:

hauler store add image quay.io/jetstack/cert-manager-controller:v1.6.1
hauler store add image quay.io/jetstack/cert-manager-cainjector:v1.6.1
hauler store serve

Error message:

ERR failed commit on ref "manifest-quay.io/jetstack/cert-manager-cainjector:v1.6.1@sha256:f254e8b5a76c7bc3516364ccc7d4f4367d0b85147ebbe8d4a6ea1b40d024b066": unexpected status: 400 Bad Request

[nikkelma]

Hacky workaround found by @joshrwolf : repeatedly running the same hauler store serve command eventually gets rid of this error, and will not be encountered again if the registry folder is untouched.

[nikkelma]

Root cause as investigated by @joshrwolf : oras.Copy function, outside of hauler direct control... will link to relevant upstream issue when opened and confirmed to be related.

[nikkelma]

Using my fork of hauler to investigate this issue, discovered that the main cause appears to be shared layers.

The cert-manager-acmesolver image pushes layer dc34538f67ce001ae34667e7a528f5d7f1b7373b4c897cec96b54920a46cde65 and therefore

/docker/registry/v2/repositories/jetstack/cert-manager-acmesolver/_layers/sha256/dc34538f67ce001ae34667e7a528f5d7f1b7373b4c897cec96b54920a46cde65/link

exists (line 102 in attached log), but then later every other image's manifest upload fails with

time="2022-01-27T13:16:56-05:00" level=debug msg="filesystem.GetContent(\"/docker/registry/v2/repositories/jetstack/cert-manager-ctl/_layers/sha256/dc34538f67ce001ae34667e7a528f5d7f1b7373b4c897cec96b54920a46cde65/link\")" go.version=go1.17.3 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host="127.0.0.1:63716" http.request.id=31cbf6e3-b23a-4e7e-9743-7a10fc0a4b62 http.request.method=PUT http.request.remoteaddr="127.0.0.1:63719" http.request.uri=/v2/jetstack/cert-manager-ctl/manifests/v1.6.1 http.request.useragent=containerd/1.5.9+unknown trace.duration="30.692µs" trace.file=/Users/matt/go/pkg/mod/github.com/distribution/distribution/v3@v3.0.0-20211125133600-cc4627fc6e5f/registry/storage/driver/base/base.go trace.func="github.com/distribution/distribution/v3/registry/storage/driver/base.(*Base).GetContent" trace.id=145607aa-dbd7-4e15-97db-a365df610fef trace.line=95 vars.name=jetstack/cert-manager-ctl vars.reference=v1.6.1
...
time="2022-01-27T13:16:56-05:00" level=error msg="response completed with error" err.code="manifest blob unknown" err.detail="sha256:dc34538f67ce001ae34667e7a528f5d7f1b7373b4c897cec96b54920a46cde65" err.message="blob unknown to registry" go.version=go1.17.3 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host="127.0.0.1:63716" http.request.id=31cbf6e3-b23a-4e7e-9743-7a10fc0a4b62 http.request.method=PUT http.request.remoteaddr="127.0.0.1:63719" http.request.uri=/v2/jetstack/cert-manager-ctl/manifests/v1.6.1 http.request.useragent=containerd/1.5.9+unknown http.response.contenttype=application/json http.response.duration=1.250924ms http.response.status=400 http.response.written=319 vars.name=jetstack/cert-manager-ctl vars.reference=v1.6.1

Full log: debug.log

[zackbradys]

Closing as resolved!