tools: need a tool to update/create manifests

[cyphar]

I've been playing around with skopeo+image-tools+#5+#8 and I've found that almost all of the necessary tooling we need is available. You can pull base images, create layers, validate layers, unpack layers and then run them. That's awesome.

However, the one missing piece is updating image manifests. This is the same as creating manifests (which is also missing). What we need is the ability to take an existing image manifest (or from scratch) change the different properties. While you could do this with some jq foo, since it's a spec thing it should live in the spec repository.

If no-one else is working on this, I'd be glad to help out. I'm thinking of a UX like

% oci-manifest new --cwd /new_cwd --layer baselayer --layer layer2 --layer layer3 >the_manifest.json
% oci-manifest update --cwd /new_cwd --layer layer4 <the_manifest.json >new_manifest.json

Management of history could be quite a pain but we could just implement it as an append operation, with the option to change the base. So something like this would allow you to change the base layer with oci-manifest update:

% oci-manifest update --base-layer baselayer

I'm not too sure, we can hash that out once I have an implementation. It might be nice to also extend this tool to handle manifest-lists. But that's for a future date, maybe we could implement it as part of oci-refs where we add features to deref a manifest list. Not sure.

[runcom]

I'm not too sure, we can hash that out once I have an implementation. It might be nice to also extend this tool to handle manifest-lists. But that's for a future date, maybe we could implement it as part of oci-refs where we add features to deref a manifest list. Not sure.

see https://github.com/estesp/manifest-tool (which could maybe be extended to work with non-list manifests) (also moby/moby#27455)

[cyphar]

Yeah, we could probably implement it as a config-based thing. It's probably not going to be that complicated to implement.

That reminds me, we need to have a way of handling descriptors for images because currently images are two-links-removed from the reference'd JSON.

[cyphar]

We also need something to handle descriptors. Descriptors appear very simple but are not. There's probably a case to be made for oci-gc as well, since oci-manifest will generate more than one blob.

[wking]

I agree that we'll want oci-gc eventually, but it seems orthogonal to this issue. And the only reason descriptors are complicated is remembering the associated media type (which you presumably knew when you pushed the blob into CAS). For folks building manifests in scripts, remembering those media types isn't complicated (just create descriptors at push time and insert them into the structure you're building (e.g. pushing into the manifest's layers). For folks building manifests on the command line, you can do something similar with jq:

$ MANIFEST='{}'
$ DIGEST=$(echo foo-bar | oci-cas put image)
$ SIZE=$(oci-cas get image ${DIGEST} | wc -c)
$ MANIFEST="$(echo "${MANIFEST}" | jq ".layers |= [{\"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\", \"size\": \"${SIZE}\", \"digest\": \"${DIGEST}\"}]")"
$ echo "${MANIFEST}"
{
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": "8",
      "digest": "sha256:5442d1bef41adc4dc3992b4d5736950092a96dea65b842813924033d5d930022"
    }
  ]
}

And if you had a tarball file available (vs. my echo foo-bar) you could get the file size with stat instead of fetching the data back out of CAS to pipe into wc. If we collect those three lines into a script, you could have:

$ MANIFEST="$(oci-manifest-add-layer.sh --media-type application/vnd.oci.image.layer.v1.tar+gzip image <(echo foo-bar))"

but I expect folks building images with image-tools commands will be comfortable without the wrapping shell, and that most folks will be using some higher-level tooling like Dockerfiles and such.

[cyphar]

@wking Yeah, my above review is the things I reckon is necessary for the OCI integration we're planning into KIWI (and by extension oboci which I'll implement in spare cycles). The idea is that I want to integrate the minimum amount of information about the spec into KIWI, so that the maintainers of KIWI don't have to handle all of the interplay between components. So I will end up writing a wrapper, I would just prefer that the wrapper be smaller rather than larger and that the majority of the non-problem-specific components are put inside image-tools as general purpose utilities.