cmd-build: Enable composeFS signing

[jbtrystram]

This is a first draft trying to implement a signed composeFS build following the steps in https://ostreedev.github.io/ostree/composefs/#signatures

Right now the ostree container image deploy step fails with :
error: Reading composefs config: Loading composefs config: Invalid tri-state value: signed

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[jbtrystram]

Also, a lot of osbuild errors :

osbuild --out cache/osbuild/out --store cache/osbuild/store --cache-max-size 14GiB --checkpoint build --checkpoint tree --checkpoint raw-image --export qemu /tmp/osbuild-iZOS.json
starting /tmp/osbuild-iZOS.jsonPipeline source org.osbuild.curl: fc1e28ae605b7e156067d7b72378db65c3299bd47cbb03b421cdaacdfbf39389
Build
  root: <host>
source/org.osbuild.curl (org.osbuild.curl): Downloaded file:///srv/builds/41.20240529.dev.3/x86_64/fedora-coreos-41.20240529.dev.3-ostree.x86_64.ociarchive
Pipeline oci-archive: 9115482a124700da33defccad94c2364d971f2192f73bcb2be8ad237673de547
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.copy: 9115482a124700da33defccad94c2364d971f2192f73bcb2be8ad237673de547 {
  "paths": [
    {
      "from": "input://inlinefile/sha256:afe6db637fd8facb75b537667971889445ff28187a821ac9ea8e3947ae44a721",
      "to": "tree:///coreos.ociarchive"
    }
  ]
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
copying '/run/osbuild/inputs/inlinefile/sha256:afe6db637fd8facb75b537667971889445ff28187a821ac9ea8e3947ae44a721' -> '/run/osbuild/tree/coreos.ociarchive'

⏱  Duration: 0s
Pipeline tree: f14f2000667d22e56465b60d2d322badd744c1cefdf8fd480a06bee136b7b3ae
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
  source-epoch: Mon Aug  1 23:42:11 2022 [1659397331]
org.osbuild.ostree.init-fs: d37c3bfc74751b4637c7bdc0291ea10e9ad1c28aeb79f7ba8a80c248f8c59109 {}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
ostree admin init-fs --modern /run/osbuild/tree --sysroot=/run/osbuild/tree

⏱  Duration: 0s
org.osbuild.ostree.os-init: e5a44fc3d4aa10637ab34ad0d530c5afb60fe8d9a72eb10b4b3a074ac3d03f02 {
  "osname": "fedora-coreos"
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
ostree admin os-init fedora-coreos --sysroot=/run/osbuild/tree

⏱  Duration: 0s
org.osbuild.ostree.config: be6e6d2b67c7d9131d0629b156bc8c622b03bef453edd5b317f186e973330dfb {
  "repo": "/ostree/repo",
  "config": {
    "sysroot": {
      "readonly": true,
      "bootloader": "none",
      "bls-append-except-default": "grub_users=\"\"",
      "bootprefix": true
    }
  }
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
ostree config set sysroot.bootloader none --repo=/run/osbuild/tree/ostree/repo
ostree config set sysroot.bootprefix true --repo=/run/osbuild/tree/ostree/repo
ostree config set sysroot.readonly true --repo=/run/osbuild/tree/ostree/repo
ostree config set sysroot.bls-append-except-default grub_users="" --repo=/run/osbuild/tree/ostree/repo

⏱  Duration: 0s
org.osbuild.mkdir: f3ff87f9d85c7070245e6e337f64ef105a41b9273a8d8ee53007b72bee590e52 {
  "paths": [
    {
      "path": "/boot/efi",
      "mode": 493
    }
  ]
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system

⏱  Duration: 0s
org.osbuild.ignition: 72eea52dbb4d21546d5b753d142c774165f8e5f867f0af5495bef1d651fcf524 {}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system

⏱  Duration: 0s
org.osbuild.ostree.deploy.container: 6ee33cad679f22ed7a54ec28e696c1c8c2ba75f2b7835af611ce26cdc2d9bd58 {
  "osname": "fedora-coreos",
  "target_imgref": "ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:rawhide",
  "mounts": [
    "/boot",
    "/boot/efi"
  ],
  "kernel_opts": [
    "rw",
    "$ignition_firstboot",
    "mitigations=auto,nosmt"
  ]
}
input/images (org.osbuild.containers): target /srv/cache/osbuild/store/tmp/buildroot-tmp-0colqq85/inputs/images
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
ostree container image deploy --imgref=ostree-unverified-image:oci-archive:/tmp/tmphv_fb4k0/image --stateroot=fedora-coreos --target-imgref=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:rawhide --karg=rw --karg=$ignition_firstboot --karg=mitigations=auto,nosmt --sysroot=/run/osbuild/tree
error: Performing deployment: Deploying tree: Initializing deployment: Checking out deployment tree: Reading composefs config: Loading composefs config: Invalid tri-state value: signed
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 72, in <module>
    r = main(stage_args["tree"],
        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 67, in main
    ostree_container_deploy(tree, inputs, osname, target_imgref, kopts)
  File "/run/osbuild/bin/org.osbuild.ostree.deploy.container", line 41, in ostree_container_deploy
    ostree.cli("container", "image", "deploy",
  File "/run/osbuild/lib/osbuild/util/ostree.py", line 205, in cli
    return subprocess.run(["ostree"] + args,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ostree', 'container', 'image', 'deploy', '--imgref=ostree-unverified-image:oci-archive:/tmp/tmphv_fb4k0/image', '--stateroot=fedora-coreos', '--target-imgref=ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:rawhide', '--karg=rw', '--karg=$ignition_firstboot', '--karg=mitigations=auto,nosmt', '--sysroot=/run/osbuild/tree']' returned non-zero exit status 1.

⏱  Duration: 11s
manifest /tmp/osbuild-iZOS.json failed
Failed
+ rm -rf /srv/tmp/build.qemu/supermin.out /srv/tmp/build.qemu/supermin.prepare /srv/tmp/build.qemu/supermin.build
+ '[' '!' -f /srv/tmp/build.qemu/rc ']'
++ cat /srv/tmp/build.qemu/rc
+ rc=1
+ '[' -n '' ']'
+ return 1
+ rm -f /srv/builds/41.20240529.dev.3/x86_64/.qemu.building
fatal: failed buildextend-qemu
failed to execute cmd-build: exit status 1

[jbtrystram]

This should probably be

Suggested change

	echo $PUBKEY > ${workdir}/overrides/rootfs/etc/ostree/initramfs-root-binding.key
	mkdir -p ${workdir}/overrides/initramfs/etc/ostree
	echo $PUBKEY > ${workdir}/overrides/initramfs/etc/ostree/initramfs-root-binding.key

[cgwalters]

I think this would make sense to do after rebasing FCOS on bootc i.e. after coreos/fedora-coreos-tracker#1726 as that would help drive code and build system sharing more. I filed https://gitlab.com/fedora/bootc/tracker/-/issues/14 specifically related to this.

[jlebon]

See also discussions in https://gitlab.com/fedora/bootc/tracker/-/issues/2.

[jbtrystram]

edit : mistake on my side: I forgot to pop a git stash entry and was building with composeFS enabled on but not signed. I am unable to get the needed rpm-ostree change in a cosa container to make the build complete

After building rpm-ostree manually with a an ostree-rs-ext fix i was able to build and boot fedora coreOS rawhide with the composeFS signed.

I also set composefs: true in cosa's src/image-defaults for good measure, but I am not sure it's needed, as my previous experiments worked without.

Some further notes :
the resulted deployed system don't use the signature still :

• I can mount /dev/vda4 /sysroot --options remount,rw and change files just fine.
• Running ostree config set ex-integrity.composefs signed results in error: opening repo: Invalid tri-state value: signed