Ignition tweaks to work with composefs

[alexlarsson]

I'm currently working on composefs use in ostree:
ostreedev/ostree#2640

I'm running into issues with ignition when testing it. Several places look at /sysroot and try to find the backing filesystem for it. This works because /sysroot is typically a bind-mount of some deploy directory in the real sysroot. However, in the composefs case, /sysroot is a composefs mount and doesn't have a real backing block device.

This change switches these cases to look at /sysroot/sysroot instead. This is the actual real sysroot after ostree-prepare-root did its work (and before pivot-rooting into it).

[cgwalters]

This is failing on

[   60.557632] coreos-boot-edit[2475]: Error: opening /sysroot/sysroot/etc/crypttab
[   60.558229] coreos-boot-edit[2475]: Caused by:
[   60.558625] coreos-boot-edit[2475]:     No such file or directory (os error 2)
[   60.559578] EXT4-fs (md127): unmounting filesystem 0e6352a8-c202-4517-b4ca-878d6275681

Which...right. Tricky. I think we may need to instead change the rootmap logic to know how to handle this?

[alexlarsson]

So, I have no idea how any of this really works, because I never looked at it before. However, it seems that rdcore rootmap really needs the actual "root" with etc and stuff, but it also assumes the root it has a block device, which is not true anymore (as it is an overlayfs mount).

I can't really think of any good way to get from the overlayfs mount back to whatever mount the erofs image in layer 1 of the overlayfs was stored on, so maybe what we need is to pass in boot the "real root" (i.e. /sysroot) and the "backing root fs dir" (i.e. /sysroot/sysroot) as arguments. The later can be optional, like:

rdcore rootmap /sysroot --root-mountdir /sysroot/sysroot --boot-mount ${bootmnt}

For the record, this is how a fully booted system looks with an overlayfs+erofs based composefs:

overlay / overlay ro,seclabel,relatime,lowerdir=/tmp/.composefs.CuUFPI:/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on 0 0
/dev/vda4 /etc xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota 0 0
/dev/vda4 /sysroot/ostree/deploy/fedora-coreos/var xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota 0 0

The erofs image is loopback mounted, initially at /tmp/.composefs.CuUFPI:

losetup 
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                                                                                  DIO LOG-SEC
/dev/loop0         0      0         1  1 /sysroot/ostree/deploy/fedora-coreos/deploy/2758a35a494c60ae0873b52ce9364c773377c56aa4f564c72a0c48801432cb65.0/.ostree.cfs   1    4096

But the actual mount is unmounted and lives now only as a private mount inside the overlayfs mount.

Hmm, I guess if we mount the overlay at something like /tmp/.composefs.loop0.XXXXXX, then we can use the loopback data to map back to the original file. Not super clean, but workable.

[alexlarsson]

Or i guess I could pass in a separate mountpoint argument for the erofs, which could be in the deploy dir, then I don't have to care about the loop part.

[alexlarsson]

Ok, I updated the composefs, and now the root mount looks like this:

overlay on / type overlay (ro,relatime,seclabel,lowerdir=/sysroot/ostree/deploy/fedora-coreos/deploy/443ae0cd86a7dd4c6f5486a2283471b3c8f76fc5dcc4766cf935faa24a9e3d34.0/.ostree.mnt:/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)

So, conceptually we could reverse map back to the deploy dir.

[alexlarsson]

Note that the directory is empty, there won't be any mount there anymore when things are running, so it will map to the right block device for the deploy dir.

[jlebon]

We should probably discuss this in coreos/fedora-coreos-tracker#1252 instead where we weigh the different possible approaches. Want to add a comment there? It sounds like you're just playing around with composefs on FCOS rather than proposing we support it, correct?

[alexlarsson]

My goal is to support composefs in ostree because we will need it on the automotive side, I'm just looking at FCOS as an easy way to test the ostree work. However, I still think it makes sense for fcos to use it.

[cgwalters]

OK, I've reduced the change here to just one for the rootfs size checking. The other bits for rdcore now require: coreos/coreos-installer#1203

I've verified that these two PRs in combination give PASS: coreos.boot-mirror.luks (87.29s) (at least with composefs, but CI here will also check without).

[cgwalters]

This one is pretty trivial and safe, any takers?

[alexlarsson]

It looks good to me, but then I was the initial author, and my knowledge of fcos is very basic, so I don't think I should really be acking this.