Strategy for file verification (IMA, fs-verity, composefs)

[jlebon]

In Fedora 37, RPM content will be signed. This will be used in at least Fedora IoT in combination with IMA to verify executed files. rpm-ostree support for propagating these signatures has already been added.

There is also keylime which is a comprehensive trust system that includes leveraging IMA for runtime monitoring (and for which we're today already experimentally generating hashes).

libostree also supports fs-verity, but currently only usable for integrity checking since no signatures are checked yet. And finally, there's composefs, which IIUC can provide additional fs-verity support on top of this to cover directories.

And I'm sure there's more relevant things I forgot to add. This is a huge topic with a lot of different intersecting technologies, but the primary question is: are we interested in leveraging any of these technologies to strengthen our security?

[cgwalters]

I think the end-to-end flow we should support is:

• user builds custom derived FCOS as a container image
• their container build is signed (fs-verity/composefs/etc.) with their own key
• Either they use our disk images and pivot-on-boot, and the system is configured thereafter to use e.g. Secure Boot with a trust root limited to their own keys, or they generate a disk image from the container

[bgilbert]

That model is useful for advanced users, but requires a lot of buy-in to gain the benefit of file integrity: the user is customizing FCOS enough to do a custom derive, and cares enough to manage their own key (including possibly in their UEFI firmware settings). Should we also do something to help users running vanilla FCOS who are comfortable trusting Fedora's keys?

[lack]

I think there's a huge amount of value in having the Fedora IMA signatures in place by default and trusting the Fedora key to attest to the integrity, especially in downstream applications like RHCOS for OpenShift.

[erpalmerny]

I agree with @lack. Recent patches will allow additional CA keys (not replacement keys) to be loaded by way of the .machine keyring. These CA keys will certify keys that can be dynamically loaded into the .ima keyring and used by IMA to verify, for example, executables. This feature was required to support a dynamically loaded (not builtin) root of trust originating from the machine owner, in addition to the one from the distro. Both key hierarchies are required in certain environments.

[alexlarsson]

I'm not really involved in FCOS as such, but for the automotive work we're doing the goal is to use composefs + fs-verity to validate all reads (files as well as dirs) from the ostree commits. For automotive this is useful for security reasons, but also for safety reasons. I.e. we don't want cosmic rays flipping bits in the assisted driving software to continue running undetected which could risk lives.

[cgwalters]

I think just verifying RPM IMA signatures provides an additional level of security, but it is weak. My canonical example here is it'd prevent exploitation of the runc vulnerability. But...particularly for FCOS we actively encourage users to inject systemd units (and shell script binaries) that have no mechanism to be signed and are sitting there writable.

The above proposal of custom signed derived builds has much stronger security properties, because it means that custom systemd units are also signed - there's one mechanism to deploy code and configuration (containers), not two (ostree and ignition).

Also, a key advantage of composefs is that it's architected in such a way that it can be used as a backend for podman/kubelet too.

[alexlarsson]

Also, it seems like the current fedora IMA signatures are not using fs-verity signatures. This means to verify them you need to read the entire file from disk when the file is accesses to compare the hash, this is much more expensive than the merkle-hash style hashing that fs-verity does.

[travier]

This would benefit from a design document that we could refine progressively. ostreedev/ostree#2640 (comment) is a good start.

[travier]

Heavily intersect with ostreedev/ostree#2753 too

[travier]

Design discussion in ostreedev/ostree#2867

[cgwalters]

composefs PRs:

• Add support for composefs coreos-assembler#3483
• rdcore: Juggle physical root versus deployment root coreos-installer#1203
• Ignition tweaks to work with composefs fedora-coreos-config#2404

With these the "unsigned" path at least seems to work pretty well for me in some basic testing. We get nice stuff like mount -o remount,rw /usr no longer works. And you can also no longer chattr -i / etc.

(But, /sysroot is still there and mount -o remount,rw works on that. It's tempting to squirrel it away much more strongly...I'm thinking of something like having a tiny daemon process that owns it in a mount namespace or something? But in the end root on the machine can also just write to the raw block devices)

[bgilbert]

It looks as though composefs won't be going upstream.

[alexlarsson]

It looks as though composefs won't be going upstream.

composefs the kernel filesystem, no, but composefs the feature will.

[cgwalters]

It's confusing, because that's talking about the "full native kernel" implementation, which indeed: containers/composefs#138

But the new composefs which is being actively developed is a combination of erofs and patches to overlayfs (the latter of which are not all upstream yet, but we believe they will be).

[hsiangkao]

It looks as though composefs won't be going upstream.

Composefs use cases are actively developing, which I hope we could be in a collaborate way (hopefully). I could do the stuffs that in my own responsibilities for composefs cases.

[travier]

LWN articles: https://lwn.net/Articles/933616/ & https://lwn.net/Articles/934047/

[travier]

Related: https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_2