Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ima: Support signing metadata objects and checkout fs tree validation #2609

Closed
cgwalters opened this issue Apr 30, 2022 · 36 comments
Closed

Comments

@cgwalters
Copy link
Member

cgwalters commented Apr 30, 2022

Today ostree's IMA signatures supports signing regular file objects. However, there are holes in this model; classically, IMA alone won't detect e.g. swapping two signed binaries, or replacing one signed binary with an older, insecure version etc.

However, we can implement a model where we do a "structural fsck" from the initramfs. Assuming that the initramfs is signed and verified (e.g. as part of a Secure Boot flow), then we would (from inside the initramfs):

  • Load the target commit object
  • Perform signapi (GPG/builtin ed25519) + content checksum verification of the target commit object or use IMA/fsverity for the commit object directly (it makes total sense to reuse IMA as another commit signature mechanism)
  • Perform "structural verification"; walk over the target physical rootfs and verify directories and symlinks but not regular file contents

Thinking of directories and symlinks as metadata and regular files as data - we're verifying all metadata up front, and then relying on the dynamic IMA (and future fsverity) signature for all the (much larger) regular files.

This was motivated by this sub-thread https://twitter.com/cgwalters/status/1520465975719645184 and this blog https://0pointer.net/blog/fitting-everything-together.html

Vs dm-verity in particular, I think what we're trading off here is really an echo of the larger picture of ostree-style updates (file based) versus block device updates. File updates are much, much more flexible and we can do things like deduplicate shared space trivially. There's not a "hard cap" on disk size beyond which the OS cannot grow, and similarly there's not "unused dead space".

Now that blog says:

I think validation is still at too high a level, since Linux file system developers made very clear their implementations are not robust to rogue images.

But yes, see e.g.: https://lwn.net/Articles/755593/

OTOH, as long as the machine has any persistent Linux filesystems it mounts and reads from the kernel, they could also be exploitable in a similar fashion. For "offline" AKA "evil maid" attacks, this is hopefully somewhat mitigated by dm-crypt; I suspect it'd be hard to corrupt the rootfs offline in such a way that it would be corrupted exploitably, but I am not aware of much research on this.

@alexlarsson
Copy link
Member

I feel like a one time verification is kinda limited. If you truly want to make this safe, then I think you would want a kernel based file system that mounted the commit directly from the repo. That way you will get rid of the hardlink farm with the vulnerable to modification directories.

@cgwalters
Copy link
Member Author

cgwalters commented May 10, 2022

The primary goal of this is to protect against:

  • offline/"evil maid" attacks - e.g. someone taking the ssd out of your laptop, injecting malware, and putting it back in. Similarly, in many IoT cases, the computer may be physically accessible to people who do not own it, and it's desired to harden against this.
  • "anti persistence" - ensuring that if malicious code gains root privileges on the machine, a reboot would clear it out (and one could do a security update that fixes the bug before starting the vulnerable app e.g. web browser, or before bluetooth is started, etc.)

There's 3rd and 4th related goals:

  • Enable "code signing enforcement" where the system can be configured such that all code executed as CAP_SYS_ADMIN must have a valid signature. Combined with things like trusted_for type hardening, this outright stops some exploit chains, and makes others much harder to exploit. It also helps protect against accidental damage (ssh'd to the wrong server), a not small problem.
  • Enable attestation systems like https://keylime.dev/ to be able to reuse extant IMA signatures

Of these, a one-shot boot time verification I think mainly addresses the first two - particularly anti-persistence. (There's a lot of details to the offline case though). I'd agree the "code-signing enforcement" case would in particular would be improved with always-on metadata verification.

@alexlarsson
Copy link
Member

I'm experimenting with ostreefs. Lets see how it works.

@alexlarsson
Copy link
Member

Initial work here: https://github.com/alexlarsson/ostreefs

@cgwalters
Copy link
Member Author

Initial work here: https://github.com/alexlarsson/ostreefs

OK, this is a big interesting topic. This clearly needs a README.md. I have a ton of thoughts on this; probably the biggest one for me is whether such a thing should really be ostree specific. Some of the design of ostree's data structures aren't bad, but it's not clear to me that we necessarily need to have the kernel parsing gvariants. I think there's likely a better, cleaner design for a data structure we could feed to the kernel that could also be shared with other userspace software rather than them having to create ostree gvariants.

There's overlap with https://github.com/giuseppe/composefs/ around this topic.

We need a decision on verification of regular files; I think that's fsverity or IMA.

Related to that topic, I've been increasingly feeling like the "lots of little metadata objects" is not what should canonically live on disk. If we have the approach of storing an entire commit/dirtree metadata as a single file, then we can also use fsverity on it, from userspace even before loading it to the kernel.

How serious are you about this effort? Is this a spike/PoC to test things out?

@alexlarsson
Copy link
Member

(CC @giuseppe)

So, this repo is just 2 days of work, it doesn't have any docs because I just basically made it work. However, it is also almost feature complete (got some issues with selinux xattrs). It was made mainly as a proof that this is possible, and to show that it can perform well in terms of performance and verification robustness.

I will spend a few more days on it to "productize" it, but its not like I've decided as "the way" things should work. Its just that I really like the elegant simplicity of both dm-verity and the ostree repo format, but I feel that a lot of these other solutions, like IMA policies and "verify at boot" are inelegant hacks that are hard to reason about (in terms of security).

As for composefs, I'm very well aware of that, and used it liberally for inspiration (ie. copied large chunks) when writing ostreefs. However, for this case I'm not sure that It is a better solution. If you want to keep the ostree repo ideas working more or less as is (rather than moving to a more image-like model) you'd then have to replace the ostree checkout phase with a "create a composefs for this commit" kind of operation. This step is unnecessary for ostreefs. Composefs also produces a larger file that has to be read in its entirety on each boot, even if not all of it is used immediately.

Another nice thing about ostreefs is that the entire directory/metadata structure is cryptographically sealed from the mount option (commit id). No data from disk is parsed or even looked at before its digest it has been recursively verified all the way back to the commit id. Composefs has no built in verification, but as you say, it could be verified with e.g. fs-verity. However, that doesn't protect against the composefs file being replaced with a different fs-verity:ed file. You would need to pre-compute the composefs file on the build host and put its fs-verity digest into the (signed) commit for this to be trustworthy.

As for the individual file validation, this is really a multi-part question. First of all. What kind of digest is used. The default IMA digest is a full-file hash, which requires you to read and hash each file in full each time you open it. This is not great for things like glibc, where maybe only parts are needed but its used a lot. I think the merkle tree of fs-verity is a clear winner here, as we only require validation of the parts being read as they are used.

The second question is how we should do the validation of the digest. It will eventually be possible to make IMA use fs-verify, which makes that one option. But even if that is possible, I feel like this is only part of the solution. How can we guarantee that each file in the object dir has the right IMA xattrs? Its feels like selinux style scattered xattrs and complex global rulesets are gonna make this hard to understand, reason about and verify.

A much simpler solution would be to use fs-verity for the regular files on disk, but somehow encode the fs-verity digest in the dirtree objects (or equivalent). Then ostreefs (or composefs/whateverfs) could verify this digest (possibly hashed with the stat data) before chaining to the backing file. This model is much easier to understand and reason about. Basically we tie everything cryptographically (like an image) to the commit id (which is generated at the build host), and don't rely on complex rules and marking individual files.

@giuseppe
Copy link
Member

I will spend a few more days on it to "productize" it, but its not like I've decided as "the way" things should work. Its just that I really like the elegant simplicity of both dm-verity and the ostree repo format, but I feel that a lot of these other solutions, like IMA policies and "verify at boot" are inelegant hacks that are hard to reason about (in terms of security).

I am not sure about "composefs" being "the way" as well, I've just used what maps better to the metadata used by eStargz and zstd:chunked. We could probably come up with a compromise since the long-term goal is to use the SHA checksum of the file payload (similar to what ostree does) instead of a file path. I like the idea of using fs-verity so we will have only to worry about the dir object.

@cgwalters
Copy link
Member Author

I just want to say thanks so much for diving into this! I certainly agree that a single pass verification at boot is suboptimal compared to continuous verification.

@cgwalters
Copy link
Member Author

somehow encode the fs-verity digest in the dirtree objects (or equivalent).

The ostree data model has no convenient space to add new things (kind of by design because extensibility and reproducibility can conflict). My offhand thought is that we can stick fsverity trees inside a magical ostree.fsverity xattr. This xattr would client side be unpacked and attached to file via ioctl(FS_IOC_ENABLE_VERITY). A tricky aspect here is that if we do this, then the fsverity tree itself becomes part of the ostree object sha256...which is not wrong, but then in order to have ostree fsck work we'd need to use FS_IOC_READ_VERITY_METADATA to get it back out (and that process needs to be bit-for-bit, which I think it is).

@cgwalters
Copy link
Member Author

cgwalters commented May 12, 2022

How can we guarantee that each file in the object dir has the right IMA xattrs?

Yes, this problem is related to why EVM was created on top of IMA; as we touch on in the ostree ima docs I think EVM is oriented towards machine-local signatures, but I think nothing blocks using EVM portable signatures. I just didn't get to testing it.

Personally...I wish fsverity had supported a model where the xattrs are also immutable and part of the signature, along with the file modes too. I honestly just think it's broken that one can make a fsverity file setuid i.e. chmod u+s for example without any problem. I guess we can try to push for adding this as an opt-in thing.

For now though my thinking is that we live with the model that IMA and fsverity provide around solely providing integrity for file contents, and think of everything else (directories, symlinks, file modes, xattrs) as metadata needs a separate integrity model. EDIT: Except, we shouldn't try to verify the hard link count, for example. I think it would make sense to include mtime/ctime etc. in the metadata too though (which for ostree, are always zero).

@alexlarsson
Copy link
Member

Ok, after some issues with kernel ASAN last week I'm almost ok with the current state of ostreefs in terms of matching current non-verifying behaviour. The one maintaining issues (other than more testing and docs) is that for some reason selinux refuses to read the selinux context from the xattrs.

However, I belive that is fixable, so I've been thinking a bit about the next step, continuous verification. This will be a two step thing. First we look at what we can do with what we currently have, and then step two is making minor changes to the ostree format to make this perform better.

For this, lets take a step back and reason from first principles. The chain of trust starts from the commit id on the mount commandline, and from this we can (cheaply) recursively verify the the contents of the .dirtree and .dirmeta file until we reach a regular file. All we know about this file is the file object id, which is computed like this:

header = file_header(uid, gid, mode, symlink_target, xattrs)
content = read_file()
object_id = sha256(concat(header, content))

So, lets say we're given such an object_id from a (verified) dirtree, and we want to verify that the file on disk matches this. We can easily reconstruct header, and if we do that we can keep using that data for the lifetime of the inode, so later changes on disk are unimportant. From this we can compute a object_id by reading the entire file and computing the hash.

However, such a verification is no guarantee that the file doesn't later change. But, assuming that when writing the repository we enabled fs-verity on the .file objects, then we could read back the verify checksum before doing the sha256 of the file contents. And given a verity csum if the sha256 hashes match, then we can trust this file, because the kernel makes it read-only and verifies on reads that it didn't change from the previously hashed content. We can even close the file as long as we store the verify digest in the in-memory inode and re-verify it next time we use the file.

This may sound lame, but actually it isn't really much worse than what non-fsverity IMA would give, as it also does a full-content validation on open. And this is without changing anything in ostree other than enabling fs-verity on the file objects in the repo.

So, next step is to add optional stuff to ostree to make this perfom better. Here we can't really put anything in the xattr of the file object, because we can't trust that. The only thing we trust is the dirtree, as that we verify its contents by sha256.

I propose that we add an ostree.alt_file_digests, which contains an array of sha256 digests, of the same length as the file array in the dirtree, and with matching order. Basically, if element N of the files list has a filename and a object_id, then element N of the alt-file-digests array has an alternative object_id for the file, which is computed like this:

header = file_header(uid, gid, mode, symlink_target, xattrs)
content = read_file()
alt_object_id = sha256(concat(header, fs-verity-checksum(content))

If this is available in the dirtree when looking up the child, then we have a trusted version of the verity checksum, and we can avoid the slow sha256 checksum of the entire file and just compare it with the on-disk verity checksum.

This should give us full performance, at a slight increase in dirtree size.

It is additionally interesting in this setup that we don't actually need to have the file attributes applied to the on-disk repository objects. Instead we can use bare-user mode repos and avoid e.g. setuid files on disk (they will only be setuid in the mount).

Opinions?

@cgwalters
Copy link
Member Author

So, next step is to add optional stuff to ostree to make this perfom better. Here we can't really put anything in the xattr of the file object, because we can't trust that.

Should it be optional though? I am sure for testing things out and a PoC, it's very useful to be able to directly run this against an existing ostree filesystem checkout. And I think this is really cool work to start out by showing we can validate structure so easily, particularly chaining to the commit hash via kargs - thanks again for starting this!.

But I think the PoC work and this discussion is showing us we may need to make changes. If we're not going to use the xattrs attached to the physical files, why put them there at all? And if we're talking about adding new metadata, then I think this argues for more of a "clean break" design where we create a new thing (a new checkout format and/or repository mode), and tooling which translates (ideally bidirectionally) from one to the other.

To flesh that out a bit, I think we can handle in-place updates from the existing format to a new layout by making it an opt-in thing attached to e.g. commit metadata. Or perhaps the presence of an ostree.verify karg. The deploy side would say "ok, we're using ostreefs" and use the new format for the filesystem checkout.

I'd reiterate that I think we should aim to have something usable for the OCI container ecosystem too - this would imply that we support capturing non-zero mtimes for example. (Though, to be efficient, mtimes should still be at least canonicalized in OCI to ensure reproducibility)

@alexlarsson
Copy link
Member

So, next step is to add optional stuff to ostree to make this perfom better. Here we can't really put anything in the xattr of the file object, because we can't trust that.

Should it be optional though? I am sure for testing things out and a PoC, it's very useful to be able to directly run this against an existing ostree filesystem checkout. And I think this is really cool work to start out by showing we can validate structure so easily, particularly chaining to the commit hash via kargs - thanks again for starting this!.

I'm gonna make it optional for now to get this code going. But you're right, long term we may want to just have a flag day kind of thing and require some new format.

But I think the PoC work and this discussion is showing us we may need to make changes. If we're not going to use the xattrs attached to the physical files, why put them there at all? And if we're talking about adding new metadata, then I think this argues for more of a "clean break" design where we create a new thing (a new checkout format and/or repository mode), and tooling which translates (ideally bidirectionally) from one to the other.

In the above model we would still apply file xattrs to the backing files for file objects. The only thing that ends up in the directory xattr metadata is the alternative object id/digest. However, I agree that this is technically not really necessary, and in fact slightly risky. A better approach is probably starting with a bare-user mode repo, where all the per-file metadata (uid,gid,mode,xattr) is in a single xattr, making it easy to read in and validate its hash.

To flesh that out a bit, I think we can handle in-place updates from the existing format to a new layout by making it an opt-in thing attached to e.g. commit metadata. Or perhaps the presence of an ostree.verify karg. The deploy side would say "ok, we're using ostreefs" and use the new format for the filesystem checkout.

I'd reiterate that I think we should aim to have something usable for the OCI container ecosystem too - this would imply that we support capturing non-zero mtimes for example. (Though, to be efficient, mtimes should still be at least canonicalized in OCI to ensure reproducibility)

I feel like it should be possible to implement this backwards compatibly by storing times in an optional xattr. I wonder what else kind of metadata OCI would require.

@alexlarsson
Copy link
Member

In the above model we would still apply file xattrs to the backing files for file objects. The only thing that ends up in the directory xattr metadata is the alternative object id/digest. However, I agree that this is technically not really necessary, and in fact slightly risky. A better approach is probably starting with a bare-user mode repo, where all the per-file metadata (uid,gid,mode,xattr) is in a single xattr, making it easy to read in and validate its hash.

Also, it seems it should be easy to "upgrade" a bare repo to bare-user repo by just iterating over all files and add the ostree.usermeta xattr. Right?

@cgwalters
Copy link
Member Author

Also, it seems it should be easy to "upgrade" a bare repo to bare-user repo by just iterating over all files and add the ostree.usermeta xattr. Right?

That'd cause ostree fsck to fail (and also in the general case break "reserialization", such as serving the local repo via a webserver, etc.). I guess the bare code path could learn to use that xattr if present though for reserialization, though fsck would likely want to verify both?

But, assuming reflink support, we can pretty cheaply convert in-place to whatever we want by creating a new inode that just shares storage.

@alexlarsson
Copy link
Member

But, assuming reflink support, we can pretty cheaply convert in-place to whatever we want by creating a new inode that just shares storage.

This is where the kernel supports hits us bad actually. fs-verity support is available on ext4 and btrfs, while reflink on xfs and btrfs, meaning the combined support is btrfs only. I wonder if we could use the fs-verity general code from the ostreefs code and use it ourselves to implement support independent of the backing filesystem.

@rhvgoyal
Copy link

Mike snitzer pointed me to this repo. This looks like very interesting piece of work. Trying to understand what does it do. Descriptions seems to suggest that it will verify checksums of the objects. I am wondering what makes sure that these checksums themselves are not modified.

@alexlarsson
Copy link
Member

@rhvgoyal The root of trust is the commit id, which is a sha256 digest. It gets passed in on the kernel command line. It is used to load the commit object (named by digest). The commit objects contains the digest of the root directory file, which we load and verify, it then contains references to other directory objects by digest, which are similarly verified.

Eventually we end up with a "leaf" regular file object. The exact way to verify those is what is being discussed now.

@rhvgoyal
Copy link

Oops, I think I might have commented on wrong thread. I wanted to ask this question in the context of ostreefs, instead. Sorry about that.

@alexlarsson
Copy link
Member

@rhvgoyal In the "regular" approach the directory metadata references the file by a digest of the metadata + file context. We can validate that, but it has weaknesses, see the README in the ostreefs repo.

@cgwalters
Copy link
Member Author

This is where the kernel supports hits us bad actually. fs-verity support is available on ext4 and btrfs, while reflink on xfs and btrfs, meaning the combined support is btrfs only. I wonder if we could use the fs-verity general code from the ostreefs code and use it ourselves to implement support independent of the backing filesystem.

Hmmmm. I think a key advantage of fsverity (AIUI) is that it's integrated with the underlying filesystem, and in particular compared to IMA this means that it's integrated with the page cache. See the last section in https://docs.kernel.org/filesystems/fsverity.html "Q: Why is anything filesystem-specific at all? Shouldn’t fs-verity be implemented entirely at the VFS level?"

@cgwalters cgwalters transferred this issue from ostreedev/ostree-rs-ext May 16, 2022
@cgwalters
Copy link
Member Author

Oops, I think I might have commented on wrong thread. I wanted to ask this question in the context of ostreefs, instead. Sorry about that.

It's a bit confusing because the code that is recently discussed here exists at https://github.com/alexlarsson/ostreefs - but the discussion is happening here because I filed this issue before that code was created, and further implementing it probably wants some changes in ostree.

(Alex, opinions on moving that repo into ostreedev/ GH organization?)

@alexlarsson
Copy link
Member

The ostreefs code is in a bit heavy churn atm, but yeah, i think we should move it to the ostreedev org once I've got the basics solid.

And in fact, the latest version I just pushed is pretty good to me. We now support for both bare and bare-user, and we verify on inode lookup all the metadata for directories and symlinks.

For regular files there is a mountoption fileverify that has three modes:

  • none: Don't verify file regular contents
  • once: Run the sha256 check on inode lookup. Doesn't handle later changes to the backing file.
  • full: Same as once, but we fail unless the backing inode has STATX_ATTR_VERITY attribute set. If this is enabled the single check will be right as the file is immutable.

I'm having some issues with testing the full mode though. First of all ostree pul" from a local repo didn't seem to trigger the fs-notify enabling for the imported files. When i run ostree commit it did, but it fails because it seems the centos kernel doesn't enable CONFIG_FS_VERITY. This is probably something we should raise with the kernel team.

I will test on a fedora kernel tomorrow.

Going forward the interesting changes is gonna require changes to the ostree format now.

@alexlarsson
Copy link
Member

Also, it seems it should be easy to "upgrade" a bare repo to bare-user repo by just iterating over all files and add the ostree.usermeta xattr. Right?

That'd cause ostree fsck to fail (and also in the general case break "reserialization", such as serving the local repo via a webserver, etc.). I guess the bare code path could learn to use that xattr if present though for reserialization, though fsck would likely want to verify both?

This wouldn't actually break though, assuming the entire conversion succeeds and we convert the repo type in the config. Then it will read the metadata from the new xattr.

However, I noticed that bare-user stores symlinks as regular files, so there are some other changes needed too.

@alexlarsson
Copy link
Member

@cgwalters So, should we move this to ostreedev/ ?

@alexlarsson
Copy link
Member

I'm looking at the optimization case now. To start with I wrote a userspace version of the fs-verity merkletree digest. Got that working now. We can use this from ostree to pre-compute verity checksums without needing specific filesystem support.

@alexlarsson
Copy link
Member

So. I've got code to compute fs-verity hashes for a file, so I can easily compute alternative digest for file objects which are based on fileheader + fs-verity-hash. However, I've been looking for a good way to inject these into the the metadata that is cheaply verifiable from the commit, which consists of the commit, dirmeta and dirtree objects. "Unfortunately", ostree is very strict about these, including things like enforcing normalized gvariants, so I was unable to find a good spot to inject these. The only two realistic places we can extend things are the commit object metadata and the directory entry xattr.

The first place I looked at was the directory entry attr. This is not ideal, as now the directory metadata depends on the contents, which is going to change as the files change. It also is practically quite nasty to implement given how commit works in terms of mutable trees. The directory metadata is serialized very early in the commit operation, before building the final directory structure. There are also issues with how this will work with checkouts of the commit. For a directory with 1000 files, the digest list will be 32k, and I'm not sure any filesystems can actually store xattrs of that size.

So, the next location is to put something in the commit metadata. This is rather expensive, as it would need to be a full mapping of file digest to fs-verity digest. I have an example, quite small (700M) cs9-based commit I'm testing things with. It has 16k files in it, so would need at least 1MB for a digest mapping table. This isn't really something we realistically can handle.

I've also thought about adding some generated dotfile in the actual commit with the mappings, but that has similar issues to the xattrs.

Basically, I'm having a hard time actually adding something to the commit metadata at all.

Then it struck me. We don't actually have to store anything in the commit. All the extra information we want to add to the commit is computable on the client from the information we have in the commit. So, we would define a virtual mapping from the set of files in the commit to the fs-verity manifests. The exact format isn't important as long as it is 100% reproducible. We generate such a mapping on the server just so that we can fs-verity hash it and store that hash in the commit. The client can then reconstruct this mapping locally, seal it with fs-verity, and the kernel filesystem can safely trust the mapping as long as it is fs-verity hashed and matches the hash stored in the commit.

Does that make sense?

@cgwalters
Copy link
Member Author

To start with I wrote a userspace version of the fs-verity merkletree digest. Got that working now.

Isn't that https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git/tree/include/libfsverity.h#n184 ?

@alexlarsson
Copy link
Member

To start with I wrote a userspace version of the fs-verity merkletree digest. Got that working now.

Isn't that https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git/tree/include/libfsverity.h#n184 ?

I was not aware of that, but having a more generic (GInputStream) implementation will make it easier to integrate with ostree (for example, handling archive mode repos with compressed files, etc), and its not really complex: https://gist.github.com/alexlarsson/765c7f2344c03d80514f0c20ee08560d

@cgwalters
Copy link
Member Author

@cgwalters So, should we move this to ostreedev/ ?

Yep fine by me! Though I do still feel ultimately this shouldn't be completely tied to ostree, we really want something like "veritytreefs" or so right? At least with support for mtimes...and in the general case perhaps everything OCI supports, such as sockets. But definitely good to have it in the ostreedev GH organization; should help with reviews, visibility, permissions etc.

I think all you need to do is this flow.

@cgwalters
Copy link
Member Author

The client can then reconstruct this mapping locally, seal it with fs-verity, and the kernel filesystem can safely trust the mapping as long as it is fs-verity hashed and matches the hash stored in the commit.

Yeah, I think that makes sense. It's basically the same as how the fsverity approach works for the content of files right? The tree is reconstructed on the client, the approach relies on the server and client computing the same thing.

@alexlarsson
Copy link
Member

@alexlarsson
Copy link
Member

@cgwalters So, should we move this to ostreedev/ ?

Yep fine by me! Though I do still feel ultimately this shouldn't be completely tied to ostree, we really want something like "veritytreefs" or so right? At least with support for mtimes...and in the general case perhaps everything OCI supports, such as sockets.

Yeah, I guess the question is if we want to do this by extending the ostree to optionally include this (via xattrs I guess), or if we would want to completely change things in a way that is incompatible with the current ostree format. For example, such a changed format could natively use the fs-verity hash for file contents, rather than plain sha256.

However, this would be a very large change that would invalidate large parts of the ecosystem around ostree. I think extending the ostree format for this would make more sense. The extensions needed are not that large, and then you can use the existing software and infrastructure around ostree.

@alexlarsson
Copy link
Member

What about this approach:

Instead of checking out a commit when we deploy it, we generate a composefs metadata file for the commit, using the ostree repo for file content (but not for directory and file metadata). This is done in a reproducible fashion, and includes the fs-verity checksums for the file contents. Then this file is mounted, instead of the checkout.

Additionally, during commit, ostree computes the same composefs metadata file and adds its digest to the commit. We can then verify that the computed composefs file matches what is in the commit.

@alexlarsson
Copy link
Member

For people who want to follow this, i've done a lot of work for this in composefs: https://github.com/giuseppe/composefs

@cgwalters
Copy link
Member Author

While this issue is about IMA, I don't think we're going to invest too much more in that beyond the basic support we have today. Going to duplicate this against #2867

@cgwalters cgwalters closed this as not planned Won't fix, can't repro, duplicate, stale May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants