App provisioning framework

[L0czek]

App provisioning

This PR adds app provisioning framework to islet project which allows for installing applications inside realm using OCI containers. The framework components handle everything from downloading, verifying, installing and running the applications. It provides encrypted, persistent storage that each application can use to store confidential data. The realm also checks the authenticity of each application by verifying the application signature against a chain of provided certificates that begin at the embedded CA.

The framework consists of several main components:

• The realm image which implements a special realm with application manager which is responsible for installing and running applications.
• The warden daemon which is responsible for creating disks, network interfaces and finally running kvmtool
• The application registry server which is an OCI container repository. It utilizes the RA-TLS protocol to ensure that the realm is trustworthy before it allows the application to be installed.
• The realm metadata tool which is used to create the realm metadata block so that the sealing keys can be based on vendors public keys.
• The RIM measuring tool is a modified kvmtool which is capable of calculating realm's RIM without launching it.
• This PR which adds key derivation and realm metadata block functionality to enable sealing keys for encryption in realms.

Running the demo

The provided demo shows how an example application can be installed in a realm in islet. The detailed instruction goes over:

• setting up islet,
• preparing the special realm image,
• creating an example application image,
• signing the application,
• setting up the application registry,
• running islet and installing the application.

The detailed instruction to reproduce the app provisioning setup can be found here.

What works now

• Installing and running applications
• Applications are preserved across reboots
• Applications are authenticated based on ecdsa signatures
• Sealing keys can be measurements dependent, if the realm metadata block was not provided to allow for running any realms securely.
• In case the metadata block is present, the sealing keys are based on realm vendor public key to allow for easy application update (the encryption keys doesn't change between application and realms versions)

Known issues

RIM calculation related

Unfortunately, due to the design of how the RIM is calculated for realms, specifying a single RIM value for the application provisioning realm is not trivial. It is caused by measuring the Device Tree Blob which changes based on the kvmtool arguments. Basically, the realm RIM is dependent on such things as:

• virtual network card(s),
• virtio block devices,
• other virtual hardware,
• the order of the hardware in the DeviceTree.

This is especially troublesome as currently the warden daemon provides one disk image for each application. As a result, the kvmtool creates a separate virtio block device for each disk, leading to a change in the DTB. This should be investigated in the future to either stabilize the DTB or provide a rim measuring software. An example of such application is the modified kvmtool. Therefore, it will be further investigated in the future.

Something wrong with Islet rmm

While running the provisioning setup, we started encountering a truly random errors from various layers of the stack that could not be reproduced in tf-rmm. In detail, we encountered the following errors:

• TLS decrypt error during application installation,
• application image hash mismatch,
• some "inode already exists" error from the Linux kernel,
• disk image corruption (the file system wasn't preserved across realm reboots),
• TLS handshake error.

Since we couldn't reproduce them while running tf-rmm there might be some memory issue introduced by islet rmm. We first encountered this issue when we tried to test the application installation using a statically linked rust "hello world" application. It weights about 1MB, due to static linking and debug build. Every time we tried to install it while running islet, the installation process would fail with TLS decrypt error. We then switch to tf-rmm to check if the issue still persisted. To our surprise, it did not. In the next step, we created a lighter "hello world" application that weight only 1KB and managed to get it installed under Islet. This let us conclude that Islet introduces some memory instability that leads to arbitrary corruption when large amounts of data are processed. As of now, we weren't able to pinpoint the issue. You can try to reproduce this issue by trying under islet to:

• install the big application (example_app from the instruction)
• install the lighter one (the light_app from the instruction).

Naturally, this issue must be investigated, as this makes running applications under Islet unstable. Apart from that, #361 also suggest an issue with realm's memory management.

[p-sawicki2]

I've managed to provision and launch the application according to the description in the Readme file.
I've encountered some issues, probably related to memory corruption (some erros when parsing the configuration files), when using Islet RMM. Re-running the whole FVP without re-building artifacts solved the problem.

[p-sawicki2]

LGTM

[bokdeuk-jeong]

I nicely describes instructions to execute the app-provisioning example, but it looks hard to understand what it is doing. It could be even nicer if explanations about what each instructions is doning is added.

[bokdeuk-jeong]

We already have linux-ream under islet/third-party. This is duplicate work.
Why don't you make it built from islet?
By putting your own config under islet/third-party/linux-realm/ and modifying fvp-cca and related scripts to use linux config for app-provisioning, we can avoid building another realm-linux at realm-manager.

[L0czek]

Sure, would you like to just build the kernel with special config? This would leave us with the need to build the initramfs from realm-manager repo.

[jinbpark]

Could we get a more developer-friendly way (not islet developers, those who want to build apps but are not knowledgeable about CCA internals) to compile, install, and run a new application?? (it might be good for testing purpose in the beginning of app development)

For example, from the develop perspective,

1. Develop and compile a new application (call this binary light_app)
2. ./scripts/fvp-cca ..... --test_app=/path/to/light_app
3. fvp-cca takes all the complexities of installing and running this app. (e.g., signing, running a host daemon, ...)

I don't think it needs to be done in this PR. But this option would be nice for external developers.
(+ Once we get this PR merged, we'd better to migrate our examples (confidential-ml, e2ee) into this new provisioning scheme)

[jinbpark]

Do you have anything, in your mind, having to do with handling multiple applications?
I found in host_daemon_details document that it seems one realm has multiple applications provisioned by the realm daemon running inside the realm.

Do you assume that one device has only one realm daemon (i.e., one realm VM) and all applications should be provisioned into this realm? Or is it flexible?? (It would be flexible I guess. Each application might take up different realm VM)

One more thing I'm wondering is, is there any isolation mechanism between applications and the realm daemon?
I think the daemon is a typical linux process so the isolation between them is the same as what Linux offers to applications. Am I correct?

Would it be beneficial to employ a stronger isolation? e.g., the realm daemon is isolated from other applications and even underlying linux kernel, meaning that the realm daemon is secure even the kernel is compromised.
A new CCA feature, named CCA planes, can be used to achieve this isolation.

[L0czek]

@jinbpark

Could we get a more developer-friendly way (not islet developers, those who want to build apps but are not knowledgeable about CCA internals) to compile, install, and run a new application?? (it might be good for testing purpose in the beginning of app development)
[...]
I don't think it needs to be done in this PR. But this option would be nice for external developers.

Sure, we should redesign the build system to automate stuff. I just think this is a task for another PR as it would most likely require signifficant changes to the build system. We should focus now on reviewing and merging this PR and then later discuss how we want to keep various compenents and next remake the build system.

Do you have anything, in your mind, having to do with handling multiple applications?
I found in host_daemon_details document that it seems one realm has multiple applications provisioned by the realm daemon running inside the realm.
Do you assume that one device has only one realm daemon (i.e., one realm VM) and all applications should be provisioned into this realm? Or is it flexible?? (It would be flexible I guess. Each application might take up different realm VM)

Current design assumes that you can have multiple realms on a single device and a single realm can run multiple applications. Naturally, we are actively researching the security concerns resulting from running multiple applications inside a realm. For now we came to the conclusion with @p-sawicki2 that enforcing all aplications in a single realm to be from a single vendor is enough. This would eliminate potential security issues arising from adding an application to an existing realm that could have access to existing confidential data of other applications. We assume that applications from a single vendor will not try to hack themselves.

One more thing I'm wondering is, is there any isolation mechanism between applications and the realm daemon?
I think the daemon is a typical linux process so the isolation between them is the same as what Linux offers to applications. Am I correct?
Would it be beneficial to employ a stronger isolation? e.g., the realm daemon is isolated from other applications and even underlying linux kernel, meaning that the realm daemon is secure even the kernel is compromised.
A new CCA feature, named CCA planes, can be used to achieve this isolation.

Currently, there is no real isolation between applications and the realm daemon. It was a deliberate design choice due to:

• We wanted to provide a setup that will be as minimal as possible. Thus, adding any isloation would complicate the design and potentially affect application compatibility and performance.
• The realms already provide isloation as they are separate vms and each application or use case on the host side should use its own realm.
• You can always provision an OCI container with containerd and for ex. docker-compose and just start whatever application you want with isolation based on namespaces.
• After the applications were installed (before the are started) the realm daemon doesn't really store any secrets. For example, the AES encryption keys are kept in kernel keyring and thus cannot be read by any user application (there is a special key type for an "use only and no reading" key).

In conclusion, we tried to make this design as simple as possible and not including application isolation was a consciousness decision. Although, nothing is really stopping you from provisioning an OCI image which implements namespaces. In the future, we might look at the CCA planes feature but for now I think that namespaces are sufficient when isolation is needed.