How do you enable features in the systemd core-kit package?

[mikn]

Now that I feel relatively confident that I can build our own packages, I realised one thing that we will need which I am not sure how to achieve.
We need to enable cryptsetup in systemd to be able to use systemd-cryptsetup to unlock the data volume of the server. This is for a metal setup (if you have not deduced already) and we want to protect the data volume from the simple threat vector of removing the disk from the server and reading it somewhere else for now.
The idea is to encrypt the volume as part of the install step of Bottlerocket (when we write out the partition layout), enroll the key into the TPM and have systemd-cryptsetup automatically unlock it on first Bottlerocket boot (it is fine to leave root and boot unencrypted as they are protected by dm-verity and contain only the general OS).

The long term goal is to use this in combination with remote attestation to only unlock it if the node can prove it is pristine, but this will require quite a bit more engineering and for now we will focus on minimally viable encryption. :)

I would prefer not having to fork the entire core-kit to achieve this, but I must say that I cannot figure out how to achieve this without doing so by just browsing the code.

One alternative would perhaps be to copy the entire systemd package and build only cryptsetup and name it specifically "systemd-cryptsetup"?

[mikn]

Success was achieved by:

• copying the systemd package to my own Twoliter repo
• packaging all new dependencies
• Bumping Release number in spec file
• enabling cryptsetup (and other dependent flags) on top of the already existing flags

The reason for this is because systemd is built with a shared library, and when I tried to compile only cryptsetup separately, the shared library did not contain the necessary code for cryptsetup. I tried briefly with static linking too, but did not manage to get that working either.

Was this the right approach?

[mikn]

@bcressey tpm2-tss-esys does compile linking against aws-lc 1.41.1 but not the FIPS version you linked.

kata-containers has a hard dependency on openssl also (their kata-agent rust binary uses the openssl backend) and I would prefer if I did not need to submit an upstream patch (which I think will not be super palatable) to use aws-lc-rs similar to what you are doing in Bottlerocket with your own tooling that you build.

For the 2.0.17 FIPS version, it seems it is missing some symbols (and HMAC support?)

  #14 4.453 make[1]: Leaving directory '/home/builder/rpmbuild/BUILD/tpm2-tss-4.1.3'
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c: In function ‘iesys_bn2binpad’:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:54:18: warning: implicit declaration of function ‘BN_num_bytes’ [-Wimplicit-function-declaration]
  #14 4.461    54 |     int len_bn = BN_num_bytes(bn);
  #14 4.461       |                  ^~~~~~~~~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:57:5: warning: implicit declaration of function ‘BN_bn2bin’ [-Wimplicit-function-declaration]
  #14 4.461    57 |     BN_bn2bin(bn, bin + offset);
  #14 4.461       |     ^~~~~~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c: In function ‘iesys_cryptossl_hmac_start’:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:399:47: error: ‘EVP_PKEY_HMAC’ undeclared (first use in this function); did you mean ‘EVP_PKEY_RSA2’?
  #14 4.461   399 |     if (!(hkey = EVP_PKEY_new_raw_private_key(EVP_PKEY_HMAC, NULL, key, size))) {
  #14 4.461       |                                               ^~~~~~~~~~~~~
  #14 4.461       |                                               EVP_PKEY_RSA2
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:399:47: note: each undeclared identifier is reported only once for each function it appears in
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c: In function ‘iesys_cryptossl_pk_encrypt’:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:675:15: warning: implicit declaration of function ‘BN_bin2bn’ [-Wimplicit-function-declaration]
  #14 4.461   675 |     if (!(n = BN_bin2bn(pub_tpm_key->publicArea.unique.rsa.buffer,
  #14 4.461       |               ^~~~~~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:675:13: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   675 |     if (!(n = BN_bin2bn(pub_tpm_key->publicArea.unique.rsa.buffer,
  #14 4.461       |             ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:688:17: warning: implicit declaration of function ‘BN_new’; did you mean ‘sk_new’? [-Wimplicit-function-declaration]
  #14 4.461   688 |     if (!(bne = BN_new())) {
  #14 4.461       |                 ^~~~~~
  #14 4.461       |                 sk_new
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:688:15: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   688 |     if (!(bne = BN_new())) {
  #14 4.461       |               ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:692:14: warning: implicit declaration of function ‘BN_set_word’ [-Wimplicit-function-declaration]
  #14 4.461   692 |     if (1 != BN_set_word(bne, exp)) {
  #14 4.461       |              ^~~~~~~~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:748:18: warning: implicit declaration of function ‘OPENSSL_strdup’ [-Wimplicit-function-declaration]
  #14 4.461   748 |     label_copy = OPENSSL_strdup(label);
  #14 4.461       |                  ^~~~~~~~~~~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:748:16: warning: assignment to ‘char *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   748 |     label_copy = OPENSSL_strdup(label);
  #14 4.461       |                ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:754:52: warning: pointer targets in passing argument 2 of ‘EVP_PKEY_CTX_set0_rsa_oaep_label’ differ in signedness [-Wpointer-sign]
  #14 4.461   754 |     if (1 != EVP_PKEY_CTX_set0_rsa_oaep_label(ctx, label_copy, strlen(label_copy)+1)) {
  #14 4.461       |                                                    ^~~~~~~~~~
  #14 4.461       |                                                    |
  #14 4.461       |                                                    char *
  #14 4.461 In file included from src/tss2-esys/esys_crypto_ossl.c:12:
  #14 4.461 /x86_64-bottlerocket-linux-gnu/sys-root/usr/include/openssl/evp.h:869:62: note: expected ‘uint8_t *’ {aka ‘unsigned char *’} but argument is of type ‘char *’
  #14 4.461   869 |                                                     uint8_t *label,
  #14 4.461       |                                                     ~~~~~~~~~^~~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:755:9: warning: implicit declaration of function ‘OPENSSL_free’ [-Wimplicit-function-declaration]
  #14 4.461   755 |         OPENSSL_free(label_copy);
  #14 4.461       |         ^~~~~~~~~~~~
  #14 4.461 In file included from src/tss2-esys/esys_crypto.h:13,
  #14 4.461                  from src/tss2-esys/esys_crypto_ossl.c:27:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:788:20: warning: implicit declaration of function ‘BN_free’; did you mean ‘sk_free’? [-Wimplicit-function-declaration]
  #14 4.461   788 |     OSSL_FREE(bne, BN);
  #14 4.461       |                    ^~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.h:17:44: note: in definition of macro ‘OSSL_FREE’
  #14 4.461    17 | #define OSSL_FREE(S,TYPE) if((S) != NULL) {TYPE##_free((void*) (S)); (S)=NULL;}
  #14 4.461       |                                            ^~~~
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c: In function ‘tpm_pub_to_ossl_pub’:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:819:16: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   819 |     if (!(bn_x = BN_bin2bn(&key->publicArea.unique.ecc.x.buffer[0],
  #14 4.461       |                ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:826:16: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   826 |     if (!(bn_y = BN_bin2bn(&key->publicArea.unique.ecc.y.buffer[0],
  #14 4.461       |                ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c: In function ‘iesys_cryptossl_get_ecdh_point’:
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:969:16: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   969 |     if (!(bn_x = BN_new())) {
  #14 4.461       |                ^
  #14 4.461 src/tss2-esys/esys_crypto_ossl.c:973:16: warning: assignment to ‘BIGNUM *’ {aka ‘struct bignum_st *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
  #14 4.461   973 |     if (!(bn_y = BN_new())) {
  #14 4.461       |                ^
  #14 4.461 make[1]: *** [Makefile:19509: src/tss2-esys/libtss2_esys_la-esys_crypto_ossl.lo] Error 1

I also tried compiling the FIPS aws-lc with the FIPS=1 flag, and got this error:

  #14 8.745 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.8EKgdr
  #14 8.745 -- The C compiler identification is GNU 11.4.0
  #14 8.745 -- Detecting C compiler ABI info
  #14 8.745 -- Detecting C compiler ABI info - done
  #14 8.745 -- Check for working C compiler: /usr/bin/x86_64-bottlerocket-linux-gnu-gcc - skipped
  #14 8.745 -- Detecting C compile features
  #14 8.745 -- Detecting C compile features - done
  #14 8.745 -- FIPS build mode configured
  #14 8.745 -- FIPS entropy source method configured: Passive
  #14 8.745 -- The CXX compiler identification is GNU 11.4.0
  #14 8.745 -- Detecting CXX compiler ABI info
  #14 8.745 -- Detecting CXX compiler ABI info - done
  #14 8.745 -- Check for working CXX compiler: /usr/bin/x86_64-bottlerocket-linux-gnu-g++ - skipped
  #14 8.745 -- Detecting CXX compile features
  #14 8.745 -- Detecting CXX compile features - done
  #14 8.745 -- Perl not found. Disabling some code generation and using pre-generated code in generated-src/
  #14 8.745 -- Run check_run file_to_test 'memcmp_invalid_stripped_check.c', flag_to_set 'MEMCMP_INVALID_STRIPPED', and compile_flags '-DNDEBUG'.
  #14 8.745 -- stdalign_check.c probe is positive, enabling AWS_LC_STDALIGN_AVAILABLE
  #14 8.745 -- builtin_swap_check.c probe is positive, enabling AWS_LC_BUILTIN_SWAP_SUPPORTED
  #14 8.745 -- linux_u32.c probe is positive, enabling AWS_LC_URANDOM_U32
  #14 8.745 -- Configuring incomplete, errors occurred!

Which doesn't even, as far as I understand contain, contain an actual error message.

[mikn]

Ah! Success! It links against: https://github.com/aws/aws-lc/releases/tag/AWS-LC-FIPS-3.0.0 which you released just the other week. Lucky us!
However building it with the FIPS=1 still results in the relatively unhelpful (at least to me) above output.

[mikn]

Ah, enabled debug and trace output - got the error message:

  #14 7.765 CMake Error at CMakeLists.txt:737 (message):
  #14 7.765   Building AWS-LC for FIPS requires Go and Perl
  #14 7.765 
  #14 7.765 
  #14 7.765    Called from: [1]	/home/builder/rpmbuild/BUILD/aws-lc-AWS-LC-FIPS-3.0.0/CMakeLists.txt
  #14 7.765 error: Bad exit status from /var/tmp/rpm-tmp.wZnalm (%build)
  #14 7.765     Bad exit status from /var/tmp/rpm-tmp.wZnalm (%build)

I think there is some truncation in the Docker build output as this should be output generally. I encountered this several times, especially when building this cmake project.

[mikn]

Yeah, ok. If I remove disabling perl and Go from the cmake configure step, it does pass with FIPS also.

[bcressey]

Is there a reason you have not yet updated to a newer systemd, by the way?

Some of the older variants still default to cgroup v1, which the systemd project has been warning about in the newer releases. My sense is it's still possible to run systemd in hybrid mode, up through last week's v257, but the testing required to validate that has been demotivating and hard to make time for while v252 is still getting stable releases.

The last of those older variants will go end-of-life in July 2025, so one way or another I expect an upgrade next year.

[mikn]

Summary:

• AWS-LC works as a libcrypto provider for tpm2-tss and cryptsetup (at least they compile)
• AWS-LC compiles using FIPS flags also, but I am unsure how to make it conditional - if you have recommendations, I am all ears!
• I would be very happy to hand over these dependencies to the Bottlerocket project.

The dependency tree looks like this:

systemd-cryptsetup
├── cryptsetup
│   └── libpopt
│   └── aws-lc-fips
├── device-mapper (both dmsetup and libdevmapper)
│   └── libaio
└── tpm2-tss
    └── aws-lc-fips

I do not think you need to enable the CLI of cryptsetup, so you could perhaps patch out the libpopt dependency, however, the systemd-cryptsetup target (i.e if you enable cryptsetup) also directly depends on libpopt for some reason (I have not explored the source code).

I also think there is some truncation happening of the output of RPM build errors, at least for CMake (it seems to eat stderr from the whole sections, %install, %files, etc), but I cannot prove it and do not have time to dig into it more.

[bcressey]

I also think there is some truncation happening of the output of RPM build errors, at least for CMake (it seems to eat stderr from the whole sections, %install, %files, etc), but I cannot prove it and do not have time to dig into it more.

I've noticed this with very verbose builds, and added this drop-in to force BuildKit to retain more logs:

# /etc/systemd/system/docker.service.d/001-env.conf
[Service]
Environment="BUILDKIT_STEP_LOG_MAX_SIZE=100000000"
Environment="BUILDKIT_STEP_LOG_MAX_SPEED=100000000"

[bcressey]

AWS-LC compiles using FIPS flags also, but I am unsure how to make it conditional - if you have recommendations, I am all ears!

Unless there's a reason, I wouldn't make it conditional - just use the AWS-LC-FIPS releases and build with FIPS=1.

[mikn]

Would you like a PR against the core-kit for this set of libraries?

[bcressey]

Would you like a PR against the core-kit for this set of libraries?

Yes please!