Make s390x and multi-arch fedora PeerPod images via mkosi

[huoqifeng]

mkosi has been used to leverage nix to build x86 PeerPod images, we need make it available for s390x (and maybe multi-arch). Initial investigation indicates that the nix OS does not support s390x, and mkosi does not support s390x natively either.

To mitigate the nix problem on s390x, we can use the mkosi ToolsTree=default since v16 feature defined in mkosi.conf to download latest tools automatically.

[Host]
ToolsTree=default

There is also a discussion thread to track the s390x support in mkosi.
So far, it's OK to build a kernel bootable fedora image for s390x with PR Support booting Fedora default image on s390x, to support a bootsable device, might need cooperate with mkosi and zipl community to make it happen. A compromise goal might be:

• In short term
  • We build a kernel bootable fedora s390x image via mkosi
  • We use a simple scripts/tool to make a bootable device/image by inject bootloader via zipl
• In longer term (not in this issues scope)
  • We ask/cooperate with mkosi and zipl to make the bootable s390x image native in mkosi

Mean while, I noticed that the binaries built via fedora builder has only x86, so we need build s390x and maybe other arches binaries via fedora also in this issue.

So, the proposed approach in this issue is:

• Build multi-arch PeerPod binaries via fedora builder
  • Either build s390x PeerPod binaries on an x86 fedora builder via gcc/rust cross platform build
  • Either build s390x PeerPod binaries on an simulated s390x fedora builder via qemu and docker buildx
  • Or, build s390x PeerPod binaries on a s390x fedora builder on s390x host
• Use mkosi ToolsTree=default to download necessary packages to build fedora PeerPod image and use it to replace nix
• Build kernel bootable fedora s390x PeerPod image
• Add a simple tool/scripts to make a bootable Fedora s390x device/image
• Considering the multi-arch framework to include other arches also.

[huoqifeng]

@katexochen @bpradipt @stevenhorsman @magowan @wainersm FYI.

[huoqifeng]

I'm facing an error when build s390x PeerPod binaries on fedora 39. Which seems related with cross platform compiler:

checking for s390x-gcc... s390x-linux-gnu-gcc
checking whether the C compiler works... no
configure: error: in `/tmp/build-libseccomp.gPHpT/libseccomp-2.5.4':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** [Makefile.inc:138: /src/cloud-api-adaptor/podvm/staticlib/kata-libseccomp/lib/libseccomp.a] Error 77

Whole logs looks like:

libseccomp-config-bad.txt

@bpradipt @wainersm @jensfr not sure whether you met this on RHEL or Fedora before?
Code I'm using is: huoqifeng@948f48d

[huoqifeng]

After run dnf install -y gcc-s390x-linux-gnu again in the build container (it's weird why need install 2nd time as its installed already via huoqifeng@948f48d), new error is:

configure:3187: s390x-linux-gnu-gcc -V >&5
s390x-linux-gnu-gcc: error: unrecognized command-line option '-V'
s390x-linux-gnu-gcc: fatal error: no input files
compilation terminated.
configure:3198: $? = 1
configure:3187: s390x-linux-gnu-gcc -qversion >&5
s390x-linux-gnu-gcc: error: unrecognized command-line option '-qversion'; did you mean '--version'?
s390x-linux-gnu-gcc: fatal error: no input files
compilation terminated.
configure:3198: $? = 1
configure:3218: checking whether the C compiler works
configure:3240: s390x-linux-gnu-gcc -O2   conftest.c  >&5
/usr/bin/s390x-linux-gnu-ld: cannot find crt1.o: No such file or directory
/usr/bin/s390x-linux-gnu-ld: cannot find crti.o: No such file or directory
/usr/bin/s390x-linux-gnu-ld: cannot find -lc: No such file or directory
/usr/bin/s390x-linux-gnu-ld: cannot find crtn.o: No such file or directory
collect2: error: ld returned 1 exit status

Full log:
libseccomp-config-bad2.txt

[huoqifeng]

This is quite a similar bug https://bugzilla.redhat.com/show_bug.cgi?id=1622375 although not same exactly.

[huoqifeng]

Sounds like ubuntu is the good choice for cross platform build but not for all other distros: dotnet/source-build#2839 (comment)