[RFE] Flatcar boot partition size shrink effort

[ader1990]

Current situation

Currently, the Flatcar boot partition is set to 128 MB. There was a slow increase over the years of the Linux kernel, dracut and systemd binaries or other required packages in the initrd. The current usage in the latest Alpha is of around 62MB, which is critically close to the half of the size.

From my limited understanding, if we pass the 64MB mark, there will be issues, as the kernel should fit twice for the A/B update to work.

Impact

Doubling the boot partition will limit the old clients to update. Maybe there are other implications, feel free to update the list.

Ideal future situation

Without increasing the boot partition, I see no way for the medium to long term to implement in an upstream clean way things like systemd upgrades, linux kernel upgrades or security features like LUKS encryption/decryption, HSM integration.

Additional comments

For example, the current upgrade to systemd 255 is blocked by this issue and the only way forward is to axe the kbd upstream package contents with the minimal necessary files. The problem is that if we implement such a hack, there are going to be unforseen issues in the future if a user wants to use the upstream kbd package features that we are removing. This solution will bring technical debt in the future, as the initrd required packages need to go through the same process.
flatcar/scripts#1679

The PR to add TPM/TANG support might be blocked too for the same reason: flatcar/scripts#1560.

If in the future, if we want to add LUKS2 support with HSM, there might be again, the same blocker.

[jepio]

Doubling the boot partition will limit the old clients to update.

We 100% need to keep supporting updates for old-clients.

[pothos]

From my observation it's often possible to cut things down a bit but yes, at some point we hit a limit. Without a limit one can easily bloat everything and even 512 MB is already too small for some distro's boot partition.

I mentioned this in the chat: A risky plan would be to repartition from the initrd: Shrink part7 to 1 MB or so and move 2-7, to grow part1 by ~64 MB.
However, we first should explore options in flatcar/scripts#1679 (comment) - I have some more hints in #1074 and #688

[ader1990]

It seems that the shim PR introduced a 0.5M increase in the initrd and the current state of the HEAD flatcar /boot is 64130 KB used vs 64909 KB free.
https://github.com/flatcar/scripts/pull/1589/files#diff-2b0f856f6376d8b6db23b7dbff406f70eb267961cb693d2cf83bfed3c0761de0R213

Which means that the current runway is ~400KB. I managed to sort out the systemd i18n module size increase problem, but the current 400KB runway is impossible for the systemd upgrade, be it 253 or 254 or 255.

Unless we find a solution to decrease the initrd size now, we cannot upgrade systemd. Even if I can squeeze the systemd upgrade to less than the 400KB current runway, the next initrd update would not be possible.

[gabriel-samfira]

This seems like a pretty difficult limitation to overcome.

What do you folks think about creating a different "channel" that would allow for larger defaults, without the need to support switching from one channel to another? I know this would add additional burden on support and maintenance, but it seems like we need a plan to transition to something that would allow Flatcar to evolve over time. The current disk space constraints seem to become a blocker for future updates.

[ader1990]

The next release window is coming next week, what would be the decision to be taken on this matter?
I see here a few options short/medium term:

• use various ways of compression to decrease the /boot partition size. Even if we compress with the best and slowest compression algos, as everything is already compressed, that would save a few 1 or 2 MB at most.
• cleanup the kernel config by removing some of the enabled modules. This is highly risky as people might be using those features / kernel modules and it will break backwards compatibility. Even is we remove some of the more exotic network/storage modules, this change won't matter on the bigger picture.
• reuse /usr in the initrd as @pothos suggested. Doing this might work, but at what cost? From what I see, the cost would be to diverge more and more from a systemd / gentoo upstream approach. Doing this in initrd means adding another level of complexity that has to be maintained.

All of this options can take weeks if not months to implement, but the current runway is very close to ending (~400KB of left space until 50% is reached on AMD64).

[ader1990]

We will need to revisit this issue soon enough, as the initial effort PR resulted in ~5MB of runway, but the recent systemd and tpm+tang PRs ate ~3MB, so we are left with around 2MB of runway.

[jepio]

Can you collect data on how systemd + tang ate 3MB?

[ader1990]

Can you collect data on how systemd + tang ate 3MB?

sure, I ll update flatcar/scripts#1731.

[pothos]

Here is the report against the previous nightly: image-diff.txt
And against the previous Alpha: image-diff-alpha.txt

[pothos]

Here is the report against the previous nightly: image-diff.txt And against the previous Alpha: image-diff-alpha.txt

Maybe tpm2 in the initrd could be loaded from /sysusr? Something also started to pull in libcurl.

[ader1990]

Linking #1397, which is part of the larger effort.

[ader1990]

FYI, the current AMD64 image is very close to reaching 50% /boot partition usage. Hopefully, this PR will give us some breathing room: flatcar/scripts#1955.

[chewi]

We're still delaying the inevitable though. I had a look at this for the first time, but I can't see any obvious gains, short of stripping bits out of the kernel, which we presumably don't want to do. I take it the OEM-CONFIG parition after the /boot partition is still used in some cases. Does it need all 64MB? That may be safer than trying to resize the ROOT partition at the end, and if we did want to take the space from ROOT, we'd need to move OEM-CONFIG anyway.

[ader1990]

Hello, while checking out the nmbl presentation https://pretalx.com/devconf-cz-2024/talk/W3AVCT/, also found out about this idea that might look promising in Flatcar's case too -> https://github.com/containers/initoverlayfs?tab=readme-ov-file#what-is-initoverlayfs. Sharing this here, maybe it is worth some investigation at a later time.

[chewi]

I mentioned on Matrix but forgot to say here that GRUB does support ext4, xfs, and btrfs these days, which probably means we could just move the whole kernel image into the usr partition, which should make managing them easier too. I was going to try this out.

[ader1990]

The gentoo profiles update PR brought a ~5% decrease flatcar/scripts#2287