Enable out-of-tree Bottlerocket variants

[cbgbt]

What I’d Like:
A supported mechanism for creating custom Bottlerocket variants that live within their own git trees. These variants should support some mechanism for defining a dependency on Bottlerocket’s core in a way that allows the custom variant to evolve independently and consume Bottlerocket-core updates safely.

We've been calling the project to enable this "Out of Tree Builds" and shortening that to OOTB, so if you see references to either, they are a reference to this issue.

Background:
Currently a set of variants are maintained in the Bottlerocket tree, but any additional variants must be created by forking the repo and maintaining a series of patches which add the special variant. Some risks and challenges of this include:

• The custom variant repo must frequently rebase a substantial volume of code to stay atop Bottlerocket changes. These rebases are unpredictable and likely non-trivial in many cases.
• The integration points of custom variants are not well-defined. For example, changes to the semantic meaning of settings could impact out-of-tree variants in ways that are difficult to discern during rebase.
• The forked repo will include a significant amount of code/documentation/information which is not relevant to the specific changes needed for the custom variant.

Edit: We have started a new GitHub repository for a tool that will drive these builds. Track bottlerocket-os/twoliter#1 for the design.

[grosser]

similarly: if we need 1 custom package we also have to fork atm, so ideally the variant workflow would address this issue too
for example having a "build" repo that uses bottlerocket as a git-submodule and all the customization stays outside

[jfbette]

Coming back to this functionnality. On my side it is a must-have capability and seems complex for now to make custom Bottlerock variants.

On my side the identified customization needed could be listed as following :

• add/remove packages
• remove some functionnalities that are not really core system but more linked to AWS ecosystem.
• ability switch to other components doing same functionnality. E.g. switch from grub to direct EFI boot (this is an example, I know the impact on update process).
• specific compilation options (kernel, ...) or apply specific patches
• customize trademarks

For exemple I tried to create a new variant but even in the name of the variant their are some hard coded reference/tests to the name itself. For example, in early-boot-config, it tests if it is a metal variant and then apply some configuration. Being a metal should not be related to the name of the variant but rather a configuration variants

[webern]

@jfbette thank you for providing these requirements. We will be posting a design doc for consideration. We have created a new repository for a tool that will drive these builds, which we have given the name twoliter. Here is the issue to track for the design: bottlerocket-os/twoliter#1

[stockholmux]

@jfbette Could you give me a little more details and a further example on what you want by "customize trademarks"?

[soostdijck]

I came here via #1225 and #2570 looking for a way to install mount.nfs in bottlerocket. My requirement is that I want to use Netapp trident in my EKS Anywhere cluster (running bottlerocket).
As stated in those other issues I also ran into the same issues using trident on native EKS, using FSx ontap.

Netapp trident wants node preparation like this

[cbgbt]

I'm happy to say that we've accomplished this! 🎉 🚀 🎉

We still have a lot of documentation to write, but in short:

• The majority of Bottlerocket's OS components have moved to bottlerocket-os/bottlerocket-core-kit
• All existing Bottlerocket variants are now built "out-of-tree", using bottlerocket-os/twoliter to drive builds
• bottlerocket-core-kit can be expressed as a dependency in a twoliter workspace.

If you're seeing this and want to try creating a Bottlerocket out-of-tree variant but encounter issues or have questions, please feel free to cut us an issue! The most sensible repo for such issues would likely by:

• bottlerocket-os/twoliter if the issue pertains to build tooling
• bottlerocket-os/bottlerocket-core-kit if it pertains to any OS packages you are consuming in your variant

[webern]

🎉