Custom Addon Manifests

[shaneutt]

What would you like to be added:

Add the ability to place custom manifests in /kind/manifests/ that will be applied to the cluster similarly to how default-cni.yaml and default-storage.yaml are done now.

The simplest option would seem to be looking for a file named /kind/manifests/custom.yaml and applying it if present.

Why is this needed:

This would make it a bit simpler to build and distribute custom node images. I'm currently using kindest/node as a base image for building images that include specific addon content and using these images in integration tests for CI. It would make my life easier if I could just drop my addon manifests into /kind/manifests/ then having to apply them separately.

[shaneutt]

Note: if the maintainers are OK with adding this, I will most likely contribute this myself.

[BenTheElder]

What prevents you from applying the manifests yourself after creation? How would we deal with ordering concerns, templating, etc.?

The existing manifests are an undocumented internal implementation detail and altering them is not supported (instead we support disableDefaultCNI since it would be hard to remove at runtime and CNI providers sometimes test with kind).

I'm sure this feature is very easy to add. It also very difficult to remove and provide a stable API for. We've already struggled to avoid breaking changes in the contract with node images.

If you just want to mechanically distribute the files alongside the node image, I recommend simply:

FROM nodeimage
COPY custom.yaml /user/

Please do NOT place your custom assets under /kind, we make no guarantee about how we'll use the contents of this directory in the future.
EDIT: or in general we've made no guarantees about the contents of the node images, just that they can produce kind clusters.

[BenTheElder]

Previous addons issue: #253

We've been waiting for a mature option from SIG cluster lifecycle, we've been asked to not produce our own bespoke thing previously.

[shaneutt]

Ok, thank you for your considerations.