Adopt Garden Linux os extension due to removal of legacy configuration

[marwinski]

How to categorize this issue?

/area os
/kind enhancement
/priority 2
/os garden-linux

What would you like to be added:

In one of the next releases we would like to make some changes to Garden Linux which will affect the Garden Linux extension and the way Kubernetes is configured. In the past we have disabled features that caused incompatibilities with Kubernetes at that time. Those disabled features are:

1. nftables
2. cgroupv2
3. usage of systemd-resolved as resolver

We want to remove the code or configuration that disables these features. We would like to change that because it is legacy and in the meantime Kubernetes has provided support. The operating system extension should disable these feature if still required for older Kubernetes versions. These are the implications and required configuration changes for older clusters:

1. nftables

nftables is supported in Kubernetes since version 1.17, see gardenlinux/gardenlinux#193 for details. Support for some Gardener owned components might still be missing. To switch back to the iptables legacy implementation run the following commands during the bootstrap process:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
systemctl mask nftables.service

2. croupv2

cgroupv2 is supported by Kubernetes starting with Kubernetes 1.19. To switch to cgroupv1 add the following kernel parameter (in /etc/kernel/cmdline.d/80-cgroup.cfg) and reboot.

CMDLINE_LINUX="$CMDLINE_LINUX systemd.unified_cgroup_hierarchy=0" 

There appears to be no other way for Docker to change this.

3. resolv.conf

Garden Linux uses systemd-resolved which is a caching and validating DNS/DNSSEC stub resolver, as well as an LLMNR and MulticastDNS resolver and responder. The functionality is not used when /etc/resolv.conf is linked to /run/systemd/resolve/resolv.conf. For the local system it is better to link to /run/systemd/resolve/stub-resolv.conf.

The problem is that this configuration does not work with kubernetes as the local resolver (which listens on 127.0.0.53 is not routable from within pods. The kubelet has the configuration option to link a different resolv.conf file with the --resolv-conf option (e.g. the /run/systemd/resolve/resolv.conf can be used). Some infrastrucutres might require additional settings (see for example gardener/gardener-extension-provider-openstack#340).

Why is this needed:

see above

[marwinski]

This is an initial investigation which provided some insights. A Garden Linux build with cgroupv2 and nftables was used:

• Pods that use iptables in their own network namespace (as opposed to the root namespace) are not relevant. They can use whatever they desire. Of course, it would be better if they would migrate to nftables.
• A recent cluster (Kubernetes 1.22) appears to work just fine with nftables enabled as the default and cgroupv2. It appears that the detection mechanisms in kube-proxy and calico work just fine. There are no iptables-legacy entries on the host.
• The original vpn implementation works with its iptables entries in the vpn-shoot pod
• The reverse vpn implementation also works fine, although the iptables rules are in the seed which is not on cgroupv3 or nftables

As a result I'd propose the following for the os extension:

If the kubernetes version is < 1.19 ensure that iptables-legacy is configured and cgroupv2 is disabled (reboot required)

[MrBatschner]

The Garden Linux extension meanwhile supports cgroup v2 and configures kubelet and containerd to use the systemd cgroup driver if a cgroup v2 enable system was found.

As for the other options: changing OS specific options from the OS extension during node bootstrap requires a restart of a worker node (i.e. cgroup version, nftables backend). It was decided in several internal discussions that such a behaviour is not desirable as it might lead to unpredictable behaviour of Gardener provisioned worker nodes and as such can significantly increase day-2 troubleshooting amounts.

For the reasons given above, this issue is no longer relevant and I will therefore /close it.

[MrBatschner]

/close