Machine pool nodes are not drained during upgrade

[bdehri]

Issue

Machine pool nodes are not drain during a cluster upgrade.

It's probably the root cause for #1993 .

[fiunchinho]

I did a bit of research on the topic. Draining of worker nodes when using AWSMachinePools is not currently implemented in CAPA. This is the issue from upstream AWSMachinePool graceful scale down.

This feature is blocked by this Graduation of EventBridge Feature. And this is the ADR explaining the proposal.

There was a PR to implement the feature but it was closed because of the mentioned "Graduation of EventBridge Feature".

I believe the final implementation would involve deploying this component in our clusters https://github.com/aws/aws-node-termination-handler

[fiunchinho]

There is also kubernetes-sigs/cluster-api-provider-aws#2023

[fiunchinho]

There is a k8s built-in mechanism that we could try https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/

The configuration flags needed to configure it (like ShutdownGracePeriod and ShutdownGracePeriodCriticalPods) are not available as kubelet flags though, so they can't be configured using kubeletExtraFlags. That means it wouldn't work with our current approach to configure the kubelet using kubeadm and the kubeletExtraFlags field. I opened this upstream kubernetes-sigs/cluster-api#8348

[mnitchev]

Regarding the above ShutdownGracePeriod. Kubeadm allows patching the Kubelet config (see here) but this was added in 1.25. I'll see if this can be done with a pre/post kubeadm command.

[mnitchev]

Released in cluster-aws@v0.30.0
A bit more info on how it works in the PR: giantswarm/cluster-aws#276