CAPA cluster upgrades stability

[alex-dabija]

Story

-As a cluster admin, I want CAPA cluster upgrades to be seamless in order to provide the a good user experience to my customers.

Background

CAPA cluster upgrade have been an ongoing struggle.

Current problems

• kubeadm token not renewed on CAPA #1720;
• Pods become unauthorized during rolling of control plane nodes;
• Control plane nodes have no health checks configured;
• Machine pools have no health checks configured;
• Proxy environment variables are not injected if Kyverno is down, but the pods still start;

Tasks

• Investigate how service-account-issuer work #1766;
• Scale tornado #1742;
• Upgrade tornado to v0.20.1 from v0.18.0 #1753;
• Upgrade golem to v0.20.1 from v0.18.0 #1755;
• Upgrade thunder to v0.20.2 #1770;
• Upgrade grizzly to v0.20.2 from v0.18.0 #1756;
• Upgrade goat to v0.20.2 from a custom v0.18.0 version #1757;
• CAPA control plane machine health checks #1978;
• Remove silence from thunder / dev00 #1825;
• https://github.com/giantswarm/giantswarm/issues/25159;
• Refactor aws-network-topology-operator to put finalizer on the AWSCluster instead of Cluster #1827;
• Machine pool nodes are not drained during upgrade #2170;
• Machine pool nodes are not rolled during update #2217;
• Tag existing hosted zones #2111;

[alex-dabija]

Our biggest stability issue during upgrade is the fact that nodes are not rolled automatically: #2217 . This was intentionally done by upstream. We are working on a fix. See @AndiDog's comment: kubernetes-sigs/cluster-api-provider-aws#4071 (comment) .