kubeadm token not renewed on CAPA

[alex-dabija]

Issue

kubeadm token is not renewed on CAPA and new nodes are unable to join the cluster. This happened on grizzly.

We are supposed to already have a fix for this, but for some reason it didn't work. We think it's probably because the cluster was created before the fix was rolled out, but it's just guessing.

The issue needs to be confirmed.

[bdehri]

@alex-dabija Does this also happen on WCs?

[alex-dabija]

@alex-dabija Does this also happen on WCs?

Yes, it should be reproducible on workload clusters because a management cluster is just a workload cluster with more applications running on it.

[tuladhar]

On grizzly MC, we tested scaling the control planes node for berk1 WC, and they were able to join the cluster.

puru@giantswarm (gs-grizzly-berk1-clientcert) ~/grizzly> kubectl get nodes
NAME                                          STATUS   ROLES                  AGE     VERSION
ip-10-31-120-92.eu-west-2.compute.internal    Ready    control-plane,master   91s     v1.23.13
ip-10-31-186-244.eu-west-2.compute.internal   Ready    control-plane,master   17m     v1.23.13
ip-10-31-220-52.eu-west-2.compute.internal    Ready    control-plane,master   2m57s   v1.23.13
ip-10-31-255-27.eu-west-2.compute.internal    Ready    control-plane,master   22m     v1.23.13
ip-10-31-83-82.eu-west-2.compute.internal     Ready    control-plane,master   19m     v1.23.13

[calvix]

@tuladhar the issue only occurs for MachinePool as for control-plane node a new kubeadmconfig is generated for each new machine and the token is always fresh

[bdehri]

We took a look at cluster-api repository with @tuladhar . Refreshment of the token secrets is dependent existence of the token secret itself which explains why we had problems with our Capa MCs at the moment. But I could not find what deletes the secrets in the first place. I have checked api-server audit logs but I could not found anything there.

[bdehri]

kubernetes-sigs/cluster-api#7717

[bdehri]

PR is merged, waiting for a release.

[bdehri]

It will be part of 1.4 release, custom kubeadm controller is deployed to all CAPA MCs with flux enabled.

[primeroz]

It will be part of 1.4 release, custom kubeadm controller is deployed to all CAPA MCs with flux enabled.

I think this was just released in 1.2.9 as a cherry pick if we want to move back to upstream image

• https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.2.9
• kubernetes-sigs/cluster-api@db9f478