Add retries for kubeadm join / MarkControlPlane

[fabriziopandini]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version: v1.17.*

What happened?

While executing Cluster API tests, in some cases it was observed kubeadm join failures when adding the master label to the joining node.

xref kubernetes-sigs/cluster-api#2769

What you expected to happen?

To make mark control plane more resilient by adding a retry loop to this operation

How to reproduce it (as minimally and precisely as possible)?

This error happens only sometimes, most probably when there is a temporary blackout of the load balancer that sits in front of the API servers (HA proxy reloading his configuration).
Also, the error might happen when the new API server enters the load balancing pool but the underlying etcd member is not yet available due to slow network/slow I/O causing delays in etcd getting online or in some cases, also change fo the etcd leader.

Anything else we need to know?

Important: if possible the change should be kept as small and possible and backported

[RA489]

/assign

[neolit123]

@RA489 i'm going to take this ticket as i have some time later today and tomorrow.
/assign

[neolit123]

@fabriziopandini

kubeadm join --control-plane calls MarkControlPlane():
https://github.com/kubernetes/kubernetes/blob/1dc5235d0a93f4594402dfc0d7bcd5db88b3b4be/cmd/kubeadm/app/phases/markcontrolplane/markcontrolplane.go#L28-L44

PatchNode() already has retries:
https://github.com/kubernetes/kubernetes/blob/5708511499fe500ae3b4bbd40204cef382f652e8/cmd/kubeadm/app/util/apiclient/idempotency.go#L312-L319

APICallRetryInterval (500 ms)
https://github.com/kubernetes/kubernetes/blob/3d7847ed01ba7bc389f22e48704087750e659997/cmd/kubeadm/app/constants/constants.go#L179

PatchNodeTimeout (2 minutes):
https://github.com/kubernetes/kubernetes/blob/3d7847ed01ba7bc389f22e48704087750e659997/cmd/kubeadm/app/constants/constants.go#L182-L183

if 2 minutes is not sufficient what value would you recommend?
possibly doubling that to 4 minutes?

[fabriziopandini]

@neolit123 thanks for pointing this out.
I think that two minutes are ok unless we get some more reports about failures
/close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

@neolit123 thanks for pointing this out.
I think that two minutes are ok unless we get some more reports about failures
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.