Add k8s endpoint check to markNodeReady

[HomayoonAlimohammadi]

Summary

onStart hook happens before onBootstrap. because of this, on non-fresh machines (non-fresh == /etc/kubernetes/admin.conf is available) we use invalid/old admin.conf kubeconfig for csrsigining controller client. this PR makes sure that we prevent running controllers until we have the correct .conf files and we can reach the k8s cluster.

How to test

build and install k8s on a fresh machine, run bootstrap and check logs and confirm the csrsigning controller is running, e.g.:

Aug 21 08:28:20 test k8s.k8sd[1642]: I0821 08:28:20.897429    1642 controller/controller.go:181] "Starting Controller" logger="k8sd.csrsigning" controller="certificatesignin
grequest" controllerGroup="certificates.k8s.io" controllerKind="CertificateSigningRequest"
Aug 21 08:28:21 test k8s.k8sd[1642]: I0821 08:28:21.005667    1642 controller/controller.go:215] "Starting workers" logger="k8sd.csrsigning" controller="certificatesigningre
quest" controllerGroup="certificates.k8s.io" controllerKind="CertificateSigningRequest" worker count=1

also confirm that there are kubeconfigs available in /etc/kubernetes/, specifically admin.conf.
now remove the k8s snap and reinstall k8s (same snap). Run bootstrap and like above confirm that csrsigning controller is started and running.

[neoaggelos]

I do not think this is what we need here. The onStart hook runs every time k8sd starts. Will this not remove the kubeconfigs of a running cluster?

[HomayoonAlimohammadi]

right, I didn't think about that. what if we run the csrsigning controller in onBootstrap hook? @neoaggelos

[neoaggelos]

Perhaps an alternative would be to block the csrsigning controller starting before we can validate that we have a proper kubeconfig in place

[neoaggelos]

For example, we could return the rest config from here

k8s-snap/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go

Line 46 in 2994b87

if err == nil {

only if we can successfully call an endpoint (e.g. /readyz or a GetNode()), otherwise keep looping.

e.g. here is what we do on bootstrap (without the loop, as we don't want to keep using stale kubeconfigs)

k8s-snap/src/k8s/pkg/client/kubernetes/status.go

Lines 17 to 19 in db5015e

	// returning the error would abort, thus checking for nil
	endpoint, err := c.CoreV1().Endpoints("default").Get(ctx, "kubernetes", metav1.GetOptions{})
	return err == nil && endpoint != nil, nil

[neoaggelos]

Going in the right direction, but let's keep things simple rather than add layers on top of layers

[HomayoonAlimohammadi]

adding a centralized wait didn't seem to be possible. in order to mark the node as ready onStart should finish so we couldn't wait there. I think adding the k8s check on markNodeReady is the best way to go.

[bschimke95]

LGTM