Managing Kubernetes on Hetzner with Cluster API: Can't move management cluster

[RaicuRobert]

Problem

I've been trying multiple times to setup a cluster and move it buta all the steps work except step 5.
Sadly I can't figure out what is wrong.
I can't seem to be able to initialize the remote cluster as a manager because of the cert-manager (at least on the surface)

When executing

KUBECONFIG=hetzner-cluster-kubeconfig.yaml clusterctl init --core cluster-api --bootstrap kubeadm --control-plane kubeadm --infrastructure hetzner

I get stuck on

Fetching providers
Installing cert-manager version="v1.16.1"
Waiting for cert-manager to be available...
Error: context deadline exceeded

Info:

kubectl get pods -n cert-manager

cert-manager-cainjector-fb79858b4-87mhx   1/1     Running   0          61m
cert-manager-fbbb9fdd5-l9tgs              1/1     Running   0          61m
cert-manager-webhook-6cc5985dd5-rvklm     1/1     Running   0          61m

kubectl logs cert-manager-fbbb9fdd5-l9tgs -n cert-manager

I1123 06:54:56.309983       1 controller.go:284] "configured acme dns01 nameservers" logger="cert-manager.controller.build-context" nameservers=["10.96.0.10:53"]
W1123 06:54:56.310760       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1123 06:54:56.315431       1 controller.go:89] "enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]" logger="cert-manager.controller"
I1123 06:54:56.315519       1 controller.go:435] "serving insecurely as tls certificate data not provided" logger="cert-manager.controller"
I1123 06:54:56.315552       1 controller.go:102] "listening for insecure connections" logger="cert-manager.controller" address="0.0.0.0:9402"
I1123 06:54:56.316011       1 controller.go:178] "starting leader election" logger="cert-manager.controller"
I1123 06:54:56.316091       1 controller.go:127] "starting metrics server" logger="cert-manager.controller" address="[::]:9402"
I1123 06:54:56.316120       1 controller.go:171] "starting healthz server" logger="cert-manager.controller" address="[::]:9403"
I1123 06:54:56.319028       1 leaderelection.go:254] attempting to acquire leader lease kube-system/cert-manager-controller...
I1123 06:54:56.342910       1 leaderelection.go:268] successfully acquired lease kube-system/cert-manager-controller
I1123 06:54:56.343752       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-venafi"
I1123 06:54:56.352312       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-issuing"
I1123 06:54:56.363299       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-key-manager"
I1123 06:54:56.368524       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-acme"
I1123 06:54:56.369119       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-readiness"
I1123 06:54:56.371747       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="gateway-shim"
I1123 06:54:56.372696       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="challenges"
I1123 06:54:56.375137       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-acme"
I1123 06:54:56.378218       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-ca"
I1123 06:54:56.378239       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-selfsigned"
I1123 06:54:56.378462       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-vault"
I1123 06:54:56.381460       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="ingress-shim"
I1123 06:54:56.386612       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-metrics"
I1123 06:54:56.390332       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-request-manager"
I1123 06:54:56.393378       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-ca"
I1123 06:54:56.395561       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-selfsigned"
I1123 06:54:56.398288       1 controller.go:225] "not starting controller as it's disabled" logger="cert-manager.controller" controller="certificatesigningrequests-issuer-vault"
I1123 06:54:56.398646       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-issuer-venafi"
I1123 06:54:56.400828       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="clusterissuers"
I1123 06:54:56.403187       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificaterequests-approver"
I1123 06:54:56.405852       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-revision-manager"
I1123 06:54:56.408544       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="certificates-trigger"
I1123 06:54:56.420269       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="orders"
I1123 06:54:56.424266       1 controller.go:248] "starting controller" logger="cert-manager.controller" controller="issuers"
I1123 06:54:56.424708       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1123 06:54:56.424957       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1123 06:54:56.434900       1 reflector.go:368] Caches populated for *v1.Certificate from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.435282       1 reflector.go:368] Caches populated for *v1.CertificateRequest from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.435498       1 reflector.go:368] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.435716       1 reflector.go:368] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.436149       1 reflector.go:368] Caches populated for *v1.Ingress from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.445548       1 reflector.go:368] Caches populated for *v1.ClusterIssuer from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.446030       1 reflector.go:368] Caches populated for *v1.Challenge from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.446379       1 reflector.go:368] Caches populated for *v1.Issuer from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.446786       1 reflector.go:368] Caches populated for *v1.Secret from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.447247       1 reflector.go:368] Caches populated for *v1.Order from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:56.447602       1 reflector.go:368] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243

kubectl logs cert-manager-cainjector-fb79858b4-87mhx -n cert-manager

I1123 06:54:53.623094       1 setup.go:119] "Registering a reconciler for injectable" logger="cert-manager" kind="mutatingwebhookconfiguration"
I1123 06:54:53.625413       1 setup.go:119] "Registering a reconciler for injectable" logger="cert-manager" kind="validatingwebhookconfiguration"
I1123 06:54:53.625656       1 setup.go:119] "Registering a reconciler for injectable" logger="cert-manager" kind="apiservice"
I1123 06:54:53.627640       1 setup.go:119] "Registering a reconciler for injectable" logger="cert-manager" kind="customresourcedefinition"
I1123 06:54:53.627942       1 server.go:208] "Starting metrics server" logger="cert-manager.controller-runtime.metrics"
I1123 06:54:53.628443       1 server.go:247] "Serving metrics server" logger="cert-manager.controller-runtime.metrics" bindAddress="0.0.0.0:9402" secure=false
I1123 06:54:53.628375       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1123 06:54:53.628821       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1123 06:54:53.631614       1 reflector.go:368] Caches populated for *v1.MutatingWebhookConfiguration from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.632497       1 reflector.go:368] Caches populated for *v1.ValidatingWebhookConfiguration from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.635763       1 reflector.go:368] Caches populated for *v1.APIService from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.683719       1 reflector.go:368] Caches populated for *v1.CustomResourceDefinition from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.728869       1 leaderelection.go:254] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1123 06:54:53.750442       1 leaderelection.go:268] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1123 06:54:53.750619       1 recorder.go:104] "cert-manager-cainjector-fb79858b4-87mhx_09c04d6b-69c1-404c-81b6-8160c937a4b5 became leader" logger="cert-manager.events" type="Normal" object={"kind":"Lease","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"eee57aa2-7c74-4ffc-888d-6d9ab1f8664a","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1425"} reason="LeaderElection"
I1123 06:54:53.751782       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration" source="kind source: *v1.MutatingWebhookConfiguration"
I1123 06:54:53.751832       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.751852       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.751869       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration" source="kind source: *v1.Certificate"
I1123 06:54:53.751882       1 controller.go:183] "Starting Controller" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration"
I1123 06:54:53.752199       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService" source="kind source: *v1.APIService"
I1123 06:54:53.752225       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.752239       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.752251       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService" source="kind source: *v1.Certificate"
I1123 06:54:53.752263       1 controller.go:183] "Starting Controller" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService"
I1123 06:54:53.753076       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration" source="kind source: *v1.ValidatingWebhookConfiguration"
I1123 06:54:53.753255       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.753556       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.753706       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration" source="kind source: *v1.Certificate"
I1123 06:54:53.753833       1 controller.go:183] "Starting Controller" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration"
I1123 06:54:53.753569       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" source="kind source: *v1.CustomResourceDefinition"
I1123 06:54:53.754291       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.754427       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" source="kind source: *v1.PartialObjectMetadata"
I1123 06:54:53.754724       1 controller.go:175] "Starting EventSource" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" source="kind source: *v1.Certificate"
I1123 06:54:53.754882       1 controller.go:183] "Starting Controller" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition"
I1123 06:54:53.765766       1 reflector.go:368] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.813109       1 reflector.go:368] Caches populated for *v1.Certificate from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:53.862750       1 controller.go:217] "Starting workers" logger="cert-manager" controller="apiservice" controllerGroup="apiregistration.k8s.io" controllerKind="APIService" worker count=1        
I1123 06:54:53.863025       1 controller.go:217] "Starting workers" logger="cert-manager" controller="validatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="ValidatingWebhookConfiguration" worker count=1
I1123 06:54:53.863297       1 controller.go:217] "Starting workers" logger="cert-manager" controller="mutatingwebhookconfiguration" controllerGroup="admissionregistration.k8s.io" controllerKind="MutatingWebhookConfiguration" worker count=1
I1123 06:54:53.863420       1 controller.go:217] "Starting workers" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" worker count=1
E1123 06:54:53.879840       1 sources.go:183] "unable to fetch associated secret" err="secrets \"cert-manager-webhook-ca\" not found" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook" secret="cert-manager/cert-manager-webhook-ca"
I1123 06:54:53.880222       1 reconciler.go:117] "could not find any ca data in data source for target" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
E1123 06:54:53.880226       1 sources.go:183] "unable to fetch associated secret" err="secrets \"cert-manager-webhook-ca\" not found" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook" secret="cert-manager/cert-manager-webhook-ca"
I1123 06:54:53.880540       1 reconciler.go:117] "could not find any ca data in data source for target" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 06:54:55.331866       1 reflector.go:368] Caches populated for *v1.Certificate from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:59.306292       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 06:54:59.308403       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 06:54:59.316417       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 06:54:59.316462       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:05:07.523102       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:05:07.535145       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:05:07.603992       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:05:07.613016       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:54:21.311410       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:54:21.321132       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="mutatingwebhookconfiguration" kind="mutatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:54:21.366615       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"
I1123 07:54:21.376195       1 reconciler.go:141] "Updated object" logger="cert-manager" kind="validatingwebhookconfiguration" kind="validatingwebhookconfiguration" name="cert-manager-webhook"

kubectl logs cert-manager-webhook-6cc5985dd5-rvklm -n cert-manager

W1123 06:54:59.164648       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1123 06:54:59.168472       1 webhook.go:133] "using dynamic certificate generating using CA stored in Secret resource" logger="cert-manager.webhook.webhook" secret_namespace="cert-manager" secret_name="cert-manager-webhook-ca"
I1123 06:54:59.168514       1 webhook.go:144] "serving insecurely as tls certificate data not provided" logger="cert-manager.webhook.webhook"
I1123 06:54:59.169381       1 server.go:192] "listening for insecure healthz connections" logger="cert-manager.webhook" address=6080
I1123 06:54:59.169543       1 server.go:183] "Registering webhook" logger="cert-manager.controller-runtime.webhook" path="/mutate"
I1123 06:54:59.169612       1 server.go:183] "Registering webhook" logger="cert-manager.controller-runtime.webhook" path="/validate"
I1123 06:54:59.169680       1 server.go:208] "Starting metrics server" logger="cert-manager.controller-runtime.metrics"
I1123 06:54:59.169746       1 server.go:247] "Serving metrics server" logger="cert-manager.controller-runtime.metrics" bindAddress="0.0.0.0:9402" secure=false
I1123 06:54:59.169823       1 server.go:191] "Starting webhook server" logger="cert-manager.controller-runtime.webhook"
I1123 06:54:59.169893       1 server.go:242] "Serving webhook server" logger="cert-manager.controller-runtime.webhook" host="" port=10250
I1123 06:54:59.171655       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1123 06:54:59.172139       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
E1123 06:54:59.173342       1 dynamic_source.go:221] "Failed to generate serving certificate, retrying..." err="no tls.Certificate available yet, try again later" logger="cert-manager" interval="1s"       
I1123 06:54:59.196792       1 reflector.go:368] Caches populated for *v1.Secret from k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243
I1123 06:54:59.272725       1 authority.go:263] "Will regenerate CA" logger="cert-manager" reason="CA secret not found"
I1123 06:54:59.290289       1 authority.go:408] "Created new root CA Secret" logger="cert-manager"
I1123 06:54:59.292539       1 authority.go:283] "Detected change in CA secret data, update current CA data and notify watches" logger="cert-manager"
I1123 06:55:00.174745       1 dynamic_source.go:290] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
I1123 06:55:00.174804       1 dynamic_source.go:172] "Detected root CA rotation - regenerating serving certificates" logger="cert-manager"
I1123 06:55:00.178493       1 dynamic_source.go:290] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]

Suggested Fix