✨ Add hetzner baremetal cluster stack

[chess-knight]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #108

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squash commits
• include documentation
• add unit tests

[chess-knight]

Hi @janiskemper. As discussed in the chat, I added you as the reviewer because you are an expert on Hetzner. This is the first version of pure Hetzner baremetal clusterstack(control plane and worker nodes in robot, loadbalancer in the hcloud). The manifests are mostly taken from CAPH(without machine health checks and remediation(missing namespace defaulting kubernetes-sigs/cluster-api#10805)), but I read that it is only for demonstration purposes. Maybe we can collaborate here, and tune this cluster stack by removing/adding/editing kubeadm settings or cluster variables, playing with cluster addons(e.g. cilium values are taken from CAPH, and CCM from syself/charts) and so on. Thanks

[janiskemper]

Thanks @chess-knight ! One thing you can maybe look at: The CCM we use still mainly is our fork under the syself organization. However, hetzner supports bare metal in the upstream one as well for some time now. You need to configure it in a certain way, but I believe that there should be some documentation about it. If not, you can surely open an issue and ask for it.

This might be a better option for SCS, since our fork usually lacks behind a bit ;)

Otherwise, I would really advice you to integrate custom remediation and MHC. There are a lot of cases where a reboot of the machine fixes it already, especially with bare metal.

[chess-knight]

Otherwise, I would really advice you to integrate custom remediation and MHC. There are a lot of cases where a reboot of the machine fixes it already, especially with bare metal.

Right now, the remediationTemplate cannot be added to the cluster stacks(clusterclass), because you need to know the namespace of HetznerBareMetalRemediationTemplate in advance. I fixed it upstream kubernetes-sigs/cluster-api#10843 but we have to wait for the merge and release.
Until the upstream patch, the user can specify MHC in Cluster.spec.topology in the same way(the user will know the namespace of HetznerBareMetalRemediationTemplate in advance).

EDIT: Namespace defaulting for the remediationTemplate will be available in CAPI v1.8.x

[chess-knight]

Thanks @chess-knight ! One thing you can maybe look at: The CCM we use still mainly is our fork under the syself organization. However, hetzner supports bare metal in the upstream one as well for some time now. You need to configure it in a certain way, but I believe that there should be some documentation about it. If not, you can surely open an issue and ask for it.

This might be a better option for SCS, since our fork usually lacks behind a bit ;)

Hi @janiskemper, I tried to replace Syself's hccm with Hetzner's but one thing doesn't work out of the box. CAPH is setting providerID field hcloud://bm-$SERVER_NUMBER for the machines and hccm is setting hrobot://$SERVER_NUMBER for the nodes. Then IMO CAPI cannot finish joining worker nodes, because the providerID field is different. E.g. it doesn't remove CABPK taint node.cluster.x-k8s.io/uninitialized:NoSchedule, see also hetznercloud/hcloud-cloud-controller-manager#611 which is maybe related. I am falling into this https://github.com/kubernetes-sigs/cluster-api/blob/v1.7.4/internal/controllers/machine/machine_controller_noderef.go#L80.

[janiskemper]

There is a config I believe. Someone from Hetzner tried this out and uses it. But I don't know where to find it

[chess-knight]

There is a config I believe. Someone from Hetzner tried this out and uses it. But I don't know where to find it

If I see correctly, everything is hardcoded:

• CAPH hardcodes it here https://github.com/syself/cluster-api-provider-hetzner/blob/main/pkg/services/baremetal/baremetal/baremetal.go#L895
• HCCM hardcodes it here https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/internal/providerid/providerid.go#L67

Right now the only way I see is to manually change it for all hetznerbaremetalmachines with e.g. KUBE_EDITOR="sed -i 's#hcloud://bm-#hrobot://#'" kubectl edit hetznerbaremetalmachine

[janiskemper]

search for "prefixRobotLegacy" in the providerid.go. I just had a quick look at the links you provided

[chess-knight]

search for "prefixRobotLegacy" in the providerid.go. I just had a quick look at the links you provided

Yeah I see, but it is there only for migration purposes and not used for the new nodes. Instead "prefixRobot" is used.

[janiskemper]

I don't know, sorry. I know that they got it running, but I don't know how exactly it works. I haven't tried. You can open an issue I guess and ask for a short guide to use the hcloud ccm with CAPH bare metal

[chess-knight]

Hi @janiskemper, can you recheck this PR, please?

[guettli]

@chess-knight I had a look at the changes. I have not tested it. But it looks good to me.