CAPIProvider pre-delete hook is not working

[hardys]

What steps did you take and what happened?

• Install Turtles helm chart (I tested v0.11.0)
• Observe capiprovider CRs until all default providers are installed
• Uninstall the helm deployment

The uninstall completes without errors, and the job pod related to the pre-delete hook added in #508 did not have any errors, but the capiprovider resources are not deleted.

What did you expect to happen?

Based on #508 I expected the capiprovider resources to be purged on deletion, but it appears they remain indefinitely:

> kubectl get capiproviders -A
NAMESPACE                   NAME                 TYPE           PROVIDERNAME   INSTALLEDVERSION   PHASE
capi-system                 cluster-api          core           cluster-api                       Ready
rke2-bootstrap-system       rke2-bootstrap       bootstrap      rke2                              Ready
rke2-control-plane-system   rke2-control-plane   controlPlane   rke2                              Ready
> kubectl get capiproviders -A -o yaml | grep deletionTimestamp
    deletionTimestamp: "2024-08-29T14:30:02Z"
    deletionTimestamp: "2024-08-29T14:30:02Z"
    deletionTimestamp: "2024-08-29T14:30:02Z"

We can see the underlying providers are still present, with a deletionTimestamp set, but they won't delete due to the controller being removed and the CRs having a finalizer set.

kubectl get coreprovider,infrastructureprovider,controlplaneprovider -A
NAMESPACE     NAME                                                 INSTALLEDVERSION   READY
capi-system   coreprovider.operator.cluster.x-k8s.io/cluster-api                      

NAMESPACE                   NAME                                                  INSTALLEDVERSION   READY
rke2-control-plane-system   controlplaneprovider.operator.cluster.x-k8s.io/rke2                      

Removing the finalizer from the coreprovider means the delete of the corresponding capiprovider resource completes.

How to reproduce it?

• Install Turtles helm chart (I tested v0.11.0)
• Observe capiprovider CRs until all default providers are installed
• Uninstall the helm deployment
• kubectl get capiproviders -A

Rancher Turtles version

v0.11.0

Anything else you would like to add?

If we rework this deletion logic, it would be good to also address the issue mentioned in #627 - e.g if the pre-install hook fails we never install the capiprovider CRD, so it should be possible to fail gracefully in the pre-delete hook under those circumstances so installation may be retried (for example when deploying both Rancher and Turtles via the RKE2 helm controller, which doesn't enable control of ordering/dependencies so we must be able to retry e.g if Turtles tries to install before Rancher is installed and ready)

Label(s) to be applied

/kind bug