Cannot create FleetsMembers resources when using AzureASOManagedControlPlane

[markjgardner]

/kind bug

What steps did you take and what happened:
When attempting to create a cluster using AzureASOManagedControlPlane and register it with fleet using fleetsmembers.containerservice.azure.com resourcetype, reconciliation fails because the FleetsMembers resource requires an owner reference of type fleet.containerservice.azure.com. This CRD, however, is not registered as part of the CAPZ install on the management cluster.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: AzureASOManagedControlPlane
metadata:
  name: fleet-cluster1-aks
spec:
  resources:
    - apiVersion: containerservice.azure.com/v1api20210501
      kind: ManagedCluster
      metadata:
        name: fleet-cluster1-aks
      spec:
        location: westus3
        owner:
          name: myfleet-rg
        dnsPrefix: fleet-cluster1
        identity:
          type: SystemAssigned
    - apiversion: containerservice.azure.com/v1api20230315preview
      kind: Fleet
      metadata:
        name: hub-fleet
        annotations:
          serviceoperator.azure.com/reconcile-policy: "skip"
      spec:
        owner:
          name: hub-rg
    - apiVersion: containerservice.azure.com/v1api20230315preview
      kind: FleetsMember
      metadata:
        name: fleet-cluster1-aks
      spec:
        group: apps
        owner:
          name: hub-fleet
        clusterResourceReference:
          group: containerservice.azure.com
          kind: ManagedCluster
          name: fleet-cluster1-aks

failed to get owner: couldn't resolve reference hub-fleet, Group/Kind: containerservice.azure.com/Fleet: no matches for kind "Fleet" in version "containerservice.azure.com/v1api20230315previewstorage"

$> kubectl api-resources --api-group containerservice.azure.com
NAME                        SHORTNAMES   APIVERSION                                        NAMESPACED   KIND
fleetsmembers                            containerservice.azure.com/v1api20230315preview   true         FleetsMember
managedclusters                          containerservice.azure.com/v1api20210501          true         ManagedCluster
managedclustersagentpools                containerservice.azure.com/v1api20210501          true         ManagedClustersAgentPool

What did you expect to happen:
Reconciliation should have detected the existing hub and registered the cluster as a member.

Anything else you would like to add:
ASO Fleet resource docs

Environment:

• cluster-api-provider-azure version: v1.17.2
• Kubernetes version: v1.30.6
• OS (e.g. from /etc/os-release): AKSUbuntu-2204gen2containerd-202411.12.0

[markjgardner]

As a workaround, you must use the armId parameter for the owner reference.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: AzureASOManagedControlPlane
metadata:
  name: fleet-cluster1-aks
spec:
  resources:
    - apiVersion: containerservice.azure.com/v1api20210501
      kind: ManagedCluster
      metadata:
        name: fleet-cluster1-aks
      spec:
        location: westus3
        owner:
          name: myfleet-rg
        dnsPrefix: fleet-cluster1
        identity:
          type: SystemAssigned
    - apiVersion: containerservice.azure.com/v1api20230315preview
      kind: FleetsMember
      metadata:
        name: fleet-cluster1-aks
      spec:
        group: apps
        owner:
          armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/hub-rg/providers/Microsoft.ContainerService/fleets/hub-fleet
        clusterResourceReference:
          group: containerservice.azure.com
          kind: ManagedCluster
          name: fleet-cluster1-aks

[willie-yao]

As a part of the implementation of FleetsMembers in CAPZ, we defined the use case of having a pre-defined Fleet hub prior to creating a cluster as a FleetsMember. However with the ASOAPI, it seems like this use case should be supported as well. I'll take a look into this, but the workaround you provided is similar to the existing FleetsMember implementation in the old CAPZ API: #4316

[markjgardner]

I think just documenting that you have to use an ARM reference would be sufficient. I opened the bug because I couldn't find any documentation on how to reference the fleet hub without also having the hub registered with ASO. I had to export the CRDs and inspect the owner schema to figure it out. I agree that in most cases the hub will be created and managed outside of CAPZ.

[nojnhuh]

CAPZ intentionally does not configure ASO to install all CRDs by default: Azure/azure-service-operator#2920. Additional ASO CRDs can be specified with the ADDITIONAL_ASO_CRDS environment variable as described here: https://capz.sigs.k8s.io/topics/aso#installing-more-crds. That makes the type available so you can create a Fleet ASO resource and then refer to it by its Kubernetes name instead of its ARM ID in the AzureASOManagedControlPlane spec.