Add or change OIDC setting after a cluster is created

[elylee]

What would you like to be added:
Add or change OIDC setting after a cluster is created

Why is this needed:
"My organization may change our OIDC provider in near future. I don't want having to recreate my EKS-A cluster just to change the OIDC setting".

[smarsh-tim]

I found two work-arounds for this.

First option - Correct the EKS deployment to include OIDC

1. Scale down the eksa-controller-manager Deployment (this allows for editing the "immutable" field)
2. Login to the cluster and manually edit the Cluster spec to include the identityProviderRefs section

   apiVersion: anywhere.eks.amazonaws.com/v1alpha1
   kind: Cluster
   metadata:
     creationTimestamp: null
     name: <cluster_name>
   spec:
     ...
     identityProviderRefs:
     - kind: OIDCConfig
       name: "<name>"
   ...
   status: {}

3. Manually create a new resource in your Kubernetes cluster for OIDC

   ---
   apiVersion: anywhere.eks.amazonaws.com/v1alpha1
   kind: OIDCConfig
   metadata:
     annotations:
       anywhere.eks.amazonaws.com/control-plane: "true"
       anywhere.eks.amazonaws.com/etcd: "true"
     creationTimestamp: null
     name: "<name>"
   spec:
   ...

4. Make a change to your Control Plane machine config - such that the eksctl upgrade command will recreate those nodes

   ---
   apiVersion: anywhere.eks.amazonaws.com/v1alpha1
   kind: VSphereMachineConfig
   metadata:
     annotations:
       anywhere.eks.amazonaws.com/control-plane: "true"
     creationTimestamp: null
     name: "<name>"
   spec:
     memory: 7196 #8192 (original value)

5. Execute the upgrade command
6. Scale back up eksa-controller-manager Deployment if needed

Second option - manually correct the kube-apiserver pod one each control plane node
(this isn't preferable, as these settings will be lost in future upgrades"

1. SSH to each control plane node
2. Edit this file: /etc/kubernetes/manifests/kube-apiserver.yaml
3. Restart kubelet to apply the changes: 'systemctl restart kubelet'

---

I think the eksctl anywhere cli tooling / bootstrap process should be able to apply those changes. But it says that identityProviderRefs is an immutable field for the Cluster resource - which it isn't and doesn't seem to need to be.