Kubernetes clusters: define infraciadmin SVC account as code

[dduportal]

The goal of this issue is to set up the service account infraciadmin as code to allow the builds of jenkins-infra/kubernetes-management to administrate chart installations in the clusters.

Todo list (TBD)

---

• Postmortem: put "as code" the setup of the infraciadmin service account used to administrate clusters (instead of a hidden script on my laptop...)

Originally posted by @dduportal in #3582 (comment)

[dduportal]

• DigitalOcean done
  • feat(doks,doks-public) create infraciadmin SA for each cluster along with their token and kubeconfigs digitalocean#134
  • Required to add doctl in the docker-hashicorp Docker image (both in the terraform project and also the pipeline-library)
  • A terraform module is used, stored in jenkins-infra/shared-tools

Next steps:

• Try the resulting kubeconfig in infra.ci
• Azure (WiP)
• AWS

[dduportal]

Update:

• Tested with success the 2 DigitalOcean new kubeconfig with a manual credential import in infra.ci
  • For each cluster :
    • terraform output kubeconfig_<clustername> => copy and paste into a file named <clustername>/kubeconfig
    • Checked it works with these commands: kubectl --kubeconfig=./<clustername>/kubeconfig cluster-info (compare with and without the --kubeconfig flag that it changes) and kubectl --kubeconfig=./<clustername>/kubeconfig get ns (check the expecte namespaces for each cluster)
    • Update the existing credential in the kubernetes-jobs job on infra.ci by uploading the <clustername>/kubeconfig
  • Build https://infra.ci.jenkins.io/job/kubernetes-jobs/job/kubernetes-management/job/main/20950/ with sucess
  • Updated the SOPS secrets with the 2 kubeconfig
• Azure wip: refactor(publick8s,privatek8s) define infraciadmin SA with a terraform module azure#448
  • Imported existintg resources
  • Gotta merge anc check if it creates resources as expected

[dduportal]

Update:

• Azure finished and new kubeconfig credentials updated in SOPS and tested manually in infra.ci ✅
• WiP on AWS: refactor(cik8s,eks-public) define infraciadmin SA with a terraform module aws#440
  • Note: There is an AWS IAM "eks_charter" user added to the EKS Auth configmap : gotta check if we cannot clean it up if the switch to infraciadmin SA works without requiring AWS IAM auth.

[dduportal]

Update:

• AWS clusters done ✅
• All credentials in jenkins-infra/kubernetes-management are up to date with the SA from the config as code

Next steps: cleaning up former unused elements:

• cleanup: remove unused credentials kubernetes-management#4235 (credentials and cloud-specific CLI usage)
• jenkins-infra/aws: we'll have to remove the "*charter" IAM users - cleanup: remove *-charter IAM users in favor of kubernetes SAs aws#442
• No need to spend time to remove the cloud CLIs from jenkins-infra/docker-helmfile though as we should migrate to packer image (which should have all CLIs)