[EPIC] SSO access to all clusters

[gawertm]

related #1432

Goal

We want to enable access to all our clusters (MC and WC) and applications (happa, grafana etc) using OIDC. As an access method for engineers interacting with clusters it has a lot of advantages. For example

• direct access to each cluster (not for example, needing to give access to an MC to grant access to WC)
• ease to log in again via refresh tokens
• SSO providers ease access management (e.g. management via groups, giving overviews on who has access)
• access can be revoked at any point and access can be easily reviewed via idp logs

Current state

We run dex-operator on each MC. The operator reconciles instances of dex-app. This includes the dex-app used to access the MC itself but also dex-app installed on workload clusters associated with the MC.
For each reconciled dex-app, the operator automatically creates connectors for the identity providers which are configured for the MC. Currently supported identity providers are azure ad and github.
The operator supports this for giantswarm and customer connectors. However, currently we do not use it for customers yet.

Every MC has dex configured but only a few WCs do. This means that we have automated SSO for giantswarm staff to all MCs via github and azure ad as well as the same access to a few WCs that have dex installed.

This is an old sketch of dex-operator (before it was implemented, so the design likely changed.)
Credentials for dex-operator are created/renewed via opsctl create dexconfig at the moment.

Desired state

We want automatic setup of SSO access to all clusters.
Right now, to enable OIDC access to a workload cluster, api server flags need to be configured. Also, RBAC configuration needs to be present. Furthermore, dex-app, athena, cert-manager, nginx-ingress-controller need to be installed.
We should be able to achieve all of these things via default app configuration.

OIDC access to management clusters is almost fully automated. However, we should improve it by integrating creation/renewal of dex-operator credentials into the bootstrap process.

make workload cluster sso the default for giant swarm staff

Preview Give feedback

1. rename default-dex-config to something else since we are getting rid of cluster namespaces #2395
   team/rainbow +1
   
2. Spec for workload cluster OIDC auth as the default auth method #1229
   team/bigmac +1
   
3. https://github.com/giantswarm/giantswarm/issues/20825
4. Improve workload cluster OIDC setup flow #2432
   team/bigmac +1
   
5. https://github.com/giantswarm/giantswarm/issues/24888
6. create auth bundle #2529
   5 of 5
   +0
7. Add workload cluster api server oidc settings by default #2530
   0 of 4
   team/shield +1
8. auth configmap merge fails at first
9. make auth bundle default app in CAPA #2658
   +0
10. support workload cluster SSO in opsctl login #2531
    +0

improve sso setup

Preview Give feedback

1. https://github.com/giantswarm/giantswarm/issues/27030
2. integrate dex operator credential creation/deletion in MC bootstrap process #2532
   team/bigmac +1
   
3. https://github.com/giantswarm/giantswarm/issues/26786
4. https://github.com/giantswarm/giantswarm/issues/23694
5. replace dex/athena apps with auth bundle #2644
   team/shield +1
6. how to deal with merging of default dex connectors, user connectors and secrets vs configmaps #1810
   team/shield +1
   

make workload cluster sso the default for customers

Preview Give feedback

1. Configure custom static clients in Dex #2359
   component/dex team/bigmac +2
   
2. include mc admin groups in auth bundle using rbac bootstrap #2610
   team/bigmac +1
   
3. Manage customers' dex instances using dex-operator #2486
   team/shield +1
4. include customer admin groups in auth bundle using rbac bootstrap #2987
   team/bigmac +1
   
5. add basic provider to dex-operator #2988
   team/bigmac +1
   

[gawertm]

we need to revisit this topic with Kubernetes v1.29 structured auth config as well as the Pinniped findings here #3453