Implement mutating webhook to inject node affinity for Kyma workload

[tobiscr]

Description

By introducing multiple worker pools, a mutating Webhook is required which ensures that workloads assigned to the namespace kyma-system are always use a node-selector which deploys them only on the Kyma worker pool (see #364 ).

The mutation webhook has to inject the nodeSelector field to all pods which are deployed into the kyma-system namespace.

AC:

• Certificates for the webhook can be generated with different approaches. Requirement is to rotate them regularly (max 90 day?). Investigate the different approaches in Kyma and decide which one we use for this webhook:
  • Create certificate in an init-container during bootstrap (like Otters are doing it)
  • Create certificate via crypto-library during bootstrap (like Goats are handling it, see code)
  • Use cert-manager for re-creating certificates (afaik Jellies are doing that)
  • Implement the certificate generation and ensure the Webhook is using only valid and automatically rotate certificate
• nodeSelector is injected by the webhook into all Kyma workloads to ensure they run only on the Kyma worker pool (see POC for multiple worker pools #364 for more details)
• Bundle the admission webhook as Kyma module to be deployed as obligatory by KLM (like Warden, see https://github.tools.sap/kyma/module-manifests/tree/main/modules/warden). Discuss in the architecture-round to bundle Warden + this mutating webhook in a kyma-internal module.
  • Depending on this decision, bundle the mutating webhook as Kyma module (either standalone module or combined module which includes also Warden).

Reasons

Ensure that Kyma workloads are only running on the Kyma worker pool.

Attachments