Sync resources from control plane to Kyma Runtimes

[pbochynski]

Description
Provide a way for central components to push some resources to Kyma Runtimes. It can be a secret, deployment, or any other k8s resource. The resource can be pushed to the individual cluster or to the group of clusters (e.g. all internal clusters, all beta clusters).

Reasons

Several central components need to create some resources in managed Kyma Runtimes. Use cases:

• KEB installs secret for BTP service operator
• compass manager needs to propagate a secret for the runtime agent
  The NFS storage controller needs to create a storage class and persistent volume
• The warden needs to create a deployment and service account and role

Introducing a resource that contains a raw k8s manifest to install could be a generic solution. This way only one component would fetch the target cluster kubeconfig and use it to apply the resource. Together with the watcher component we can provide a tooling that will allow central components to avoid direct connection with thousands of Kyma Runtimes. As a result, we could achieve better performance (optimize the number of connections) and better security (fewer components have cluster-admin access to customer runtimes).

Acceptance Criteria

• Describe the [KCP Sync Resource] behaviour
  • Given an SKR Cluster
    • And a KCP Kyma CR is created on the KCP Cluster
    • And a Kubeconfig Secret is created on the KCP Cluster
    • And there is a Kubernetes Resource Manifest to be synced to the SKR Cluster
    • And there is a target SKR Cluster or a group of those determined
  • When the [KCP Sync Resource] is created on the KCP Cluster
    • Then the correct SKR Cluster group is determined as the sync target
      • And the Kubernetes Resource Manifest inside the [KCP Sync Resource] is synced to the SKR Cluster
      • And the [KCP Sync Resource] has a determined state
  • When the [KCP Sync Resource] is removed from the KCP Cluster
    • Then the Kubernetes Resource Manifest is removed from the SKR Cluster

[janmedrek]

We would need some kind of design for the [KCP Sync Resource] (name TBD). I believe it should be a separate CRD that would function similarly to the Manifest CR, it would have an inline k8s manifest inside though.

With a generic solution, we would be able to address the issue of Mandatory Modules as well, it would not require additional implementation on our side (i.e. Warden would be just a synced k8s manifest and not a module).

This would fulfil almost all of the Mandatory Module ACs, so it is worth considering at least.

[ruanxin]

A proposal for this KCP Sync Resource #991

[ruanxin]

A proposal for this resource sync setup design, I have a PR in kyma community as SDD document.

[a-thaler]

We learned today that the VPC Peering use case requires a sync in the opposite direction as well. @pbochynski could you formulate that in the description additionally to not make it transparent for all.

[ruanxin]

• Implement SyncResource #1364
• Implement ModuleConfig #1368