Exploration on MCS API

[allenlsy]

MultiClusterService (MCS) API is a project in Kubernetes sig-multicluster. It defines a set of Kubernetes CRDs describe the policies in a multicluster environment.

This doc briefly summarizes the mechanism of MCS API and it's potential relationship with OSM. Here we focus on non-headless services. For headless services, the scenarios are discussed in detail in the original KEP doc.

New CRD

There are two new CRDs introduced:

• ServiceExport (SE): in a cluster, SE declares which service are exposed to remote clusters
• ServiceImport (SI): in a cluster, SI declares which service it has access to on remote clusters

Main components and Terminologies

• ClusterSet: a set of clusters that form the multicluster environment. It is assumed that cross cluster requests are routable in the network.
• ClusterSetIP: an exported service should have a static IP to be accessible in the ClusterSet.
• MCS controller: it manages the SE and SI resources. It is the responsibility of the MCS API user to implement a MCS controller.
• Optional DNS: for uniformed service discovery mechanism, especially for resolving address to headless services.

MultiClusterService

A MCS is a service that shared within the whole clusterset. It is required that it has the same name and namespace on different clusters. Its FQDN should be <svc>.<ns>.svc.clusterset.local by default.

Typical Use Cases

1. Exporting a Service

This operation exposes a local cluster service to the remote clusters.

1. Create a SE resource on the local cluster
2. mcs-controller notices the SE resource and derives the SI resources within mcs-controller, which associates the endpoints from the serving cluster to the export service.
3. mcs-controller will update the SI resources in each consuming cluster.

2. Importing a Service

This operation is a consuming cluster asks the access to services on the remote clusters.

1. Create a SI resource on the consuming cluster
2. If the requested service is exported in the ClusterSet, mcs-controller will send the available service endpoints to the consuming cluster via EndpointSlice resources.

Considerations

• RBAC rules should be created on operating SE and SI resources
• For an exported service, if all its SE instances are deleted, should we delete all its related SI resources on all importing clusters as well? KEP doc recommends to delete them.

OSM with MCS API

Among the discussions in sig-multicluster, MCS API and KubeFed are two projects standing out. In a recent interview with Jeremy Olmsted-Thompson, a co-chair of sig-multicluster, he mentioned that the user group of KubeFed is getting smaller. MCS API is the more active one between the two.

OSM does not need to implement the MCS API itself, as it is a Kubernetes orchestration feature rather than a service mesh feature. But when OSM works towards a multicluster mesh, it will be based on an existing multicluster Kubernetes environment. In the near future, MCS API is likely to be the winner. Therefore, OSM should consider developing the multicluster mesh features on top of MCS API.

Some existing MCS API implementations can be used to set up the development environment:

• https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s
• https://github.com/submariner-io/submariner

These two projects both use CRDs from the multicluster.x-k8s.io group, which is the experimental version of the MCS API.

References

• Spotlight on SIG Multicluster

[keithmattix]

Thanks for the write up! So one thing I'm wondering about is the maturity/availability of various mcs-controllers. My understanding is that the MCS SIG doesn't provide one: each Kubernetes service provider has to provide one themselves; is that correct? If so, are we pigeonholing ourselves by putting our chips behind the MCS API since we'd only be offering multicluster to a subset of possible users?

[allenlsy]

Right, MCS API does not provide the mcs-controller. We should use one of the MCS API implementations above for development. From their API group naming x-k8s, I think they are not validated for production use yet

[allenlsy]

A blocking issue found while setting up multicluster env using AKS and Submariner. I have created 1 broker cluster and 2 worker clusters and tried to connect workers to broker. Installing the broker with subctl works fine. But once I started joining working clusters, some components in kube-system namespace start to crash. Here is a piece of log.

│ I0729 22:44:15.164812       1 request_send.go:116] Sending request - REQUEST_NUMBER: 0xe286954a71edfcb6, SENDER: "submariner-cable-c2-10-224-0-4", RECEIVER: "submariner-cable-c1-10-224-0-4", USING_SRC: 10.224.0.4:4490, USING_DST: 10 │
│ I0729 22:44:15.165804       1 request_handle.go:53] Received request from 10.224.0.4:4490 - REQUEST_NUMBER: 0xe286954a71edfcb6, SENDER: "submariner-cable-c2-10-224-0-4", RECEIVER: "submariner-cable-c1-10-224-0-4"                     │
│ W0729 22:44:15.165901       1 request_handle.go:57] Received NAT discovery packet for cluster "c1", but we are cluster "c2"                                                                                                              │
│ I0729 22:44:15.165942       1 request_handle.go:118] Sending response to 10.224.0.4:4490 - REQUEST_NUMBER: 0xe286954a71edfcb6, RESPONSE: UNKNOWN_DST_CLUSTER, SENDER: "submariner-cable-c2-10-224-0-4", RECEIVER: "submariner-cable-c2-1 │
│ I0729 22:44:15.166121       1 response_handle.go:32] Received response from 10.224.0.4:4490 - REQUEST_NUMBER: 0xe286954a71edfcb6, RESPONSE: UNKNOWN_DST_CLUSTER, SENDER: "submariner-cable-c2-10-224-0-4", RECEIVER: "submariner-cable-c │
│ E0729 22:44:15.166201       1 listener.go:83] Error handling message from address 10.224.0.4:4490: remote endpoint "submariner-cable-c2-10-224-0-4" responded with "UNKNOWN_DST_CLUSTER" : &proto.SubmarinerNATDiscoveryResponse{state:i │
│ 00000000  1a 6a 08 b6 f9 b7 8f a7  a9 a5 c3 e2 01 10 02 1a  |.j..............|                                                                                                                                                           │
│ 00000010  24 0a 02 63 32 12 1e 73  75 62 6d 61 72 69 6e 65  |$..c2..submarine|                                                                                                                                                           │
│ 00000020  72 2d 63 61 62 6c 65 2d  63 32 2d 31 30 2d 32 32  |r-cable-c2-10-22|                                                                                                                                                           │
│ 00000030  34 2d 30 2d 34 22 24 0a  02 63 32 12 1e 73 75 62  |4-0-4"$..c2..sub|                                                                                                                                                           │
│ 00000040  6d 61 72 69 6e 65 72 2d  63 61 62 6c 65 2d 63 32  |mariner-cable-c2|                                                                                                                                                           │
│ 00000050  2d 31 30 2d 32 32 34 2d  30 2d 34 42 0f 0a 0a 31  |-10-224-0-4B...1|                                                                                                                                                           │
│ 00000060  30 2e 32 32 34 2e 30 2e  34 10 8a 23              |0.224.0.4..#|                                                                                                                                                               │
│ I0729 22:44:15.166279       1 request_send.go:116] Sending request - REQUEST_NUMBER: 0xe286954a71edfcb7, SENDER: "submariner-cable-c2-10-224-0-4", RECEIVER: "submariner-cable-c1-10-224-0-4", USING_SRC: 10.224.0.4:4490, USING_DST: 52 │
│ W0729 22:44:17.167662       1 natdiscovery.go:182] NAT discovery for endpoint "submariner-cable-c1-10-224-0-4" has timed out                                                                                                             │
│ I0729 22:44:17.167689       1 remote_endpoint.go:122] using NAT legacy settings for endpoint "submariner-cable-c1-10-224-0-4", using public IP "52.226.248.158"                                                                          │
│ I0729 22:44:17.167733       1 cableengine.go:193] Installing Endpoint cable "submariner-cable-c1-10-224-0-4"                                                                                                                             │
│ 002 listening for IKE messages                                                                                                                                                                                                           │
│ 002 adding UDP interface vx-submariner 240.224.0.4:500                                                                                                                                                                                   │
│ 002 adding UDP interface vx-submariner 240.224.0.4:4500                                                                                                                                                                                  │
│ 002 forgetting secrets                                                                                                                                                                                                                   │
│ 002 loading secrets from "/etc/ipsec.secrets"                                                                                                                                                                                            │
│ 002 loading secrets from "/etc/ipsec.d/submariner.secrets"

Worker clusters are stuck in connecting state when running subctl show connections.

I suppose that Submariner should update the config of coredns, but the coredns configmap is not updated with more entries.

I noticed the change of coredns-autoscaler configmap:

Data
====
ladder:
----
{"coresToReplicas":[[1,2],[512,3],[1024,4],[2048,5]],"nodesToReplicas":[[1,2],[8,3],[16,4],[32,5]]}

Crashing components:

• coredns-autoscaler
• metrics-server

[jsturtevant]

I suppose that Submariner should update the config of coredns, but the coredns configmap is not updated with more entries.

How does it go about updating coredns? Is it trying to update the core CM directly? AKS requires a specific way to extend it and the limitation of no plugins: https://docs.microsoft.com/en-us/azure/aks/coredns-custom

[allenlsy]

According to the submariner doc (https://submariner.io/operations/usage/), the coredns CM should be updated by the lighthouse component and adds an clusterset.local entry to the CM. It didn't happen for AKS cluster.

[jsturtevant]

looks like it tries to update the coredns spec directly, which isn't supported in AKS. Does look like there is ability to specific a CustomDNS CM: https://github.com/submariner-io/submariner-operator/blob/d0e6ba625180ca5c5ca488cbb116138021b796a7/controllers/servicediscovery/servicediscovery_controller.go#L163

[allenlsy]

Another try we made is using Azure CNI plugin in the cluster network settings. This time there is no crashloop on components, but worker clusters still cannot be connected. No error found in coredns pods. The error was in submariner-operator pod:

[keithmattix]

I found submariner-io/submariner-operator#1047

But it looks old; still may be relevant?