Envoy Admin operations with multizone and HA Global CP can fail

[jakubdyszkiewicz]

Description

Envoy Admin operations (xds config, stats, clusters available through GUI) are sent over KDS from Global to Zone CP.
The problem is that Zone CP connects to only one instance of Global CP initiating KDS.

[jakubdyszkiewicz]

We need to let one Global CP instance somehow pass this information to another CP instance.

Ideas

Each CP instance is configured with other CP instances IPs

We could configure each CP with IPs of other CPs.
CP Instance receives a request for a config dump. If it has a Zone CP connected, it executes as is. If not, it forwards the request to all instances and returns the response if one of the instances did not error.

This is challenging because

• You need to update this information. CPs can go down.
• You need to handle authentication on API Server or open an extra communication channel between CP instances.

Request/Response via ConfigMap

CP Instance receives a request for a config dump. If it has a Zone CP connected, it executes as is. If not, it creates a ConfigMap with a request.
Other instances poll config maps for requests. The instance that has Zone CP executes a request and saves it in the config map. The initiating instance reads the result, passes it to the user, and deletes the config map.