Up kubernetes/client-go QPS and Burst config

[julianvmodesto]

kubernetes/client-go uses rate-limiting to prevent spamming the apiserver.

We had to rapidly scale our backend up, however Voyager Operator stopped updating the HAProxy ConfigMap's list of backend Endpoints, and unfortunately the ConfigMap didn't have a majority of our endpoint. Due to kubernetes/client-go throttling, the ConfigMap wasn't updated in an adequate time period so it was quicker for us to delete the Voyager Operator pod.

Voyager Operator logs showed many entries of the form request.go:638] Throttling request took 130.848105ms, ... showing that apiserver calls were being throttled client-side, including hooks for PUTs to update the HAProxy ConfigMap.

We can use the QPS and Burst config to prevent throttling to this extent https://godoc.org/k8s.io/client-go/rest#Config

[tamalsaha]

ConfigMaps are watched / updated from operator pod and watched/read from the HAProxy pods. Should not this mean that both of those flags are configurable? Currently we use inCluster.Config. So, I am guessing this is just using the default values.

I don't see how the deployment flags can be configured. One option is annotation, the other option is to pass from the operator pods (if using non-default setting).

[julianvmodesto]

Looks like the client config will be built from the cluster config, but it seems like it's primarily getting auth.

Could we set it directly on the prometheus-operator client in run.go? As for what the config should be, the default Burst is 10 and QPS is 5. I looked at Prometheus Operator, and they use rest.InClusterConfig, but they also set Burst to 100 and QPS to 100. Nginx Ingress Controller also uses rest.InClusterConfig and overrides to even higher values, of Burst to 1e6 and QPS to 1e6 -- this makes sense and is probably closer to what we want since Endpoints may be changing rapidly if there are many Ingresses watching many Services with many Endpoints.

[julianvmodesto]

Ahh i see what you're saying -- yes it seems both kubeClients should be configured w/ a higher Burst and QPS.

This looks like a good pattern to use inClusterConfig w/ overrides:
https://github.com/coreos/prometheus-operator/blob/9d68ecf289d711c66bef39d2f83429265abc6986/pkg/k8sutil/k8sutil.go#L79-L97

[tamalsaha]

appscode/voyager:4.0.0-rc.17 has this pr.