perf: use concurrent map when storing metrics

[rarruda]

What this PR does / why we need it: In busy/large clusters, will prevent timeouts from long living locks/concurrency issues, as the writing to the map takes overly long, blocking the metrics-reading thread and as the lock doesn't get released in a timely manner, timing out the request.

In these graphs you see:

1. VMagent scrape duration (yellow: with this patch applied, green: no patch)
2. VMagent samples scraped: (yellow: with this patch applied, green: no patch)
3. Heatmap from selfMetrics with patch applied
4. Heatmap from selfMetrics no patch
5. Number of pods in LOG scale, max is close to 16k
6. Number of nodes in cluster
   

You can see that with this patch we reduce latency significantly under all scenarios in our cluster.

Inspired by previous PR at #1028

How does this change affect the cardinality of KSM: does not change cardinality

Which issue(s) this PR fixes:
Fixes #995

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: rarruda / name: Renato Arruda (f2a7639, 3e2d1e9)

[k8s-ci-robot]

Welcome @rarruda!

It looks like this is your first PR to kubernetes/kube-state-metrics 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kube-state-metrics has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[mrueg]

Thanks for your contribution!
/lgtm
and
/hold
for others to review as well.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrueg, rarruda

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mrueg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dgrisonnet]

/triage accepted
/assign

[dgrisonnet]

The changes looks good to me, thank you @rarruda for pushing that improvement :)

I'll update our perfs-tests and see how much we improve with this.

[dgrisonnet]

Please refrain from merging until we have the results.

[rarruda]

Note: for clusters with very little load, and/or sequential read/write access patterns (synthetic data getting generated, and only after then read), I would expect that the original implementation having one mutex for the Map would perform better.

But that's not real world performance.

In our (real world, production) case, as seen by the graphs, the locking contention gets to be too great as too many things are happening in parallel.

More fine grained locking has an overhead associated with it, but the benefit only is apparent under heavy multithreaded read/write loads. The marginal cost of extra locking/atomic operations in sync.Map might be measurable in small clusters. (I have not done much testing in small clusters).

As work arounds, we also tried sharding, clustering, dividing up metrics to go do different instances, but those were really just palliative measures for the underlying lock contention. It increased complexity in our setup significantly, and felt like ugly hacks.