kube-apiserver persistently high memory usage with large number of CRDs

[matthchr]

kube-apiserver uses a surprisingly large amount of memory when CRDs are added.

ASO v2.0.0 installs ~125 CRDs. Testing quickly on a local kind cluster (version 1.26.3) shows that resident memory grows from:
304mb (empty cluster) -> 380mb (cert-manager installed) -> 2.5g (aso just installed) -> 2.0g (steady state).

This means that each of our 125 CRDs causes ~13mb of memory usage in kube-apiserver.

Customer impact

It's very easy to cross managed cluster kube-apiserver memory limits. If this happens, kube-apiserver may get OOMKilled by the managed cluster provider (such as AKS). This has negative downstream effects on pretty much everything in the cluster, including triggering pod restarts due to expired watches, failed kube api requests, etc.

Quick instructions for profiling kind apiserver

1. Start kind.
2. Install aso
3. kubectl proxy --port=8080 &
4. go tool pprof -png "http://localhost:8080/debug/pprof/heap" > out.png - The actual trailing path can be anything: /debug/pprof/profile, debug/pprof/heap, etc
5. Or alternatively you can do curl localhost:8080/debug/pprof/heap > out.pprof and then analyze it with go tool pprof out.pprof

Experiments

• Is memory usage reduced if CRDs have no documentation included?
• Is memory usage reduced if v1beta versions are removed?
• Examine in detail w/ pprof where kube-apiserver memory usage is coming from

Prior art

• https://blog.upbound.io/scaling-kubernetes-to-thousands-of-crds/
• Summary of crossplane CRD scaling issues
• Great Crossplane summary of kube-apiserver memory usage

Some key snippets from Crossplane's summary:

Further investigation reveals that zapcore.newCounters has allocated 815 objects in the process heap (where there are 780 CRDs) and each allocation is _numLevels * _countersPerLevel * (Int64 + Uint64) = 7 * 4096 * (8+8) = 448 KB! Thus, for 780 CRDs (plus other zap loggers), we are using 815 * 448 KB = ~356 MB of heap space. zapcore.newCounters is responsible for the ~31% of the heap space allocated!

On large CRDs:

However, to exacerbate the situation, we have generated ~190 CRDs each with 5000 OpenAPI v3 schema properties. Even with only 190 such CRDs, heap profiling data reveals that ~5 GB of heap space is allocated and zapcore.newCounters is no longer the largest allocation site anymore, as expected. This profiling data better emphasizes allocation sites which are inflated with large (deserialized) OpenAPI v3 CRD schemas. One can generate such CRDs using the following command:

Proposed fixes

• Now: Users on AKS should use Standard (previously Paid) tier, and not Free tier.
• Shorter term: Finish work on Install Selected CRDs only by default #1433
• Longer term: Engage with upstream on improving CRD memory usage. For example this PR would likely help: Create and reuse a single etcd client kubernetes/kubernetes#114458

[matthchr]

Here's a pprof for kube-apiserver from a local kind cluster for ASO v2.0.0:

Interestingly it only shows 1.1g used whereas resident memory was 2.0+g, I'm not sure where that discrepancy comes from, will need to dig further possibly: