Concurrent collector runs lead to an OOM kill

[kokes]

Maybe somewhat related to the other OOM problems mentioned in other issues, but this time I was able to reproduce the issue. The issue is not some leak or accumulated objects - it's more that the collector is slow and if your Prometheus refresh interval is sufficiently short, multiple concurrent collectors will run, all accumulating their stats into memory, eventually leading to an OOM kill.

To reproduce, I created a bunch of namespaces and TLS issues

#!/bin/bash

num_secrets=500

for ((i=0; i<=$num_secrets; i++))
do
    echo "Generating Secret $i..."

    openssl genrsa -out key$i.pem 2048
    openssl req -new -key key$i.pem -out csr$i.pem -subj "/CN=mydomain.com"
    openssl x509 -req -days 365 -in csr$i.pem -signkey key$i.pem -out cert$i.pem

    kubectl create ns my-namespace-$i
    kubectl create secret tls -n my-namespace-$i my-new-tls-secret-$i --cert=cert$i.pem --key=key$i.pem

    echo "Secret $i created."
done

rm *.pem

And observed memory use of the x509 exporter (both 3.3.0 and 3.12.0).

---

I do have a potential fix for this, will file it shortly.

[age9990]

We have faced the same issue in our environment with thousands of certificate secrets.
Really hope this fix is merged soon.

[monkeynator]

🎉 This issue has been resolved in version 3.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀