Does argocd-repo-server cache git repos only under local /tmp directory?

[KihyeokK]

Some context:

Recently, we had a case where all argocd components were reassigned to nodes after a kubernetes upgrade. After the upgrade, argocd-repo-server pod was seen to be constantly getting OOMKilled. I thought this was due to all git repos trying to be fetched at the same time, so I increased the memory requests and limits temporarily, put it back to original configuration after a while, and it resolved OOMKilled issue. However, when I applied new memory configuration, argocd-repo-server pod restarted, which means that all git repos cached under /tmp must have been lost (and to clarify, an emptydir is mounted to /tmp). This confused me, since that does not explain how increasing memory and putting it back resolved the issue. Some questions stemmed from this:

1. Is argocd-repo-server only dependent on /tmp directory for cache?
2. Does argocd-repo-server depend on redis deployment in any way? I noticed that every time I rollout restart redis server, argocd-repo-server gets OOMKilled.
3. Is there any other reason why argocd-repo-server may have memory usage peaking after all argocd components were restarted?

Would really appreciate any insights.
Thank you!

[pasha-codefresh]

Yes, ArgoCD Repo Server uses redis for caching purposes, it cache some git information, also generated manifests and so on.

I think your memory usage may spike due to missed cache. Are you using Helm applications ?

[KihyeokK]

Thank you for your answer!

What's a bit weird is that, even after a while all the components were restarted, when things should be cached in redis, argocd-repo-server was constantly getting OOMKilled. And, while it was getting OOMKilled, it wasn't outputting any cache miss logs (which can be found in this issue). Maybe things were not fully cached in redis. Is there any other possible reasons of memory usage peaking that you see by any chance?

Also to answer your question, we are not using argocd helm chart, although we do use kustomize for deployment.

Thank you!

[pasha-codefresh]

I saw exactly same complain about helm, and he said that for plain manifests such issue not exists. looks like some caching missed when it is related to helm or kustomize

[pasha-codefresh]

https://cloud-native.slack.com/archives/C04SURUPDL2/p1724168899945489