ArgoCD at scale of 500k applications

[ronklein]

We are managing about 500k applications which spread over 10k K8s cluster.
We are exploring a few approaches:

1. Using ArgoCD Dynamic Cluster Distribution, but seems like this feature is not stable yet.
2. Having an ArgoCD on each cluster to handle the local deployments. In this scenario. there is a overhead of managing all of the ArgoCD deployments.
3. Having ArgoCD on each region, this introduce sharding by cluster region. This approach also introduce management overhead.

Do you have any recommendation on the best-practice approach using ArgoCD for this task? Any other thought how to tackle this with ArgoCD?

[ErikLundJensen]

I guess some of the 500k applications are the same just deployed multiple times, for example deploying ExternalSecrets to every cluster. How many different applications are we talking about?

Using "divide and conquer" technique you could look at:

• Split test and production applications into separate the clusters.
• Group applications that belong together (e.g. group foundation components that are installed in every cluster)
• Use ApplicationSets to generate the Applications
• Let's ArgoCD install/manage ArgoCD in other clusters (decouple dependencies to cockpit clusters)
• Make sure you don't have dependencies between regions (unless you can accept the loss of control if region is faulty)

There is no simple answer for your setup... For example, if you are adding/removing 100 clusters every day that have a much higher impact upon you architecture than the number of clusters and applications.

[wanghong230]

A lot of my ideas regarding this are reflected in this blog - https://akuity.io/blog/argo-cd-architectures-explained/

You are balancing and shifting the complexity to different levels. Connectivity, resilience, and manageability play huge roles in balance.

• Be aware of the connectivity challenges, especially in the networking setup
• Be aware of the data transfer bandwidth and cost
• Be aware of the API server stress in the high load
• Be aware of the blast radius and security posture
• Be aware of the overhead of the management ability (thinking of doing CVE patch on 10K Argo CD instances)