application-controller pod has very High CPU usage

[sunidhi121]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.
• I've asked the question in https://cloud-native.slack.com/archives/C01TSERG0KZ/p1633928068217000

Describe the bug

The application-controller pod uses about 769% of CPU with ~129 apps.
Our argocd was using 2215% of CPU before upgrading it from v1.x to 2.0.3. Not it has reduced to 769%.
Argocd is configured for 7 different external clusters. Our workflow has main-app subapp architecture contributing to around 129 apps running in 8 clusters.
There are many applications that are in constantly in syncing state or progressing and many are out of sync. Few applications also have other sync issues. For few applications the health check is constantly happening.

To Reproduce

1. Deploy argo cd 2.0.3 into a 3 node kubernetes cluster.
2. Add 7 different external clusters to deploy applications and deploy applications into all the clusters with appofapps structure in gitlab.
3. Observe resource usage via Grafana

Expected behavior

The application-controller uses less CPU.

Screenshots

Version
v2.0.3+8

Logs

LOGS cant be shared due to company security reasons.
 I could not find anything odd in the argocd controller logs.

[jessebye]

@sunidhi121 possibly related: #6108

[agaudreault]

The behavior can be configured in ignoreResourceUpdates to resolve this issue.