-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Repo Server high number of git requests when no changes are requested #12878
Comments
I've also just verified that our gitlab instance has no authenticated API request rate limits set, as we are using SSH creds I assume this is how Repo Server is making requests. |
The interesting part of this to me is that repo Server seems to be pulling all repos on startup, even though we have disabled automatic sync and only intend to trigger Syncs from Webhooks themselves. Entirely possible I'm misunderstanding the Repo Server startup process though |
I experienced similar issue, argocd multi-source Applications stayed in Unknown state until manually refreshed. I also noticed some github rate limiting during this issue. |
I turned on ApplicationSet and Application Controller Debug logs and started to see that there were a ton of reconciliation loops being created by the Application Controller due to Orphaned resources. I had set orphanedResources tag on all my ArgoCD Projects that made my applications attempt to claim ownership of all orphaned resources within its namespace that the application is deployed to.
Here is the difference in reconciliation and git ls-remote calls. There is some reconciliation loops still in place that I need to investigate, but it's significantly better now. Ref: #8100 (comment) |
@crenshaw-dev Did you see sth similar? I saw you asked us to create a separate issue. Please could you help us with it? |
I've noticed the issue starts at version 2.6.3 and an endless loop of reconciliation happening to applications that have |
Same Here! |
Is everyone here using ApplicationSets? I suspect the issue might be related to the ApplicationSet controller failing to normalize the App spec before applying it. The Application controller and the ApplicationSet controller end up fighting over the correct App manifest, resulting in constant reconciliation. I've merged a fix: #14481 |
Yes, we're using ApplicationSets as the other guys mentioned #14712. I've tried version 2.7.10 with no success. |
Relevant comments to this. Adding them here for reference: #14712 (comment), #14712 (comment), #14712 (comment), #14712 (comment), #14712 (comment), #14712 (comment) |
@crenshaw-dev We did the test, after upgrading Argocd and scaling down the appset controller to zero the reconciliation activity kept increasing. |
@crenshaw-dev Do you think the case: #14725 is related? I was reading the recent messages that seem very similar to our problem, we're using the app-of-apps pattern with multi-source apps. |
@crenshaw-dev I am wondering if there will be any actions on this. It is happening for several months, it has been mentioned by several and seems that there's no really activity on this issue. It's a pity that we cannot even upgrade to latest versions of ArgoCD and catchup with security vulnerabilities and latest improvements. Is there any other workaround? |
@nromriell We're following your amazing work in this issue where two parts has already merged. We believe our issue is related and we want to share some results after upgrading to 2.9.5. We've ~150 apps with multi-source. Do you believe is this expected until the merge of the third and fourth parts of the issue? Thanks. |
Hi @andrleite as of the last state I would expect the checkouts at least to be lower My changes are primarily around fixing the number of git requests between cache invalidations. Looking at the graphs you shared here it looks like your cache is nearly constantly invalidated which looks like the primary issue and likely why you aren't seeing the behavior you'd expect. Have you tried setting My time has been pretty limited lately so I haven't been able to continue to make improvements here but should at least be able to look at opening up the remaining two PRs here shortly. I think as is though those probably won't fix what you're seeing since they rely on the items being cached, it would just reduce the call count per cycle. |
Checklist:
argocd version
.Describe the bug
After upgrading to ArgoCD version 2.6.4 from 2.6.2 we experienced an issue where Repo Server was unable to resolve a git client and forced many apps into an
Unknown
state. When I manually refreshed the apps they were able to resolve the git client without issues, however they would automatically refresh on their own leading once again to an Unknown state regardless of them now beingHealthy
. As you can see from screenshots it appears that the repoServer kept attempting connections to the git client in large amounts seen by thels-remote
dashboard panel screenshot.Our current ArgoCD cluster has 5 clusters total that it deploys to using an App of Apps generator method, we currently have ~120 applications managed by this centralized ArgoCD cluster.
Our current setup for applications is roughly as follows:
Per team and environment we have an App of Apps that creates another set of App of Apps for each application that we would like to deploy, this sub App of Apps will then deploy the application (for example prometheus) to the appropriate external clusters.
We are also utilizing the new Multi-source applications to use values files contained in our private git repositories with a mix of private and public helm charts.
To Reproduce
We have disabled the automatic refresh on apps in favour for git webhooks to refresh apps when there are changes to the repos
Have ~100 applications on a HA ArgoCD setup, with the following relevant settings:
I assume that the repoServer reached a rate-limit built into our internal gitlab instance and kept sending requests after getting failures.
Expected behavior
I expect Repo Server to eventually fail on calls to the git service and not keep sending requests when there are no changes and the application is healthy.
Screenshots
https://user-images.githubusercontent.com/13317139/225333581-0e118090-e311-457a-987e-0a9861860129.png
https://user-images.githubusercontent.com/13317139/225334095-3786b36b-87ab-45b4-a572-18593fa371ee.png
Version
Unable to determine the exact SHA as it took out our git version, but it was
v2.6.4
as of approx.Tue Mar 14 12:03:49 2023 -0400
Logs
The text was updated successfully, but these errors were encountered: