-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
2.6.1 Multi Source App - Application status / Sync / Refresh #12379
Comments
we are seeing this issue too |
Yeah, I am experiencing some of the same issues. Not exactly the same, but the same sort of idea. What I experienced:
|
We also seeing this issue. As I can see application controller does not refresh(every 3 min) status of application with multi path. I mean, I didn't see any logs messages like this |
I wonder if the order you define them would fix this? |
@mulhotavares I'm tried both variant, the same result |
Yeah, was able to reproduce it here too. |
FYI @crenshaw-dev , @ishitasequeira . I saw other issues related to sync/refresh, eg. #11772 |
I think this may be a dupe of #12301. |
Seeing this too |
Thanks for everyone's patience on this. Since it's not a super destructive bug, I've been focused on other stuff. I'm going to pop open a debugger today and see what I can figure out. |
In another of what I know must feel like painfully slow and sparse updates: debugger is open. :-) |
You're the best, I can feel your pain. :) |
FOUND IT. Dodgy |
…oproj#12379) Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
Wonder if this is related, but I found a similar issue on argo 2.5 after adding a |
@mulhotavares if related, it's loosely related. By my reading, the problems directly caused by this bug should be limited to only multi-source applications. @lukma99 poooossibly. I'll try to reproduce it in the master branch. |
Thanks @crenshaw-dev. I was just trying to help as I guessed it could be part of the same code path. |
@mulhotavares do you know if there's another issue open for that problem? I think you're right, the fix is very close in the code path to the fix for this issue. |
This seems to be related: The author says it applies and it still shows as out-of-sync, as if it the unrelated field was ignored. But it could be that the manifests are just not being refreshed? |
@mulhotavares yikes, a lot's going on in that issue. Would you mind opening a new issue with only the problem that you know can be solved with refresh? I think I know how to fix it. Once that's done, we can link to the original issue and see if it solves those various problems. |
Sure. Let me create one. |
) (#12576) * fix: evaluate all possible refresh reasons for multi-source apps (#12379) Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * remove redundant parentheses Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * tests Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't auto-sync, it makes tests flaky Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * auto-sync because sync CLI doesn't work for multi-source apps Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't require out-of-sync - app may sync quickly Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * timeout 60 Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * fix timeout Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
) (#12576) * fix: evaluate all possible refresh reasons for multi-source apps (#12379) Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * remove redundant parentheses Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * tests Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't auto-sync, it makes tests flaky Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * auto-sync because sync CLI doesn't work for multi-source apps Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't require out-of-sync - app may sync quickly Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * timeout 60 Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * fix timeout Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
) (#12609) * fix: evaluate all possible refresh reasons for multi-source apps (#12379) * remove redundant parentheses * tests * don't auto-sync, it makes tests flaky * auto-sync because sync CLI doesn't work for multi-source apps * don't require out-of-sync - app may sync quickly * timeout 60 * fix timeout --------- Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> Co-authored-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
I plan to release the fix Monday. :-) |
Amazing! Thanks a lot @crenshaw-dev! |
Yep, it'll be 2.6.3 |
@crenshaw-dev As promised, I created this issue: |
FYI: looks like now we have some infinite loop
|
@khorn7sk would you mind opening a new issue for this? Could you also bump the log level up to debug to see if we log a reason why the refresh was requested? |
Yesterday I believe I experienced a similar issue to @khorn7sk. I had upgraded to 2.6.4 and noticed that the repo server was making many requests each minute around the same time and was hammering our git instances with an insane amount of requests (See images 1 and 2). My current ArgoCD setup has about 120 apps, and auto-refresh is turned off in favour of git webhooks, which don't have changes every minute. All the logs look like this, unfortunately I was unable to turn on debug mode as I had to fix it quickly after I figured out the issue.
@crenshaw-dev please let me know if I should create a separate ticket to track this issue as it took out of git instance last night :D |
@d-wierdsma please do file a new issue! If you go back to the beginning of the logs, do you get anything different? I suspect the error above is due to rate-limiting. But we won't know until this is merged, because the actual error message is obscured: #12876 |
For me it was issue with kyverno, it generating a lot of reports, and that report triggered ArgoCD. |
I can definitely look at the beginning of this to determine what might have been going on. Shouldn't repo server only retry a connection a certain amount of times per app then fail @crenshaw-dev ? I've set Entirely possible that it is getting rate-limited by git, in fact I expect so with that amount of requests 😅 |
@crenshaw-dev created a ticket! #12878 Please let me know if there is anything I can do to help out with investigation or potential fixes as well, always eager to help out :D |
…oproj#12379) (argoproj#12576) * fix: evaluate all possible refresh reasons for multi-source apps (argoproj#12379) Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * remove redundant parentheses Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * tests Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't auto-sync, it makes tests flaky Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * auto-sync because sync CLI doesn't work for multi-source apps Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * don't require out-of-sync - app may sync quickly Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * timeout 60 Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> * fix timeout Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> --------- Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
@khorn7sk We've at the moment a similiar problem with ArgoCD and we're using Kyverno too. |
Checklist:
argocd version
.Describe the bug
Application status of multi source Apps in WebUI and ArgoCD CLI doesn't match with live system/deployed resources.
To Reproduce
kubectl apply -f demo-multi-source.yaml
Bildschirmaufnahme.2023-02-09.um.18.30.58.mov
If the multi-source app is configured for auto sync, the resources gets created immediately, but the Argo App will remain OutOfSync. If the
values.yaml
changes, an App sync isn't sufficient, refresh is required. If a resource is out of sync and need pruning, then it get even worse, because then a combination of app sync + prune followed by refreshing the app is required.Expected behavior
App Sync should update the App and update the App status.
Version
The text was updated successfully, but these errors were encountered: