Application refresh is triggered by untracked resources

[duizabojul]

Describe the bug

For a while, Argo CD feels slow to reconcile (refreshing syncing) and i activated debug logs. I noticed a lot of application refresh where caused by untracked resources. We run argocd with tracking with annotation, so we expect all app controller refresh should be triggered only by changes in resources with argocd.argoproj.io/tracking-id annotation, and not by every resources watched by the controller.

Version

argocd: v2.8.2+dbdfc71
  BuildDate: 2023-08-24T20:05:39Z
  GitCommit: dbdfc712702ce2f781910a795d2e5385a4f5a0f9
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64

Logs

One example among others

time="2023-09-21T08:39:37Z" level=debug msg="Requesting app refresh caused by object update"
api-version=kubernetes.crossplane.io/v1alpha1 application=argocd/admin-infra-it-central cluster-name=it-admin 
fields.level=0 kind=Object name=central-postgres-engine-configuration 
namespace="(cluster-scoped)" server="https://kubernetes.default.svc"

This update is caused by untracked child resource owned by tracked resource

apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
metadata:
  annotations:
    crossplane.io/composition-resource-name: postgres-engine-configuration
    crossplane.io/external-create-pending: '2023-07-22T17:30:40Z'
    crossplane.io/external-create-succeeded: '2023-07-22T17:30:40Z'
    crossplane.io/external-name: central-postgres-engine-configuration
    crossplane.io/paused: 'false'
  name: central-postgres-engine-configuration
  ownerReferences:
    - apiVersion: storage.crossplane.airnity.net/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: XDatabaseEngine
      name: central
      uid: 1c6697d2-6212-4d26-997f-47ea02aacba0

apiVersion: storage.crossplane.airnity.net/v1alpha1
kind: XDatabaseEngine
metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/tracking-id: >-
      admin-infra-it-central:storage.crossplane.airnity.net/XDatabaseEngine:crossplane-infra-it-central/central
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"storage.crossplane.airnity.net/v1alpha1","kind":"XDatabaseEngine","metadata":{"annotations":{"argocd.argoproj.io/sync-options":"SkipDryRunOnMissingResource=true","argocd.argoproj.io/tracking-id":"admin-infra-it-central:storage.crossplane.airnity.net/XDatabaseEngine:crossplane-infra-it-central/central","xfn.crossplane.airnity.net/backup":"true"},"name":"central"},"spec":{"compositionRevisionSelector":{"matchLabels":{"channel":"stable"}},"compositionSelector":{"matchLabels":{"engine":"postgres","provider":"aws"}},"params":{"clusterRef":{"name":"central"},"deletionProtection":false,"instanceType":"small","masterUserSecretRef":{"name":"postgres-central-credentials","namespace":"crossplane-infra-it-central"}},"writeConnectionSecretToRef":{"name":"postgres-central-connection","namespace":"crossplane-infra-it-central"}}}
    xfn.crossplane.airnity.net/backup: 'true'
  name: central

[pepe9012]

Similar issue: #9558 (comment)

[crenshaw-dev]

The refresh allows the UI to display the current state of the updated resource.

If you don't always need the latest state, you can exclude certain updates from triggering a refresh: https://argo-cd.readthedocs.io/en/latest/operator-manual/reconcile/

[duizabojul]

I don't understand why a refresh of application is needed when an untracked resource is modified. Can't Argo watch these resources and update UI accordingly without refreshing applications?

[crenshaw-dev]

The word "refresh" is ambiguous. There are different levels of refresh, with different amounts of work involved. I believe that this case (an update in a child of a managed resource) would involve the lowest refresh level and therefore the smallest amount of work.

argo-cd/controller/appcontroller.go

Lines 77 to 86 in 03c6e1a

	const (
	// Compare live application state against state defined in latest git revision with no resolved revision caching.
	CompareWithLatestForceResolve CompareWith = 3
	// Compare live application state against state defined in latest git revision.
	CompareWithLatest CompareWith = 2
	// Compare live application state against state defined using revision of most recent comparison.
	CompareWithRecent CompareWith = 1
	// Skip comparison and only refresh application resources tree
	ComparisonWithNothing CompareWith = 0
	)

[duizabojul]

Some of untracked resources are modified every 5s by operators, I don't think the good behavior for Argo is to react to these changes (more of just updating UI).

We had to deactivate orphaned resources because of this same issue, app controller being flooded by refreshes of orphaned resources.

Letting people debug app controller to list and ignore every potential changes of untracked resources is not a good behavior imho.

[duizabojul]

The word "refresh" is ambiguous. There are different levels of refresh, with different amounts of work involved. I believe that this case (an update in a child of a managed resource) would involve the lowest refresh level and therefore the smallest amount of work.

argo-cd/controller/appcontroller.go

Lines 77 to 86 in 03c6e1a

	const (
	// Compare live application state against state defined in latest git revision with no resolved revision caching.
	CompareWithLatestForceResolve CompareWith = 3
	// Compare live application state against state defined in latest git revision.
	CompareWithLatest CompareWith = 2
	// Compare live application state against state defined using revision of most recent comparison.
	CompareWithRecent CompareWith = 1
	// Skip comparison and only refresh application resources tree
	ComparisonWithNothing CompareWith = 0
	)

Ok this make more sense.

[crenshaw-dev]

The default behavior is intended to provide the best possible end-user experience. This choice does cause a trade-off with admin effort to tune Argo CD.

I think we'd favor improving the default experience rather than changing defaults in a way that degrades UX. For example, we're working on rate-limiting refreshes, which would likely mitigate the problem you're encountering: #15233

[duizabojul]

I'm okay with this, maybe adding additional information in debug logs about the kind of refresh should allow us to better isolate real issues.

Feel free to close this issue.

[crenshaw-dev]

The information is there, it's just not the most readable: fields.level=0. Maybe the ComparisonWithNothing string would be better, but it still requires some context to understand.