Resources are created during dry-run when using Force=true,Replace=true sync options

[rafal-jan]

If resources have the argocd.argoproj.io/sync-options: Force=true,Replace=true annotation, they are recreated twice:

• when Argo CD performs dry-run
• when Argo CD performs the actual sync

This behavior can be observed when using multiple Jobs with the argocd.argoproj.io/sync-options: Force=true,Replace=true annotation and different sync waves. All Jobs are created immediately on the cluster during the dry-run phase and then recreated one-by-one as their respective sync wave is processed by Argo CD.

More information about using replace with dry-run and force options can be found in kubernetes/kubectl#1222. kubectl is used as library in gitops-engine but only Run() method is called. However, the fix in kubernetes/kubernetes#110326 updated the Validate() method.

I believe that the force option should not be set to true when performing a dry-run replace operation. This would avoid unnecessary resource recreation and make it usable with sync waves.

[mmclane]

I believe I am seeing this too. I have this annotation set on a K8s job. When that job gets replaced, it happens twice therefore the job runs twice even though it succeeds the first time.

[mmclane]

Having done a little more testing, I have found that the job gets recreated twice when something else has the annotation: argocd.argoproj.io/sync-wave: "-1"

To test this out I created an app that installs a custom helm chart. That helm chart creates two things, a configmap and a job. The job runs a hello world container.

If I have no annotations, ArgoCD can not update the job. To accommodate this I add the annotation argocd.argoproj.io/sync-options: Replace=true,Force=true to the job. Everything works as expected if I change the image tag for that job via a parameter override and the job gets replaced once.

If however I add the annotation argocd.argoproj.io/sync-wave: "-1" to the configmap, then when I update the jobs image tag, ArgoCD will replace the job twice causing it to run twice.

Here are some additional observations:

• If I set the configmap's annotation to argocd.argoproj.io/sync-wave: "0", things work as expected.
• If I set the job to have annotation to argocd.argoproj.io/sync-wave: "1", the job gets replaced twice.

I am running ArgoCD version v2.12.3+6b9cd82