feat: add e2e test for hook finalizer which prevents external resource deletion

[dejanzele]

This PR adds a test for the issue with stuck hooks when a hook resource gets deleted externally (i.e. Job with ttlSecondsAfterFinished: 0).

This partially closes #21055

This PR uses a replace in the go.mod to use my fork of the gitops-engine as a proof that the change works as expected in ArgoCD.

This PR is dependant on the following gitops-engine PR argoproj/gitops-engine#646

Related issues:

• waiting for completion of hook and hook never succeds #6880
• fix: set post-install-hook ttlSecondsAfterFinished to 60 (#6765) aws/karpenter-provider-aws#6825
• ArgoCD stuck in waiting for completion of hook batch/Job/argocd-redis-secret-init argo-helm#2887
• https://github.com/G-Research/gr-oss/issues/316

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❗ Preview Environment deployment failed on Bunnyshell

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🚀 /bns:deploy to redeploy the environment
• ❌ /bns:delete to remove the environment

[crenshaw-dev]

In my mind, the perfect e2e test would:

1. sync an app with a hook that takes ~1min to complete
2. stop app reconciliation (turn off the controller? disconnect the target cluster?)
3. kubectl delete the hook resource
4. re-enable reconciliation
5. assert that the sync operation completes successfully

Ideally this test would fail on master, because Argo CD will miss the status updates on the hook resource, but it would succeed on this PR, because the hook resource's deletion would be blocked by the finalizer.

[dejanzele]

Ok, so this test is a really simple check to validate the finalizer gets added/removed for all sync phases, and that app is healthy and sync fine.

Do you think it makes sense to have a simple and advanced test? The simple test just validates the logic quickly, and the advanced test validates what you just mentioned, because in the advanced test, there are mulitple pieces?

[crenshaw-dev]

Yep, probably fine to have both!