feat: Decoupling application sync using impersonation

[anandf]

Implementation of proposal #14255
Addresses issue #7689
Related PR in gitops-engine: argoproj/gitops-engine#534

Many engineers from Red Hat worked on this effort. This PR consolidates all their effort to have a single PR/merge commit for the entire feature implementation so that it easy for maintainers to review and merge it.
CLI changes - @ishitasequeira
GUI changes - @raghavi101 @keithchong
E2E Tests - @Mangaal

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

Testing this feature

Prerequisites

• make
• docker
• sed
• kubectl
• kind

Procedure

1. Clone the repo and checkout the branch

git clone git@github.com:anandf/argo-cd.git
cd argo-cd
git checkout sync_with_impersonate

2. Build the docker image and the CLI client. Push the docker image to quay.io for testing.

export QUAY_USER=<username_in_quay.io>
IMAGE_NAMESPACE=quay.io/$QUAY_USER make image build-local
docker push quay.io/$QUAY_USER/argocd:latest
ls -l ./dist/argocd

3. Create a kind test cluster

kind create cluster --name argocd

4. Modify the image name to use the image built and pushed in step 2.

export QUAY_USER=<username_in_quay.io> 
sed -i "s/quay.io\/argoproj\/argocd/quay.io\/$QUAY_USER\/argocd/g" manifests/install.yaml

5. Install ArgoCD

kubectl create ns argocd
kubectl apply -f manifests/install.yaml -n argocd
kubectl config set-context --current --namespace argocd

6. Enable the Application sync impersonation feature in argocd-cm

kubectl patch cm/argocd-cm -n argocd --type=merge -p='{"data":{"application.sync.impersonation.enabled":"true"}}'

7. Create an App project

./dist/argocd proj create guestbook-proj -d https://kubernetes.default.svc,guestbook -s https://github.com/argoproj/argocd-example-apps.git --core

8. Add destination service account configuration for guestbook ns as below

./dist/argocd proj add-destination-service-account guestbook-proj https://kubernetes.default.svc guestbook guestbook-deployer --core

9. Create an argo application guestbook associated with AppProject guestbook-proj

./dist/argocd app create guestbook --core \
    --repo https://github.com/argoproj/argocd-example-apps \
    --path guestbook \
    --project guestbook-proj \
    --dest-server  https://kubernetes.default.svc \
    --dest-namespace guestbook \
    --directory-recurse \
    --sync-policy automated \
    --sync-option ServerSideApply=true

10. Check if the application fails to sync as the service account is not created yet.

kubectl get application guestbook -n argocd -o yaml

Sample error message:

message: 'Namespace auto creation failed: namespaces "guestbook" is forbidden:
          User "system:serviceaccount:guestbook:guestbook-deployer" cannot get resource
          "namespaces" in API group "" in the namespace "guestbook"'

12. Now create the service account guestbook-deployer in guestbook ns with the required access.

kubectl create ns guestbook
kubectl create sa guestbook-deployer -n guestbook
kubectl create rolebinding guestbook-deployer-rb -n guestbook --clusterrole cluster-admin --serviceaccount guestbook:guestbook-deployer

14. Sync the application and see if the sync operation succeeds now.

./dist/argocd app sync argocd/guestbook --core
./dist/argocd app list --core

15. Check the negative scenario when the sync operation fails with error when no matching SA is present.

./dist/argocd proj add-destination  guestbook-proj https://kubernetes.default.svc guestbook-dev --core

./dist/argocd app create guestbook-dev --core \
    --repo https://github.com/argoproj/argocd-example-apps \
    --path guestbook \
    --project guestbook-proj \
    --dest-server  https://kubernetes.default.svc \
    --dest-namespace guestbook-dev \
    --directory-recurse \
    --sync-policy automated \
    --sync-option ServerSideApply=true

16. Check if the application fails to sync as the service account is not created yet.

kubectl get application guestbook-dev -n argocd -o yaml

Sample error message:

failed to find a matching service account to impersonate: no matching
      service account found for destination server https://kubernetes.default.svc
      and namespace guestbook-dev. (retried 5 times)

[codecov]

Codecov Report

Attention: Patch coverage is 15.78947% with 128 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@f071fdc). Learn more about missing BASE report.
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/argocd/commands/project.go	0.00%	96 Missing ⚠️
cmd/util/project.go	0.00%	15 Missing ⚠️
pkg/apis/application/v1alpha1/app_project_types.go	10.00%	8 Missing and 1 partial ⚠️
controller/sync.go	79.16%	2 Missing and 3 partials ⚠️
util/settings/settings.go	66.66%	1 Missing and 1 partial ⚠️
server/settings/settings.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #17403   +/-   ##
=========================================
  Coverage          ?   55.80%           
=========================================
  Files             ?      320           
  Lines             ?    44206           
  Branches          ?        0           
=========================================
  Hits              ?    24669           
  Misses            ?    16977           
  Partials          ?     2560           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jannfis]

LGTM

[alpozcan]

Keen to see this feature stabilise!

FYI, I receive the following error following
argocd proj add-destination-service-account guestbook-proj https://kubernetes.default.svc guestbook guestbook-deployer --core on Argo CD 2.13.0 RC2:

W1001 11:13:09.905522   44304 warnings.go:70] unknown field "spec.destinationServiceAccounts"

which I'm guessing is because #19966 is still to be merged.

Also, is having the SA association at Application level planned? It seems to me that having this at Project level would force users to have one Project per Application if each app has its own deployer SA.

[anandf]

Thanks @alpozcan for trying out this feature and providing a feedback. The error you are seeing, looks like the server/client may not be running the version with this code change. Could you try with the latest changes.
Please check the server and client versions using the following commands to confirm if you are using the right versions.

argocd version --server
argocd version --client

Also, is having the SA association at Application level planned?
There is a plan to implement it, if we receive sufficient interests. Since, it affects the security posture, we want to have the admins to control it. You can still add multiple destinations to the same project and refer them in multiple applications.
For eg:

argocd proj add-destination-service-account guestbook-proj https://dev-cluster.example.com:6443 guestbook-dev guestbook-deployer --core
argocd proj add-destination-service-account guestbook-proj https://staging-cluster.example.com:6443 guestbook-stage guestbook-deployer --core
argocd proj add-destination-service-account guestbook-proj https://prod-cluster.example.com:6443 guestbook-prod guestbook-deployer --core

Different applications having different destinations can then be targeted to the same guestbook-proj.
Let me know if the above solution can work for your case, or is it something else that I misunderstood.