Applications in any namespace - privileges issues to manage Application details in Argocd UI

[dgeorgievski]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have followed the instructions about enabling deployment of Applications in any namespace as described at
https://argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/#applications-in-any-namespace

Everything works with the exception of two minor issues

1. Argocd UI reports errors at loading Application settings.
   "Unable to load data: permission data"

The error appears to be related to loading of Links data. I can create the alert by clicking the "try again" link.
See the attached image.

The required RBAC ClusterRole and -Binding were applied as described here
https://argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/#adapt-kubernetes-rbac

I can confirm that the status of k8s Application resource is updated as expected which confirms the argocd-server has access to Application deployed in another namespace.

A similar error appears if I try to save the Application Details in the UI. See the attached images.

2. argocd-events could not create k8s Events in other namespace.
   I had to add RBAC privileges for creating Events in the examples/k8s-rbac/argocd-server-applications/argocd-server-rbac-clusterrole.yaml ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: argocd-server-cluster-apps
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/component: server
  name: argocd-server-cluster-apps
rules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
- apiGroups:
  - "argoproj.io"
  resources:
  - "applications"
  verbs:
  - create
  - delete
  - update
  - patch

To Reproduce

1. Follow the steps as described in https://argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/#applications-in-any-namespace

2. Create a new Application in a namespace other than argocd
   The new application should appear in the UI

3. Open Application details in the UI
   You should see "Unable to load data: permission denied" error message
   Click on the "try again" HTTP link appearing under LInks field. This should create another instance of the alert.

4. Try to change the Application settings and save.
   This should create "Unable to save: permission denied" alert.

5. Check argocd-server logs for errors indicating it cannot create k8s Events in the namespace where the Application is deployed.

Expected behavior
I should be able to manage an Applicaiton deployed in a namespace other than the control plane namespace through the UI.

Screenshots

Version

argocd version
argocd: v2.7.2+cbee7e6
  BuildDate: 2023-05-12T14:06:49Z
  GitCommit: cbee7e6011407ed2d1066c482db74e97e0cc6bdb
  GitTreeState: clean
  GoVersion: go1.19.9
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.2+cbee7e6
  BuildDate: 2023-05-12T14:06:49Z
  GitCommit: cbee7e6011407ed2d1066c482db74e97e0cc6bdb
  GitTreeState: clean
  GoVersion: go1.19.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.1.0 2023-06-19T16:58:18Z
  Helm Version: v3.10.1+g9f88ccb
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

Save Application details logs entries. It looks like argocd-server is still trying to locate the Application in argocd namespace

argocd-server-dbc866dbf-67x8n argocd-server time="2023-07-12T15:22:31Z" level=info msg="received streaming call /application.ApplicationService/WatchResourceTree" grpc.method=WatchResourceTree grpc.request.content="applicationName:\"gw-tekton-events-dev\" appNamespace:\"app-health-5g-dev\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-12T15:22:31Z" span.kind=server system=grpc
argocd-server-dbc866dbf-67x8n argocd-server time="2023-07-12T15:22:31Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.content="name:\"gw-tekton-events-dev\" appNamespace:\"app-health-5g-dev\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-12T15:22:31Z" span.kind=server system=grpc
argocd-server-dbc866dbf-67x8n argocd-server time="2023-07-12T15:22:31Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.content="name:\"gw-tekton-events-dev\" appNamespace:\"app-health-5g-dev\" " grpc.service=application.ApplicationService grpc.start_time="2023-07-12T15:22:31Z" span.kind=server system=grpc
argocd-server-dbc866dbf-67x8n argocd-server time="2023-07-12T15:22:32Z" level=warning msg="application does not exist" application=gw-tekton-events-dev namespace=argocd

Related to the lack of privileges to create k8s Event in the Application namespace. The issue was addressed by modifying the ClusterRole as described above.

argocd-server-5898cdcc5f-6dsg4 argocd-server time="2023-07-10T19:09:01Z" level=error msg="Unable to create audit event: events is forbidden: User \"system:serviceaccount:argocd:argocd-server\" cannot create resource \"events\" in API group \"\" in the namespace \"app-health-5g-dev\"" application=gw-tekton-events-dev dest-namespace=app-health-5g-dev dest-server="https://kubernetes.default.svc" reason=OperationStarted type=Normal user="Unknown user"

[dgeorgievski]

This fix #13854,which required upgrade to ArgoCD 2.7.7, resolved the error messages in the UI.

However, I think the ClusterRole should still be reviewed to enable argocd-server creating Events in other namespace. I'd gladly create a PR for that.

[crenshaw-dev]

@dgeorgievski the documentation mentions that the argocd-server RBAC has to be modified to support apps-in-any-namespace. What do you think of adding the events access to the example ClusterRole?

[dgeorgievski]

@dgeorgievski the documentation mentions that the argocd-server RBAC has to be modified to support apps-in-any-namespace. What do you think of adding the events access to the example ClusterRole?

That is what I had in mind, adding the events access to the example ClusterRole. It's a minor addition but I find it very helpful for troubleshooting Apps.

[crenshaw-dev]

@dgeorgievski if you put up the PR, I'll happily merge. :-)