User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"

[denis1skr]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

WF template: https://argo-workflows.readthedocs.io/en/latest/walk-through/hello-world/
Version: v3.5.1 - no error
Version: v3.5.2 and higher- with error

Error:

hello-world-hnd8w   Error    9m31s   Error (exit code 1): pods "hello-world-hnd8w" is forbidden: User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"

Version

v3.5.2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
spec:
  entrypoint: whalesay          # invoke the whalesay template
  templates:
    - name: whalesay              # name of the template
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "hello world" ]
        resources: # limit the resources
          limits:
            memory: 32Mi
            cpu: 100m

Logs from the workflow controller

time="2024-03-11T21:25:52.335Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.339Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.339Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.339Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.340Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.340Z" level=info msg="Pod node hello-world-nnkdk initialized Pending" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.346Z" level=info msg="Created pod: hello-world-nnkdk (hello-world-nnkdk)" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.347Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.347Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:25:52.351Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=395640 workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.354Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.356Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.357Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=hello-world-nnkdk old.message= old.phase=Pending old.progress=0/1 workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.360Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.360Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:02.402Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=395673 workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.408Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.410Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.411Z" level=info msg="Pod failed: Error (exit code 1): pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" displayName=hello-world-nnkdk namespace=argo pod=hello-world-nnkdk templateName=whalesay workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.411Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1): pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" new.phase=Error new.progress=0/1 nodeID=hello-world-nnkdk old.message= old.phase=Running old.progress=0/1 workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.413Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.413Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.413Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.413Z" level=info msg="Updated message  -> Error (exit code 1): pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.413Z" level=info msg="Marking workflow completed" namespace=argo workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.419Z" level=info msg="cleaning up pod" action=deletePod key=argo/hello-world-nnkdk-1340600742-agent/deletePod
time="2024-03-11T21:26:12.452Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=395700 workflow=hello-world-nnkdk
time="2024-03-11T21:26:12.459Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/hello-world-nnkdk/labelPodCompleted

Logs from in your workflow's wait container

time="2024-03-11T21:26:06.146Z" level=info msg="No output parameters"
time="2024-03-11T21:26:06.146Z" level=info msg="No output artifacts"
time="2024-03-11T21:26:06.171Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-03-11T21:26:06.174Z" level=warning msg="Non-transient error: pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2024-03-11T21:26:06.174Z" level=error msg="executor error: pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2024-03-11T21:26:06.175Z" level=info msg="Alloc=8280 TotalAlloc=13096 Sys=24677 NumGC=3 Goroutines=8"
time="2024-03-11T21:26:06.177Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2024-03-11T21:26:06.178Z" level=warning msg="Non-transient error: pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2024-03-11T21:26:06.178Z" level=error msg="executor error: pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2024-03-11T21:26:06.180Z" level=fatal msg="pods \"hello-world-nnkdk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""

[agilgur5]

You need to set proper RBAC for Pods of a Workflow: https://argo-workflows.readthedocs.io/en/latest/workflow-rbac/.

Version: v3.5.2 and higher- with error

This is duplicative of #12391 (comment) et al -- this was always required in the docs but simple steps were able to bypass this before v3.5.2.

[denis1skr]

@agilgur5 Hello, I would appreciate some kind of documentation about "workflowtaskresults" Resource.
What are its use cases?
Is it cleaned automatically by Argo exec?

[agilgur5]

would appreciate some kind of documentation about "workflowtaskresults" Resource.

Duplicate of #11167. It's not user-facing, it's used by the Executor and Controller to communicate with each other.

[denis1skr]

Hi,
if I use only minimum required Role rule (to create/patch workflowtaskresults), is there a way while running WF#1 to patch workflowtaskresults of WF#1 by running WF#2 in way to bring WF#1 to wrong status?

[Joibel]

Yes technically, if they are in the same namespace.

You don't say whether this is something you desire to do, or something you're trying to prevent.

[denis1skr]

@Joibel I would like to be able to reproduce this case. And to prevent it.

I saw the only change between running and succeded WF in WorkflowTaskResult CR is workflows.argoproj.io/report-outputs-completed: "true", but patching this value by WF#2 did not influence the running state of WF#1