Fleet container fails: /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied

[patrijua]

With Rancher v2.6.3 running in AWS EKS with kubernetes version 1.21, fleet fails to reconcile the downstream cluster because "fleet" container fails. Previous "step-git-source" is successful and clones the git repository.

containerStatuses:
 - name: fleet
state:
terminated:
exitCode: 1
reason: Error
message: >
 time="2022-06-08T11:46:40Z" level=fatal msg="open
 /var/run/secrets/kubernetes.io/serviceaccount/token: permission
 denied"

Seems that the user running the pod (1000) does not have access to token. I did a test by modifying the pod definition to run as root and triggered the pod manually, then fleet container was ran successfully.

  securityContext:
    runAsUser: 0

[patrijua]

Any advice on effective workaround? I tried to modify the related CRD object (gitjobs.gitjob.cattle.io) but the modification does not stay, it immediately returns back to original.

[patrijua]

Related? kubernetes-sigs/external-dns#1185

[patrijua]

Also setting the fsgroup on the related gitjobs.gitjob.cattle.io object can be used as a workaround.

securityContext: fsGroup: 65534 runAsUser: 1000

[aiyengar2]

@patrijua it does seem to be related to the issue you linked. Relevant snippet from AWS documentation in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html.

By default, only containers that run as root have the proper file system permissions to read the web identity token file. You can provide these permissions by having your containers run as root, or by providing the following security context for the containers in your manifest.

However, the followup workaround you listed seems to contradict this since runAsUser 1000 (non-root) seemed to work as expected?

[patrijua]

Both workarounds did work. Either run as root OR run as 1000 and add fsGroup: 65534 to securityContext.