Missing permissions while tekton bundle resolver pulls from a private Google Artifact registry repository

[shruthipuranik]

Expected Behavior

tekton pipelines bundle resolver pulls the image from Google artifact registry using the service account passed in the pipeline run spec.

Actual Behavior

Error retrieving pipeline for pipelinerun default/: error requesting remote resource: error getting "bundleresolver" "default/bundles-b662ee4154232724d067bb82f6b25578": cannot retrieve the oci image: GET https://us-west2-docker.pkg.dev/v2/token?scope=repository%!A(MISSING)%!F(MISSING)%!F(MISSING)%!A(MISSING)pull&service=: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects//locations/us-west2/repositories/" (or it may not exist)

Steps to Reproduce the Problem

1. Create a KinD cluster with at least one worker node.
2. Install Tekton pipelines version v0.50.0 on the cluster:

kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.50.0/release.yaml

3. Create a JSON key for the GCP service account and store it in private-key.json. The service account should have Artifact registry reader and Artifact registry writer roles.
4. Create an image pull secret in default namespace with the service account key as follows:

kubectl create secret docker-registry artifact-registry \
  --docker-server=https://us-west2-docker.pkg.dev \
--docker-email=<service account email> \
  --docker-username=_json_key \
  --docker-password="$(cat private-key.json)"

5. Create a service account in default namespace with the image pull secret as follows:

kubectl apply -f - <<EOF                                                                                                                                                                                                                                        
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-artifact-registry
imagePullSecrets:
- name: artifact-registry
EOF

6. Apply the pipeline run in the default namespace as follows:

kubectl apply -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: sample-pipelinerun
spec:
  serviceAccountName: sa-artifact-registry
  pipelineRef:
    resolver: bundles
    params:
    - name: serviceAccount
      value: sa-artifact-registry
    - name: bundle
      value: us-west2-docker.pkg.dev/<GCP project name>/<repository name>/<bundle name>:latest
    - name: name
      value: <pipeline name>
    - name: kind
      value: pipeline
EOF

This throws the error as follows:

Error retrieving pipeline for pipelinerun default/<name of pipeline run>: error requesting remote resource: error getting "bundleresolver" "default/bundles-b662ee4154232724d067bb82f6b25578": cannot retrieve the oci image: GET https://us-west2-docker.pkg.dev/v2/token?scope=repository%!A(MISSING)<GCP project name>%!F(MISSING)<repository name>%!F(MISSING)<bundle name>%!A(MISSING)pull&service=: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/<project name>/locations/us-west2/repositories/<repository name>" (or it may not exist)

Additional Info

• Kubernetes version:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.24.0
WARNING: version difference between client (1.28) and server (1.24) exceeds the supported minor version skew of +/-1

• Tekton Pipeline version:

Client version: 0.32.0
Pipeline version: v0.50.0

Things I have tried out already:

• I have tried applying a pipeline run without using the bundle resolver from the same artifact registry repository using the same service account and it works fine. The step images are also in the same artifact registry repository BTW.
• I have tried doing a docker pull for the bundle image with the same private key outside the KinD cluster and that works fine as well.
• I have tried deleting the default service account from the bundleresolver-config config map in tekton-pipelines-resolvers namespace just in case there was a fall back to the default service account in the bundle resolver and I still get the same error.
• I have tried using a non-existing service account in the pipeline run manifest and it still throws the same error (Errors from the keychain library are ignored in the bundle resolver code:

  pipeline/pkg/resolution/resolver/bundle/resolver.go

  Line 97 in f5578a8

  kc, _ := k8schain.New(ctx, r.kubeClientSet, k8schain.Options{

  )
• Logs from the pod in the tekton-pipelines-resolvers isn't giving any additional information.

Please let me know if I'm missing anything

[shruthipuranik]

I have tried few other versions of Tekton as well like v0.49.0, v0.45.0, current nightly and so on. I get the same error.

[anton-livewyer]

We are facing the exact same error while trying to refer both tasks and pipelines as bundles stored in our own private registry (google artifact registry)

Task spec

kind: Task
metadata:
  name: demo-task
spec:
  steps:
    - image: ubuntu
      env:
      - name: HOME
        value: /tekton/home
      script: |
        #!/usr/bin/env bash
        [[ $HOME == /tekton/home ]]

Taskrun spec

kind: TaskRun
metadata:
  name: remote-task-reference
spec:
  serviceAccountName: default
  taskRef:
    params:
    - name: bundle
      value: us-central1-docker.pkg.dev/<project-name> /tekton-test/task:task
    - name: name
      value: demo-task
    - name: kind
      value: task
    - name: serviceAccount
      value: default

Service account spec

imagePullSecrets:
- name: regcred
kind: ServiceAccount
metadata:
  name: default
secrets:
- name: regcred

[jerop]

@shruthipuranik thank you for reporting the issue, and apologies for the delay -- will try to find someone to look into this issue

@anton-livewyer there's a small issue with your taskrun specification, you haven't specified the resolver field, could you try out this spec?

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: remote-task-reference
spec:
  serviceAccountName: default
  taskRef:
    resolver: bundles
    params:
    - name: bundle
      value: us-central1-docker.pkg.dev/<project-name>/tekton-test/task:task
    - name: name
      value: demo-task
    - name: kind
      value: task
    - name: serviceAccount
      value: default

[Yongxuanzhang]

Hi @shruthipuranik, thanks for reporting this bug! I have opened an PR to fix it. Unfortunately the service account approach doesn't work since the feature is implemented, and I believe the tests are also missing, error is also omitted as you mentioned so the root cause (the resolver doesn't have the permission to read ServiceAccount from the cluster) of your case is also hidden.

We will change the bundle resolver to pass in a secret name instead of a service account name. So you don't need to create a SA and attach secret to it.

If you want to quick fix, you could grant SA read permission to resolver.
i.e. change this

pipeline/config/resolvers/200-clusterrole.yaml

Line 33 in 695311e

resources: ["secrets"]

to

resources: ["secrets", "serviceaccounts"] 

and patch this yaml to your cluster. But this is not recommended due to security concerns. Will keep you updated about the fix PR as well.
If you have any suggestions, feel free to let me know. Thanks!

[shruthipuranik]

Thank you. I'm gonna wait for this PR to be merged and be available in the latest tekton release. Do you have any estimate for how long it would take?

[Yongxuanzhang]

Thank you. I'm gonna wait for this PR to be merged and be available in the latest tekton release. Do you have any estimate for how long it would take?

It will be this month: https://github.com/tektoncd/pipeline/milestone/74.

[shruthipuranik]

Hi @Yongxuanzhang,

I tried out your temporary fix with patching the cluster role yaml. But I see the same error again. Any suggestions?

[Yongxuanzhang]

Hi @Yongxuanzhang,

I tried out your temporary fix with patching the cluster role yaml. But I see the same error again. Any suggestions?

Oh sorry, I missed this comment. I wonder if you're able to install Tekton's latest main? The fix PR is merged

[shruthipuranik]

Would this be part of the latest nightly/staging release? Or is it already merged to the latest official release?

And instead of service account in the params, I pass the secret to make this work as follows?

kubectl apply -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: sample-pipelinerun
spec:
  serviceAccountName: sa-artifact-registry
  pipelineRef:
    resolver: bundles
    params:
    - name: secret
      value: artifact-registry
    - name: bundle
      value: us-west2-docker.pkg.dev/<GCP project name>/<repository name>/<bundle name>:latest
    - name: name
      value: <pipeline name>
    - name: kind
      value: pipeline
EOF

[Yongxuanzhang]

Would this be part of the latest nightly/staging release? Or is it already merged to the latest official release?

And instead of service account in the params, I pass the secret to make this work as follows?

kubectl apply -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: sample-pipelinerun
spec:
  serviceAccountName: sa-artifact-registry
  pipelineRef:
    resolver: bundles
    params:
    - name: secret
      value: artifact-registry
    - name: bundle
      value: us-west2-docker.pkg.dev/<GCP project name>/<repository name>/<bundle name>:latest
    - name: name
      value: <pipeline name>
    - name: kind
      value: pipeline
EOF

It should be in next official release. I wonder if you can install the latest main (clone the repo, and run kubectl apply -f config) to test it out? It works from my side with GCP's artifact registry.

Yes, this example is correct, as long as the artifact-registry is the name of the secret in your default namespace

[vdemeester]

It should be in next official release. I wonder if you can install the latest main (clone the repo, and run kubectl apply -f config) to test it out? It works from my side with GCP's artifact registry.

It would be ko apply -R -f config 🙃 . That said, it should be in the nightly releases.

[shruthipuranik]

Tried this out with the nightly release and it's working!

[Yongxuanzhang]

Tried this out with the nightly release and it's working!

Cool! I will close this issue, and if you have any suggestions feel free to reopen it.