Issue when trying to access our GitHub repo

[delpic]

Expected Behavior

I am expecting that the process running should access the GitHub repo I defined and clone it. Our GitHub account is an Enterprise account.

Actual Behavior

Getting the following error from the credential-initializer container:

invalid value "elastic-github-ssh-key=github.com" for flag -ssh-git: exit status 1
Usage of /ko-app/creds-init:
  -basic-docker value
    	List of secret=url pairs.
  -basic-git value
    	List of secret=url pairs.
  -docker-cfg string
    	Docker .dockercfg secret file.
  -docker-config string
    	Docker config.json secret file.
  -ssh-git value
    	List of secret=url pairs.

Steps to Reproduce the Problem

Here are the yaml files I created:

apiVersion: v1
kind: Secret
type: kubernetes.io/ssh-auth
metadata:
  name: elastic-github-ssh-key
  annotations:
    tekton.dev/git-0: github.com
data:
  ssh-privatekey: XXXXXXXXX
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-github-build-bot
secrets:
  - name: elastic-github-ssh-key
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: test
spec:
  resources:
    inputs:
      - name: repo
        type: git
  steps:
    - name: run-test
      image: golang:1.14-alpine
      workingDir: /workspace/repo/src
      command: ["go"]
      args: ["test"]

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: git-pull
spec:
  type: git
  params:
    - name: url
      value: git@github.com:XXXXX/elastic.git
    - name: revision
      value: master

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: testrun
spec:
  serviceAccountName: elastic-github-build-bot
  taskRef:
    name: test
  resources:
    inputs:
      - name: repo
        resourceRef:
          name: git-pull

Deployed in this order:

1. kubectl apply -f git-task.yaml
2. kubectl apply -f git-pipeline.yaml
3. kubectl apply -f git-taskrun.yaml

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

v0.12.1

[delpic]

Can provide further info as needed. Thanks in advance.

[ghost]

Hey @delpic thanks for the issue report!

There are a couple things you can try to see if they remedy the problem:

• First, do you make use of a corporate VPN? If so, another user mentioned here that being disconnected from the VPN can cause this error.
• Second, does it work if you specify a known_hosts file in the Secret? This shouldn't be required but has been mentioned as another possible source of this error.

Essentially the problem seems to appear when there is network connectivity trouble of some kind. So that might be the first thing to try debugging.

If you continue to hit the problem, could you provide the pod YAML that backs the TaskRun? The TaskRun YAML should show you the Pod's name under the podName field. Then you can get the pod's YAML with something like:

kubectl get -o yaml pod testrun-pod-<random letters>

[delpic]

Thanks for the follow up @sbwsg!

I think this issue most likely falls into the second scenario that you mentioned as our GitHub is not on-prem.

I tried specifying known_hosts into the Secret. I am now getting the following error:

{"level":"error","ts":1590543652.4407206,"caller":"git/git.go:41","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force master]: exit status 128\nssh: connect to host github.com port 22: Operation timed out\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:41\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:116\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:203"}

I tried troubleshooting further, but no luck.

Here is the pod YAML that you requested:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 192.168.56.134/32
    cni.projectcalico.org/podIPs: 192.168.56.134/32
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"tekton.dev/v1beta1","kind":"Task","metadata":{"annotations":{},"name":"test","namespace":"default"},"spec":{"resources":{"inputs":[{"name":"repo","type":"git"}]},"steps":[{"args":["test"],"command":["go"],"image":"golang:1.14-alpine","name":"run-test","workingDir":"/workspace/repo/src"}]}}
    pipeline.tekton.dev/release: devel
    tekton.dev/ready: READY
  creationTimestamp: "2020-05-27T01:38:36Z"
  labels:
    app.kubernetes.io/managed-by: tekton-pipelines
    tekton.dev/task: test
    tekton.dev/taskRun: testrun
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cni.projectcalico.org/podIP: {}
          f:cni.projectcalico.org/podIPs: {}
    manager: calico
    operation: Update
    time: "2020-05-27T01:38:37Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:pipeline.tekton.dev/release: {}
          f:tekton.dev/ready: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
          f:tekton.dev/task: {}
          f:tekton.dev/taskRun: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"82a0534b-9dbf-4771-80e3-2ec063eaed82"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:containers:
          k:{"name":"step-git-source-git-pull-hprct"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"HOME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"TEKTON_RESOURCE_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:ephemeral-storage: {}
                f:memory: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tekton/downward"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/home"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/results"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/tools"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/workspace"}:
                .: {}
                f:mountPath: {}
                f:name: {}
            f:workingDir: {}
          k:{"name":"step-run-test"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"HOME"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:ephemeral-storage: {}
                f:memory: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tekton/home"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/results"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/tools"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/workspace"}:
                .: {}
                f:mountPath: {}
                f:name: {}
            f:workingDir: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:initContainers:
          .: {}
          k:{"name":"credential-initializer"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"HOME"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tekton/creds-secrets/elastic-github-ssh-key"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/home"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/results"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/workspace"}:
                .: {}
                f:mountPath: {}
                f:name: {}
          k:{"name":"place-tools"}:
            .: {}
            f:command: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tekton/tools"}:
                .: {}
                f:mountPath: {}
                f:name: {}
          k:{"name":"working-dir-initializer"}:
            .: {}
            f:args: {}
            f:command: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tekton/home"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/tekton/results"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/workspace"}:
                .: {}
                f:mountPath: {}
                f:name: {}
            f:workingDir: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:serviceAccount: {}
        f:serviceAccountName: {}
        f:terminationGracePeriodSeconds: {}
        f:volumes:
          .: {}
          k:{"name":"tekton-internal-downward"}:
            .: {}
            f:downwardAPI:
              .: {}
              f:defaultMode: {}
              f:items: {}
            f:name: {}
          k:{"name":"tekton-internal-home"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"tekton-internal-results"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"tekton-internal-secret-volume-elastic-github-ssh-key-96llp"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:secretName: {}
          k:{"name":"tekton-internal-tools"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
          k:{"name":"tekton-internal-workspace"}:
            .: {}
            f:emptyDir: {}
            f:name: {}
    manager: controller
    operation: Update
    time: "2020-05-27T01:38:40Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:initContainerStatuses: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"192.168.56.134"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: "2020-05-27T01:40:53Z"
  name: testrun-pod-z4bmn
  namespace: default
  ownerReferences:
  - apiVersion: tekton.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TaskRun
    name: testrun
    uid: 82a0534b-9dbf-4771-80e3-2ec063eaed82
  resourceVersion: "5486479"
  selfLink: /api/v1/namespaces/default/pods/testrun-pod-z4bmn
  uid: 824e496c-4af5-41a3-b275-d1b777b3e1fe
spec:
  containers:
  - args:
    - -wait_file
    - /tekton/downward/ready
    - -wait_file_content
    - -post_file
    - /tekton/tools/0
    - -termination_path
    - /tekton/termination
    - -entrypoint
    - /ko-app/git-init
    - --
    - -url
    - git@github.com:XXXXXX/elastic.git
    - -revision
    - master
    - -path
    - /workspace/repo
    command:
    - /tekton/tools/entrypoint
    env:
    - name: HOME
      value: /tekton/home
    - name: TEKTON_RESOURCE_NAME
      value: git-pull
    image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.12.1@sha256:d82c78288699dd6ee40c852b146cb3bd89b322b42fb3bc4feec28ea54bb7b36c
    imagePullPolicy: IfNotPresent
    name: step-git-source-git-pull-hprct
    resources:
      requests:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
    terminationMessagePath: /tekton/termination
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /tekton/downward
      name: tekton-internal-downward
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: elastic-github-build-bot-token-4sghc
      readOnly: true
    workingDir: /workspace
  - args:
    - -wait_file
    - /tekton/tools/0
    - -post_file
    - /tekton/tools/1
    - -termination_path
    - /tekton/termination
    - -entrypoint
    - go
    - --
    - test
    command:
    - /tekton/tools/entrypoint
    env:
    - name: HOME
      value: /tekton/home
    image: golang:1.14-alpine
    imagePullPolicy: IfNotPresent
    name: step-run-test
    resources:
      requests:
        cpu: "0"
        ephemeral-storage: "0"
        memory: "0"
    terminationMessagePath: /tekton/termination
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: elastic-github-build-bot-token-4sghc
      readOnly: true
    workingDir: /workspace/repo/src
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - -ssh-git=elastic-github-ssh-key=github.com
    command:
    - /ko-app/creds-init
    env:
    - name: HOME
      value: /tekton/home
    image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init:v0.12.1@sha256:6266d023172dde7fa421f626074b4e7eedc7d7d5ff561c033d6d63ebfff4a2f2
    imagePullPolicy: IfNotPresent
    name: credential-initializer
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/creds-secrets/elastic-github-ssh-key
      name: tekton-internal-secret-volume-elastic-github-ssh-key-96llp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: elastic-github-build-bot-token-4sghc
      readOnly: true
  - args:
    - -c
    - mkdir -p /workspace/repo/src
    command:
    - sh
    image: busybox@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
    imagePullPolicy: IfNotPresent
    name: working-dir-initializer
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: elastic-github-build-bot-token-4sghc
      readOnly: true
    workingDir: /workspace
  - command:
    - cp
    - /ko-app/entrypoint
    - /tekton/tools/entrypoint
    image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint:v0.12.1@sha256:7f3db925f7660673a74b0e1030e65540adea36fe361ab7f06f5b5c47cdcef47d
    imagePullPolicy: IfNotPresent
    name: place-tools
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tekton/tools
      name: tekton-internal-tools
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: elastic-github-build-bot-token-4sghc
      readOnly: true
  nodeName: ip-10-1-3-254.us-east-2.compute.internal
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: elastic-github-build-bot
  serviceAccountName: elastic-github-build-bot
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: tekton-internal-workspace
  - emptyDir: {}
    name: tekton-internal-home
  - emptyDir: {}
    name: tekton-internal-results
  - name: tekton-internal-secret-volume-elastic-github-ssh-key-96llp
    secret:
      defaultMode: 420
      secretName: elastic-github-ssh-key
  - emptyDir: {}
    name: tekton-internal-tools
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations['tekton.dev/ready']
        path: ready
    name: tekton-internal-downward
  - name: elastic-github-build-bot-token-4sghc
    secret:
      defaultMode: 420
      secretName: elastic-github-build-bot-token-4sghc
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-05-27T01:38:39Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-05-27T01:40:53Z"
    message: 'containers with unready status: [step-git-source-git-pull-hprct step-run-test]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-05-27T01:40:53Z"
    message: 'containers with unready status: [step-git-source-git-pull-hprct step-run-test]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-05-27T01:38:36Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://38e064440bcd90d6e5ae6b7483d1642221c7cea45af48abb49e9f7c1534eebb9
    image: sha256:0e4ac3b9cfdec8ede4e3939fc8176f16ccd60a0a9bb252c957582dce2f079cfe
    imageID: docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:d82c78288699dd6ee40c852b146cb3bd89b322b42fb3bc4feec28ea54bb7b36c
    lastState: {}
    name: step-git-source-git-pull-hprct
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://38e064440bcd90d6e5ae6b7483d1642221c7cea45af48abb49e9f7c1534eebb9
        exitCode: 1
        finishedAt: "2020-05-27T01:40:52Z"
        message: '[{"key":"StartedAt","value":"2020-05-27T01:38:42Z","resourceRef":{}}]'
        reason: Error
        startedAt: "2020-05-27T01:38:39Z"
  - containerID: docker://f2f231160a83c7abee6a9f5c9d0720275e584db5762f1d9e49078e9f79c5bf9c
    image: golang:1.14-alpine
    imageID: docker-pullable://golang@sha256:d3a08e6a81ef8f25c7b9f4b8f2990fe76790f057ef7f8053e8884511ddd81756
    lastState: {}
    name: step-run-test
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: docker://f2f231160a83c7abee6a9f5c9d0720275e584db5762f1d9e49078e9f79c5bf9c
        exitCode: 1
        finishedAt: "2020-05-27T01:40:52Z"
        message: '[{"key":"StartedAt","value":"2020-05-27T01:40:52Z","resourceRef":{}}]'
        reason: Error
        startedAt: "2020-05-27T01:38:39Z"
  hostIP: 10.1.3.254
  initContainerStatuses:
  - containerID: docker://071f0fb9b78a7bfbe634c33c833cba972b8b23793ba2bf860c2e62c18db5d453
    image: sha256:a65df427c4954dc3cf4cf5af061509fa36dc2c4a1f4457b5de47d3ce0d17a24d
    imageID: docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/creds-init@sha256:6266d023172dde7fa421f626074b4e7eedc7d7d5ff561c033d6d63ebfff4a2f2
    lastState: {}
    name: credential-initializer
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://071f0fb9b78a7bfbe634c33c833cba972b8b23793ba2bf860c2e62c18db5d453
        exitCode: 0
        finishedAt: "2020-05-27T01:38:37Z"
        reason: Completed
        startedAt: "2020-05-27T01:38:37Z"
  - containerID: docker://9e1e84a0db42a26a4053ddc0158211e127e332e02c3858ca3d679d3c88b4a15a
    image: sha256:be5888e67be651f1fbb59006f0fd791b44ed3fceaa6323ab4e37d5928874345a
    imageID: docker-pullable://busybox@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
    lastState: {}
    name: working-dir-initializer
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://9e1e84a0db42a26a4053ddc0158211e127e332e02c3858ca3d679d3c88b4a15a
        exitCode: 0
        finishedAt: "2020-05-27T01:38:37Z"
        reason: Completed
        startedAt: "2020-05-27T01:38:37Z"
  - containerID: docker://d9bbb4a6666bddefebe1d68699c35c7f8c07220ce743d4ef1e0f155ab53d1dbb
    image: sha256:5dd2894996c89ac136c71624b684a9f13fbb2354056343ea4cb6e159cb664b95
    imageID: docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:7f3db925f7660673a74b0e1030e65540adea36fe361ab7f06f5b5c47cdcef47d
    lastState: {}
    name: place-tools
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://d9bbb4a6666bddefebe1d68699c35c7f8c07220ce743d4ef1e0f155ab53d1dbb
        exitCode: 0
        finishedAt: "2020-05-27T01:38:38Z"
        reason: Completed
        startedAt: "2020-05-27T01:38:38Z"
  phase: Failed
  podIP: 192.168.56.134
  podIPs:
  - ip: 192.168.56.134
  qosClass: BestEffort
  startTime: "2020-05-27T01:38:36Z"

Thanks again for your help!

[delpic]

@sbwsg, my colleague pointed my to:

#1271

seems like this maybe the issue I am currently running into at the moment. Going to try the fix that you mentioned in the above issue.

[delpic]

Tried the fix, but same result. Though there is a symlink error after introducing the fix from above:

$ kubectl logs testrun-pod-2cqx6 --all-containers 
{"level":"info","ts":1590583900.3125732,"caller":"creds-init/main.go:44","msg":"Credentials initialized."}
{"level":"warn","ts":1590583903.8721306,"caller":"git/git.go:196","msg":"Unexpected error: creating symlink: symlink /root/.ssh /root/.ssh: file exists"}
{"level":"error","ts":1590584034.904626,"caller":"git/git.go:41","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force master]: exit status 128\nssh: connect to host github.com port 22: Operation timed out\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:41\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:116\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:203"}
{"level":"fatal","ts":1590584034.9047163,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [master]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:203"}
2020/05/27 12:53:55 Skipping step because a previous step failed

[ghost]

Thanks for the great info, this is all very helpful as we debug. So, it appears right now like the credentials are being initialized as we expect them to be. We can tell this because: a) the creds-init initContainer has both the correct arg (-ssh-git=elastic-github-ssh-key=github.com) and mounted secret (mountPath: /tekton/creds-secrets/elastic-github-ssh-key, name: tekton-internal-secret-volume-elastic-github-ssh-key-96llp) and b) the warning creating symlink: symlink /root/.ssh /root/.ssh: file exists implies to me that the Git PipelineResource is doing the right thing, which is symlinking from /tekton/home/.ssh to /root/.ssh.

You mentioned trying the fix from #1271 but could you confirm what the fix was you tried? If you flipped the disable-home-env-overwrite feature flag then I first suggest flipping it back to "false" so that we're debugging from the same baseline as before.

Here's what I suggest trying next: Run a TaskRun that will give us more insight into what the initialized credentials look like. Here's an example:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: debug-testrun
spec:
  serviceAccountName: elastic-github-build-bot
  params:
  - name: url
    value: git@github.com:xxxxx/elastic.git # TODO: update this url
  taskSpec:
    params:
    - name: url
      type: string
    steps:
    - name: check-creds
      image: alpine/git:v2.24.3
      script: |
        #!/usr/bin/env ash
        set -xe
        id
        for f in /tekton/home/.ssh/* ; do
          cat $f
        done
        ln -s /tekton/home/.ssh /root/.ssh
        git clone $(params.url)

Update the url param in the above TaskRun and then kubectl apply it. Once that's done, get the log output from the TaskRun's Pod. You'll need the pod's name from the podName field from the TaskRun YAML again and then you can run kubectl logs -f <pod name>.

Once you have the pod logs, take a look at the SSH private key that was printed. Does it match the original you have? If not then this is likely the problem - the key has not survived the translation to base64 in the Secret and back again. If it does match what you have locally then were there any errors from the git clone at the end of the TaskRun? If not then I think the problem is that the SSH key doesn't have permission to access the repo in question. If it does then something else, very specific to the Git PipelineResource, is going wrong and we'll need to debug further.

I'm very curious to hear the outcome from this debugging, please keep me posted. Remember to sanitize the log output of any credentials or other sensitive information if you want to post it here to show findings.

[delpic]

Thanks @sbwsg!

Regarding #1271, I had set the disable-home-env-overwrite to True, but I reverted it back to False.

I ran the TaskRun yaml you provided. Here is the error I received:

Cloning into 'elastic'...
ssh: connect to host github.com port 22: Operation timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I can confirm that the private key does match up with the original. As far as permissions go, I added this as a deploy key, but did not give it write access as it will only be used to clone.

So something specific to the Git PipelineResource then?

[ghost]

Hmmm this is really odd. If the SSH key is working then I would have assumed that the git clone should have succeeded.

ssh: connect to host github.com port 22: Operation timed out

This seems very much like a network problem to me. To double check this I suggest running a new TaskRun with a script that does sleep 3000, then kubectl exec into the container and try wgetting github.com. Here's another example:

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: debug-testrun
spec:
  serviceAccountName: elastic-github-build-bot
  params:
  - name: url
    value: git@github.com:xxxx/elastic.git # TODO: update this url
  taskSpec:
    params:
    - name: url
      type: string
    steps:
    - name: check-creds
      image: alpine/git:v2.24.3
      script: |
        #!/usr/bin/env ash
        set -xe
        id
        ln -s /tekton/home/.ssh /root/.ssh
        sleep 3000

Then kubectl apply this taskrun and run the following command:

kubectl exec debug-testrun-pod-<random letters> -c step-check-creds -it /bin/ash

This will put you inside the container created for the Task's Step. Then you can run something like this:

cd /tekton/home
wget github.com

This should get you a response like this:

Connecting to github.com (140.82.113.3:80)
Connecting to github.com (140.82.113.3:443)
saving to 'index.html'
index.html           100% |**************************************************************************************************************|  128k  0:00:00 ETA
'index.html' saved

If that succeeds then try something like this:

ssh git@github.com

Which should net you a response like this:

PTY allocation request failed on channel 0
Hi sbwsg/private! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

Then finally run exit to quit the container's shell and delete the TaskRun to clean up the sleeping process. Let me know what happens when you try these commands, I'm very puzzled.

Also, another question: which platform are you running on? Is this locally in minikube, on something like gke / aws, on something like openshift, or a bare metal k8s cluster?

[delpic]

Thanks @sbwsg. Doing some further debugging on our end, it seems that we are blocking port 22 on the network that the clusters are running on. Our platform runs on aws, so we will be updating to allow traffic through.

Thank you for all the help you have provided!!