[Bug Report] Cannot use ssh private key in task steps

[chengjingtao]

Expected Behavior

We could use ssh private key in task steps in order to execute command like git pull or git push

Actual Behavior

When using command like git pull in task steps , it shows errors

[config] Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.7) ...
[config] Setting up git (1:2.17.1-1ubuntu0.4) ...
[config] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[config] Processing triggers for ca-certificates (20180409) ...
[config] Updating certificates in /etc/ssl/certs...
[config] 0 added, 0 removed; done.
[config] Running hooks in /etc/ca-certificates/update.d...
[config] done.
[config] Host key verification failed.
[config] fatal: Could not read from remote repository.
[config]
[config] Please make sure you have the correct access rights
[config] and the repository exists.

container step-config has failed  : Error

Steps to Reproduce the Problem

Flow example https://github.com/tektoncd/pipeline/blob/master/examples/taskruns/taskrun-git-ssh.yaml and just change the args of config step

taskrun-git-ssh.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: test-git-ssh
  annotations:
    tekton.dev/git-0: github.com
type: kubernetes.io/ssh-auth
data:
  # Generated by:
  # cat id_rsa | base64 -w 0
  # This deploy key has read-only permissions on github.com/knative/build
  ssh-privatekey: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKSndJQkFBS0NBZ0VBc0YrY3ZIdUlKRjBwNWZNVDBUUVZhMWpTZVBNVlhtUWhGa1F0andsamJFSXVKaVh5CmVrU1I0WUtDbFZIZVo0M2VOa2p5SzFwTDVmVnZnY3J2bkVUMXE5dmFmOThtNzdHWjZUdnJGY0RyZjViZDFIZGUKTU1MTFZRRU8vN1lkcEJFeVJ1UjA4RXRzbWp0RUVaNERVZnBUWGQ5RnAyTU5UWTYwVzFpa0p0Mm53dGhWVlF0OApubFcxTTRvc1d0SHE1OVFDbnhqUkVmNzNhdHdtUTZ6enlXcCtJM3ZBRFRJUmZpTWcvSjc0UDJHOWlwL0U1U0YrCnNuRmRnU2x0cjhyd0tmTE5jTDVYandOSE53aHErQ2NJRHBUZGhuVTBUaEFZNlRsa25KWFJudXRicGNRajA4MjAKMyt1aXRxaXhJcnIwQmkrU2NDQ21GNERxZGdRUXIwRzdieXRKemJkdjJOL3dGbGFpWmVOTWtXSVd5TzRpdnU4TQoxbGUyajVHeDZFaVQ0TG5sNitoRHd3aXozUGR4Y1I0S0VxZzVqRmQ3QjNDcHQ0cUswRDFHVVhnMmx1ZHRsMU40CjVoNUJaNEZZcWF0VU1LQkVjeHd4ekZVVE1iNFZmdFhuMEJOSHkyVXlxUkx6VnBMZWgyRHRYcmtaZnRnTlFCeFAKNkNKc2FnYkpuRmZ1Mk1IaTh5VkE3UFZnUXExS3hPWnF0c0JKb3RvSzJXb0lONldpbnl0d0lwUFhsblJrZ3duYgo4dVBZdmhoZDB0ZVllZ2pYNW85azcwaDhFOEFpZGtqaFpKRkwzakRjc0NFdGJQSWtBMHp3bCs0UFN5aWV2UXpVCldFTkFTSzhXc1NxSk40bUtNSEJMbDU2WWZpTnBJNXVEUGx6eC9hS3hnQis2RExYL25GRzB0Y2ZTYlJFQ0F3RUEKQVFLQ0FnQnRjQlpZR2FxakxhK213RXFHbi9PK2RSRzcvTkZBalk2K096V3RzRnlodXZyWnJ1Z2pGOU1SQTZNMQpOZVJ6aXVMYitpMVIyeWlIR3VEQ3Y0SW9DOTRJY2xYRXpGbGpsMmJ5QXJ6ZTlhTHF1MWtQMGJWL09nOWl3bEpsCk5zSnpvRHY3SXh5d1VZeWtzcXVNNXVnV2dPWWp0RHVMSG5vTnFzT2Nmb2IwbXNyU3F0dGY5czJ2ald6dSt3aUUKbmdaTkxROG5Xd1czVGVhRnh1NXpSYWpMVEtuU0tLSkZUZER1M0t0R05QbkNra3BDN3RvRWR1RWZwaG9uMytYdwprMUtzSkwxMi9KUmw1OW5UZXdTc3o4TS9FUDNYeFhmY0RENzQwQ1R1eUVPV2dXR1BuUUhDRmJ6cXlGSmhWcEQ3CkJOZ1Awb3JoeWxCayt4Z0FmSXRCdzk2aW4xQ2VMZHZKek9TdDNXSUVrY0w3ZVB0bkpzWVpBa1J4eFU1NWF6ajEKbFJHNWRWN010ZzgwWXJTcDdoYXIxamhTQ0JjVzZMUHdueTdXK1psWEd6VG9Gb1o2clIwdytJdGRjNGpPSEhtTgpRVk9tWFd3SFJuZm5rQWZlNXdDQkZQbXpMT2tYbnloWjdaNWxmVlJQaE9lT0hlV09NeHJvWFZoRmJPcTFxemo1Cm85a1dMVGxwTk42YXhrOW41R1N6NlZDVDRHSW1iOUsyWmpQUklOMktQZ3Vxd0RRSFVmd0ViZ0NnTkVUOG5wc2UKelYxRnRpRUJaeVJPa2RwdDhDd2NrbGxHc0Nab0dIa3Z1OXlRSytMUldvV2xJS041WGJXYVljaW01VVhIZlIzUgp4ZGx2QmgxNDl2MW02Y3haMWpzNUFJM09Xb0lpaXlaWjdKUGUxbUIycDE2dUlsUGhsUUtDQVFFQTRNZmNPRVluCkdrQVEzaEdKMm5ia0ZzZ1hJemJrQkQvanIrY1JHd1pjeitBZ2x0cXFZRnQ2Rzh3eElBMGd2V1BLNDFhR1UzU1AKdHVjUEF6VjVEYlR3WklFdjVpY3kycHJlbHJ1ZTB4RkV4WWRZNFFCVGRiVFM4TE9ZWncxTEZpTGRXb1VKVlhiWApTaE4rSktpSDVpM3J1K2dwTHArT0ZHUSthWFdQSFp0TjdjTUZidEpUUkNQUlZqUkVHb25qejY3c2NNcWxZNjAyCkYxaFlURGRkTGNnNnhiN0M3ZmdXc25pdEh4ZWY5UHZFUlZSUWdyRW9RT3lOcFczL0VOU1RQN1IzTzhnNnNHTXYKUXI4SDd1a3RGUlAxUVRqSGNvdmtrdm9aaERHOHRGREd4T0FhbFMxK2phQVpBMmNVdEI4VU5ueHpqMFJFNE43VAp3OHZST3NHbjJ2aG9vd0tDQVFFQXlONmV1d0FFdllpanVlV21pYWNJTUQ0bExaL3JjLzF6T0RUQ3grWm05SHBCCjZyUGMvMG8yR3lGV2Y2aVN3UjhOWlVyM2dvREh6ck8vTHdZUTNyUEhhZFpkRGRFVWh5UXBHeDFuOG9ONXhSU0sKVkVkdnVESkF4SnhHdXRWYk9sNzlDUitadjBqa3hGY2xaOTBaSGdhVEl5TnBlK1dIcTI3Mm1oaENaOXNmK0RTMgppM2V2MEkrQ0t6eXppV2NUdzJKL0ROSlpIN0ZnUlc0U1M3OXBLcDFPOXk0a2hQMk0xQU9nVUp4eFBta0xMc05oCk45dlB0QmpXMENZZmxiei9zWjBYZEVtL1hEUE9xNWVXc0tsZm5XcnpzR3ZZVHRvZkxpN1pVekJhMjRyc1AzT3UKT0JqeFVQWDBhTnQ2bVlOSWVSQkNkVk9BZlFsbGFqSW1hVENNNXZQcXV3S0NBUUF1RXFrZjI3aytGeTJkNXVoagpvQVRtYW9MYmsrK3VzMDIvQzFTbFBtdWRyQ25oNGRZVVJBdkdVTmNzNDZQM1RkSVRRUFB0ZlRITDRwWmdLUUN6Ci80ZGFnbVJILzNiYitleHY1dXNLQmpzYlpiK2c5c2tra2ppK1g0RjRkZkFUREhoa2ltbzhXMHlMZ1cxVXBEcWUKd29EVUZpVUN5bTk3M0lReVo0YzdJVThhMGVOMk1LUWlqbGpsRzNkdUhSOVZ2VnYzVG9JUjZUbnlGUWljTHFqcApybVhsakZFMko2aTNVaWJoL1p5cXdVUWNsY2llMFNNWUU2U0E2UWZqOGo1d1MxUTVLN1JqM1NDZHhMd2lpeWJUCmNERFArb3hlL1MrUHRMV3dNK1dpNi8raDhQRmg3RjVKTnl5SlZ3S1dYTnh1d3BtSkh3d2pMMmttYzh0aUNMbVkKRjF5VEFvSUJBRWpJdWRWaW5WVWNKTTlnVW1jQWVpd1k2Sm5PbjU5bzBBNG11QWw5Q0FOQUpaQm8yZ1E3SEwzWgo1NkRjTVVqKytuaWdvR3pPUXhnMzgyZVRtQnRLSTFETFZ1NkJ1eUY0emRoQVlJRDBlUFFiakVBYm5uU3M2T0VWCkMxLzVuVlBhUlcrLzNwTDhVOEI5SDFXeVBMNnZmVE0zU3FxVk1nTEJjMlBFdEpNelNETkV6bnhjd3YwY2N2czkKeElSZU1FQVUrWnBvd2RpRkNqQStRU3puVDdFY1RSZXR4WmZXMCtlaTVMMHc0cXZObmpxMG13NkZmSTV5amF2ZwpsSU5LZDVBaEx2VjVIM2pHWVBXMk9Pa01obFBoQkNvQitlQWdrem5GeWdPbHV0WmdVVmh3ajBsT2FHcUFwSzQyCkVRZDlGWTZnZ0tSSXZIOFNINGhLWWJhdDIvQjNvNjhDZ2dFQVNnUGZubVdZcXpQTlpwYXh4VFZ3NXl0cXovMGcKUHlhcW41elhDSHFsdDluNkZFYWdRVjE4bjl0bUZzTWtnOFd4MHp1TjY5ajNvcXVDaWVDSkVRTTFDR05GUWg2MQpEV0xrd210R1FPNXZxc0VhcXpLZk9yWktnYTJ5L3BuejY2SkNlUW83UlJEdVVZVHpPY3NpQmtxNmNUNHFLQjY0ClBCVlVhQTROQkVCZlRoc1o5dDhpWHFDQWhUQnVlTDdhQ3pHV0ZjR1k5dndrZkJHNjNacURXbVlqZkZwRGlqSWIKL21ySnJVTTB0a09HcUhESk5aUWhpZXc1VHF6NWp3VElyTDY3YUdZNFpuNEVyODFlb3JOZnpEeDVnak1NYnoweQoyTGZoN1c0MFlWMDA4ZmZJTk5BbXA4VjNqay9qcWFUSFlWYlFQNGxhdTRIWkc5Q2g1Q1VpSmFvN2xBPT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
  # Generated by:
  # ssh-keyscan github.com | base64 -w 0
  known_hosts: Z2l0aHViLmNvbSBzc2gtcnNhIEFBQUFCM056YUMxeWMyRUFBQUFCSXdBQUFRRUFxMkE3aFJHbWRubTl0VURiTzlJRFN3Qks2VGJRYStQWFlQQ1B5NnJiVHJUdHc3UEhrY2NLcnBwMHlWaHA1SGRFSWNLcjZwTGxWREJmT0xYOVFVc3lDT1Ywd3pmaklKTmxHRVlzZGxMSml6SGhibjJtVWp2U0FIUXFaRVRZUDgxZUZ6TFFOblBIdDRFVlZVaDdWZkRFU1U4NEtlem1ENVFsV3BYTG12VTMxL3lNZitTZTh4aEhUdktTQ1pJRkltV3dvRzZtYlVvV2Y5bnpwSW9hU2pCK3dlcXFVVW1wYWFhc1hWYWw3MkorVVgyQisyUlBXM1JjVDBlT3pRZ3FsSkwzUktyVEp2ZHNqRTNKRUF2R3EzbEdIU1pYeTI4RzNza3VhMlNtVmkvdzR5Q0U2Z2JPRHFuVFdsZzcrd0M2MDR5ZEdYQThWSmlTNWFwNDNKWGlVRkZBYVE9PQo=
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-robot-git-ssh
secrets:
- name: test-git-ssh
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: test-taskrun-with-serviceaccount-git-ssh
spec:
  serviceAccount: test-robot-git-ssh
  inputs:
    resources:
    - name: gitssh
      resourceSpec:
        type: git
        params:
        - name: revision
          value: master
        - name: url
          value: git@github.com:tektoncd/pipeline.git
  taskSpec:
    inputs:
      resources:
      - name: gitssh
        type: git
    steps:
    - name: config
      image: ubuntu
      command: ["/bin/bash"]
     ######################   changes
      args: ["-c", "apt-get update && apt-get install -y git && git pull && cat /workspace/gitssh/README.md"]

2. kubectl create -f taskrun-git-ssh.yaml
3. tkn tr logs -f test-taskrun-with-serviceaccount-git-ssh
4. it prints errors

[config] Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.7) ...
[config] Setting up git (1:2.17.1-1ubuntu0.4) ...
[config] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[config] Processing triggers for ca-certificates (20180409) ...
[config] Updating certificates in /etc/ssl/certs...
[config] 0 added, 0 removed; done.
[config] Running hooks in /etc/ca-certificates/update.d...
[config] done.
[config] Host key verification failed.
[config] fatal: Could not read from remote repository.
[config]
[config] Please make sure you have the correct access rights
[config] and the repository exists.

container step-config has failed  : Error

Additional Info

I analysised the reason, It seems git clone will use /root/.ssh/config but not /builder/home/.ssh/config and use /root/.ssh/known_hosts but not /builder/home/.ssh/known_hosts.
SSH Credential save private key into folder of /builder/home, So , git command will not use private key correctly.

When I change args to

args: ["-c", "apt-get update && apt-get install -y git && mkdir -p /root/.ssh && chmod 700 /root/.ssh && cp -pv /builder/home/.ssh/known_hosts /root/.ssh/ && cp -pv /builder/home/.ssh/config /root/.ssh/config &&  cd /workspace/gitssh && git pull && cat /workspace/gitssh/README.md"]

It works well.

[vdemeester]

/kind bug

There is a small "hack" around that for git-init (see here). Maybe we should "add this hack" in creds-init, aka creating both files to make sure either the GitResource or a random git … command work.

/cc @chmouel

[jlewi]

I'm hitting a similar issue. It looks like the .ssh config is set to /tekton/home/.ssh.

It looks like an environment variable $HOME is set to point to /tekton/home/.ssh but it looks like this is insufficient to override the home directory set for the root user in /etc/passwd.

I guess one option would be to change the home directory in my docker image for the user /root but that feels kind of kludgy.

As far as I can tell there is no environment variable to explicitly set the path of the .ssh directory. The best I could come up with so far was creating a symbolic link

/tekton/home/.ssh -> /root/.ssh

[ghost]

Note: we've introduced a feature-flag to disable the defaulting /tekton/home $HOME directory. This will result in each Step/container dictating its own HOME env var and also in credentials being written to the user's actual home directory. It should, at the very least, result in the .ssh directory being correctly written to /root/.ssh. The feature-flag is stored in a ConfigMap in the tekton-pipelines namespace and is described here: https://github.com/tektoncd/pipeline/blob/master/config/config-feature-flags.yaml#L21-L30

And we have a plan to make this the default behaviour in an upcoming release: #2013