/run/host-services/ssh-auth.sock does not respect $SSH_AUTH_SOCK

[Frederick888]

• I have tried with the latest version of my channel (Stable or Edge)
• I have uploaded Diagnostics
• Diagnostics ID: D2C80A94-78DD-4FCA-BF2F-25831CF64FD8/20200130025252

Expected behavior

SSH agent socket specified via environment variable SSH_AUTH_SOCK is forwarded to containers when using --env SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock -v /run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock.

Actual behavior

Docker brings up a new ssh-agent and tries to read identities through it.

Information

• macOS Version: 10.15.2

Diagnostic logs

It only shows Diagnose succeeded though? Anyway I'm using 2.2.0.0.

Steps to reproduce the behavior

1. Set up GnuPG by creating at least 1 Authentication (sub)key
2. Run sudo mount -uw /
3. Edit /etc/bashrc or /etc/zshrc (depending on which login shell you are using) and add the following lines to the end of the file

if [[ -S "$HOME/.gnupg/S.gpg-agent.ssh" ]]; then
    export SSH_AUTH_SOCK="$HOME/.gnupg/S.gpg-agent.ssh"
fi

4. Reboot
5. Run docker run --rm --tty --interactive --env SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock -v /run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock debian bash
6. (In the container,) run apt update; apt install -y openssh-client; ssh-add -l; and it shows The agent has no identities.
7. (In host,) run ps aux | grep ssh-agent, a process /usr/bin/ssh-agent -l can be seen

This is particularly annoying to me as I'm using YubiKey and it's impossible for me to export my keys to ssh-agent.

Related: #410 #483

[cnjr2]

I am in exactly the same situation: On a mac, trying to use a gpg-agent with key in Yubikey.

[coehne]

Same here. Although I'm trying to forward the agent to use it for a SSH auth against a private GIT repo.

[coehne]

We solved the issue today. It was as simple as sticking 100% to the docker docs to forward the ssh-agent into the container. Use exactly the volume definitions and don't change the paths. For some reason it has to be like that. Maybe this helps you too, even though our problems were slightly different:

https://docs.docker.com/docker-for-mac/osxfs/#ssh-agent-forwarding

Note: the ssh-agent needs to be started beforehand (ssh-add) on the host machine.

[Frederick888]

@Dakicka I don't use macOS anymore but that sounds like a different issue. Sure if you've already got a running ssh-agent Docker will be able to reuse it, but in case you are using a foreign SSH agent protocol implementation, e.g. the one provided by GnuPG, AFAIK Docker is not able to forward it atm.

[brian-maloney]

Just wanted to say "me too" on this issue. Looks like Docker's SSH Agent forwarding support only forwards the real ssh agent, not whatever you have specified in SSH_AUTH_SOCK. I can work around for now by using another key but my office's standard is to use the GPG/Yubikey ssh agent.

[kdambekalns]

For me the documented way only works when I start Docker from my shell, using /Applications/Docker.app/Contents/MacOS/Docker. When docker is started during boot, it will not work, but simply tell me:

root@035851fd7c02:/application# ssh-add -l
The agent has no identities.

And if Docker is started from the shell, it only works as root inside the container, being a non-root user gives me:

beach@9f680436a91c:/application$ ssh-add -l
Error connecting to agent: Permission denied

Which makes sense, as it belongs to root:

beach@9f680436a91c:/application$ ls -la $SSH_AUTH_SOCK
srwxr-xr-x 1 root root 0 Mar 13 15:46 /run/host-services/ssh-auth.sock

[5n00p4eg]

Solution with hardcoded path not looks ideal, because needs modifications to cross-platform docker-compose files.

And yes, it not respect my active agent. I have few agents on my machine and want to control which one is used in container.

Next undocumented thing is how to add keys from host-machine.
/run/host-services/ssh-auth.sock is not avaliable on host machine. But, then how to connect ?
I found a way:
First of all, ssh-agent only created when you start docker container with this magic volume.
After that(and only) you can find pid of new ssh-agent process
ps -ax | grep ssh-agent
Then use
lsof $PID and look for unix file descriptor. For me its
/private/tmp/com.apple.launchd.CTFTsdvchB/Listeners

Then I can export SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.CTFTsdvchB/Listeners and ssh-add ~/my_key to it.

And then, after container restart, my container can use my keys to do it's job.
It's absolutely unusable and should be impoved.

[5n00p4eg]

Also I faced permission problem with the socket file.

My container runs with non-root user and ssh-add -l from that user gives me permission error.

So to fix this problem I did next things:

1. screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty - connect to VM's shell
2. chmod 777 /run/host-services/ssh-auth.sock

And it solves the problem.

[troisdiz]

In order to change the rights of /run/host-services/ssh-auth.sock you can also run :

• docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh to enter in the VM
• docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh -c 'chmod o+w /run/host-services/ssh-auth.sock'to make the change in one command

[AndrewRayCode]

Following the exact docker-compose instructions from ssh-agent-forwarding, I don't understand why it seems to have no effect. I am trying to clone a private git repository in the container.

On the host:

ssh git@github.com
Hi AndrewRayCode!

but in the container:

root@9f844a37de32:/data/app# ssh git@github.com
The authenticity of host 'github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.255.113' (RSA) to the list of known hosts.
git@github.com: Permission denied (publickey).

root@9f844a37de32:/data/app# ssh-add -l
The agent has no identities.

Edit At least one issue was that on mac, I was manually trying to start ssh agents on the host with eval $(ssh-agent -s), but mac automatically starts an ssh daemon on boot. Now after a restart of my computer, followed by a ssh-add -K on the host, i'm able to ssh git@github.com in the container, but I'm still not able to git clone, i'm getting fatal: Authentication failed

[zedtux]

Thank you @kdambekalns for sharing the idea of starting docker for mac from the terminal, this solved my issue!

So basically, the very first step was to ensure to have one and only one running ssh-agent which I solved by using the short script from http://mah.everybody.org/docs/ssh in my .zshrc file:

##### START Fix for ssh-agent #####
# Ref: http://mah.everybody.org/docs/ssh
SSH_ENV="$HOME/.ssh/environment"
function start_agent {
     echo "Initialising new SSH agent..."
     /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
     echo succeeded
     chmod 600 "${SSH_ENV}"
     . "${SSH_ENV}" > /dev/null
     /usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     #ps ${SSH_AGENT_PID} doesn't work under cywgin
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
     }
else
     start_agent;
fi
##### END Fix for ssh-agent #####

Then I had to kill all the running ssh-agent:

killall ssh-agent

And create a new session which started an ssh-agent instance.

Then start docker in a separated terminal with /Applications/Docker.app/Contents/MacOS/Docker as @kdambekalns told.

After Docker has started, I could start my docker-compose stack and got my identities accessible and refreshed in real-time (what I mean here is that I'm using Teleport to connect to my servers, and login in fills in the ssh-agent with new keys).

Thank you so much 🎉

[gzm55]

An portable way without knowing the magic path:

macos_ssh_agent_ops=( -v /run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock:ro -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock )

// only input password for the first time
docker "${macos_ssh_agent_ops[@]}" --rm -it \
   -v /path/id_rsa:/id_rsa:ro \
   -v /path/id_rsa.pub:/id_rsa.pub:ro \
   image-with-ssh-add \
   ssh-add /id_rsa

// reuse the agent from /run/host-services/ssh-auth.sock
docker "${macos_ssh_agent_ops[@]}" --rm -it    image-with-ssh ssh a-host-fqdn

The magic ssh-agent is replacing the named container in other ssh-agent sharing methods, and providing a global shared ssh-agent namespace.

[gzm55]

The mac os native way to get the system auth agent:

launchctl getenv SSH_AUTH_SOCK // print the magic socket path

SSH_AUTH_SOCK=`launchctl getenv SSH_AUTH_SOCK` ssh-add -l // list keys

[acidprime]

Solution with hardcoded path not looks ideal, because needs modifications to cross-platform docker-compose files.

And yes, it not respect my active agent. I have few agents on my machine and want to control which one is used in container.

Next undocumented thing is how to add keys from host-machine.
/run/host-services/ssh-auth.sock is not avaliable on host machine. But, then how to connect ?
I found a way:
First of all, ssh-agent only created when you start docker container with this magic volume.
After that(and only) you can find pid of new ssh-agent process
ps -ax | grep ssh-agent
Then use
lsof $PID and look for unix file descriptor. For me its
/private/tmp/com.apple.launchd.CTFTsdvchB/Listeners

Then I can export SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.CTFTsdvchB/Listeners and ssh-add ~/my_key to it.

And then, after container restart, my container can use my keys to do it's job.
It's absolutely unusable and should be impoved.

export SSH_AUTH_SOCK=`launchctl getenv SSH_AUTH_SOCK`

is slightly easier

[gzm55]

on old iterm2, launchctl getenv may not work due to this issue, and the work around by Enable multi-server daemon

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[kdambekalns]

/remove-lifecycle stale

[deepsweet]

Another YubiKey + S.gpg-agent.ssh user here.

The main problem is that macOS provides default /usr/bin/ssh-agent and defines its SSH_AUTH_SOCK in /System/Library/LaunchAgents/com.openssh.ssh-agent.plist. And when you overwrite that env var using a GPG agent SSH socket, something like SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket), it happens only on a particular shell level in userland. Started by macOS Docker doesn't know about that environment context and therefore sees only the default SSH_AUTH_SOCK which is essentially launchctl getenv SSH_AUTH_SOCK. That's why starting Docker manually from terminal helps (open -a Docker).

The solution is quite simple and a bit hacky at the same time: solve the above mess by having only one SSH_AUTH_SOCK on all levels.

First, create a custom launch agent ~/Library/LaunchAgents/gpg-agent.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>gpg-agent</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/MacGPG2/bin/gpgconf</string>
    <string>--launch</string>
    <string>gpg-agent</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

launchctl load -F ~/Library/LaunchAgents/gpg-agent.plist

This will automatically launch GPG agent.

⚠️ Mind the gpgconf path: it might be different if you're using GPG from Homebrew, in my case it's the one from GPG Suite.

Second, overwrite and kinda ruin that default macOS SSH agent (it shouldn't be a real problem since we are going to replace it with GPG agent SSH solution in any way) by creating another ~/Library/LaunchAgents/link-ssh-auth-sock.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>link-ssh-auth-sock</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/sh</string>
    <string>-c</string>
    <string>/bin/ln -sf $HOME/.gnupg/S.gpg-agent.ssh $SSH_AUTH_SOCK</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

launchctl load -F ~/Library/LaunchAgents/link-ssh-auth-sock.plist

Now it's time to remove that custom SSH_AUTH_SOCK from your shell config because the above launch agent literally symlinks the GPG agent SSH socket to the default macOS one on quite a low level before Docker/SourceTree/whatever app is launched.

Reboot and verify:

ls -l $SSH_AUTH_SOCK
ssh-add -l
gpg -K

Works for me with VSCode Remote Container inside of a Docker.

Heavily inspired by this.

[cameri]

@deepsweet Your solution partially worked for me. I was able to call ssh-add -l on a Docker container and see the SSH key from my Yubikey. Yay!
However, gpg -K did not work. I keep getting connect(5, AF=1 "/home/node/.gnupg/S.gpg-agent.host", 36): Connection refused

I'm running macOS Big Sur, gpg 2.2.27 locally and gpg 2.2.12 on the remote.

I'm mounting $HOME/.gnupg/S.gpg-agent.extra (host) to $HOME/.gnupg/S.gpg-agent.host (remote).
I'm running socat as root on the remote like:
# socat UNIX-LISTEN:$HOME/.gnupg/S.gpg-agent,fork,mode=660,user=${USERNAME},group=${USERNAME} UNIX-CONNECT:/home/${USERNAME}/.gnupg/S.gpg-agent.host where $USERNAME is the user on the remote.

Permissions on the local S.gpg-agent.extra socket are 700, owned by the local user (id=501) and local group (id=20).
Permissions on the remote S.gpg-agent.host are 700, owned by the remote user (uid=1000) and remote group (gid=1000).

[deepsweet]

@cameri I believe that you should also SSH-forward agent to the remote in addition to socat, just mounting from macOS host to linux remote won't work because Docker for Mac doesn't support mounting unix sockets properly. Take a look at docker-ssh-agent-forward as an example, also here is my working (but obsolete now) solution for VSCode Remote Containers before they've added SSH_AUTH_SOCK magic natively.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[gzm55]

export SSH_AUTH_SOCK=`launchctl getenv SSH_AUTH_SOCK`

is slightly easier

from the discussion ChrisJohnsen/tmux-MacOSX-pasteboard#78 (comment) the portable way (with different terms, and within/without tmux) to get SSH_AUTH_SOCK is

 export SSH_AUTH_SOCK=`launchctl asuser "${UID:-"$(id -u)"}" launchctl getenv SSH_AUTH_SOCK`

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked