docker-machine doesn't work with this version of OpenSSH

[WojtekKozaczynski]

"OpenSSH for Windows" version
0.0.10.0

OS details
Windows 10 Pro

What is failing
A detailed description of the issue is in the docker/machine repo @ docker/machine#4053.

The summary is that when OpenSSH is installed on Windows 10 the docekr-machine tries to use it to communicate with the Hyper-V VMs it creates and manages, and the Win32 OpenSSH implementation fails.

Repro steps are

• Install Windows Docker toolkit
• Install OpenSSH following the guidelines at https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH, including adding it to the Path
• In Hyper-V Manager create (if you don’t have one) a Virtual Switch and connect it to an external network device (either the WiFi card or local network card)
• Start Docker
• Execute “docker-machine -debug create -d hyperv --hyperv-virtual-switch ” command

The last command hangs when the docker-machine tries to open an SSH connection to the newly created VM.

Cheers, Wojtek

[manojampalam]

I looked into this. Apparently, docker is launching ssh with the following commandline:

ssh.exe -F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@10.123.171.161 -o IdentitiesOnly=yes -i C:\Users\manojamp.docker\machine\machines\localhost\id_rsa -p 22 "exit 0"

There are certain Unix specific elements in the command line (/dev/null) that are not mapped in the Windows port yet. Targeting a fix for April-End.

[bagajjal]

Please check the may end release.

[gbraad]

@bagajjal when will the May-end release be available? The current release is from 14 days ago...

[WojtekKozaczynski]

Please confirm if "May-end release" is the same as "Win32-OpenSS v0.0.14.0". This is the latest version I can find, yet it was released 14 days ago.
The list of bugs that v0.0.14.0 fixes does not include this bug (bug #666). Should I test v0.0.14.0 for this bug or wait for the next drop?

[bagajjal]

May-end release will be available today.. it will be v0.0.15.0

[gbraad]

This now works as expected with the release from choco install openssh. Thanks

[WojtekKozaczynski]

Release 0.0.15.0 fails for me in the same place that is, when the docker-machine it trying to contact a newly created host. Below is how a single such attempt looks in the log

• it finds the newly created host's IP address (I can ping the host on that address fine)
• it calls the OpenSHH executable, but that invocation fails with error 255

(testHost) DBG | [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM testHost ).networkadapters[0]).ipaddresses[0]
(testHost) DBG | [stdout =====>] : 192.168.0.100
(testHost) Calling .GetSSHPort
(testHost) DBG |
(testHost) DBG | [stderr =====>] :
(testHost) Calling .GetSSHKeyPath
(testHost) Calling .GetSSHKeyPath
(testHost) Calling .GetSSHUsername
Using SSH client type: external
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.0.100 -o IdentitiesOnly=yes -i C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa -p 22] C:\Program Files\OpenSSH\ssh.exe <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:

[bagajjal]

@WojtekKozaczynski -

1. Can you please check the ssh version,
   ((Get-Item (Get-Command C:\Program Files\OpenSSH\ssh.exe).Source).VersionInfo.FileVersion)

2. If its 0.0.15.0 then please run this command and provide us the logs
   ssh.exe -vvv -F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.0.100 -o IdentitiesOnly=yes -i C:\Users\wojtek.docker\machine\machines\testHost\id_rsa -p 22

[WojtekKozaczynski]

Balu,

I executed the first command, and I am running 0.0.15.0

PS C:\WINDOWS\system32> ((Get-Item (Get-Command "C:\Program Files\OpenSSH\ssh.exe").Source).VersionInfo.FileVersion)
0.0.15.0

Here are the steps I followed

• I stopped the SSHD service and I modified the following line in the sshd_config file: LogLevel DEBUG3
• I restarted the SSHD service and executed the following docker-machine command

docker-machine --debug create -d hyperv --hyperv-virtual-switch ExternalWireless testHost

• as expected, the command created the testHost VM, but failed trying to connect to it using OpenSSH
• I pinged the machine and it responded
• I executed your ssh.exe command

ssh.exe -vvv -F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.0.104 -o IdentitiesOnly=yes -i C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa -p 22

• this generated a bunch of debug messages directly to the console

I am enclosing 3 files

• two SSH log files
• a file with a copy of console messages starting at the end of the testHost creation sequence and following with the above ssh.exe command and the debug messaegs.

sshd.log.txt
ssh-agent.log.txt
Win32-OpenSSH.txt

Wojtek

[WojtekKozaczynski]

After I sent the message above I also noticed that the docker-machine complains that it cannot verify docker version of the host because the cert is signed by unknown authority. This may be relevant.

c:\Program Files\OpenSSH>docker-machine ls
NAME       ACTIVE   DRIVER   STATE     URL                        SWARM   DOCKER    ERRORS
testHost   -        hyperv   Running   tcp://192.168.0.104:2376           Unknown   Unable to query docker version: Get https://192.168.0.104:2376/v1.15/version: x509: certificate signed by unknown authority

[bagajjal]

Looks like the problem is with the keys (C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa)... With the recent changes (0.0.15.0 release), ssh will check the file permissions of the key files.. please go through https://github.com/PowerShell/Win32-OpenSSH/wiki/OpenSSH-utility-scripts-to-fix-file-permissions and fix the file permission issues..

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:nc8cfdobTTiHazdqv+/bJ3ZNe3+Xcdmo+EU5ypt/X1A C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:nc8cfdobTTiHazdqv+/bJ3ZNe3+Xcdmo+EU5ypt/X1A
debug3: sign_and_send_pubkey: RSA SHA256:nc8cfdobTTiHazdqv+/bJ3ZNe3+Xcdmo+EU5ypt/X1A
debug3: open - handle:00000000000001DC, io:00000222FB40AA60, fd:4
debug3: Bad permissions. Try removing permissions for user: S-1-5-21-2127521184-1604012920-1887927527-26969423 on file C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug3: close - io:00000222FB40AA60, type:2, fd:4, table_index:4
Load key "C:\Users\wojtek\.docker\machine\machines\testHost\id_rsa": bad permissions
debug2: we did not send a packet, disable method

[davehowson]

I had this issue and I tried all the suggestions here and on almost everywhere else but nothing worked. However, I ran the VM command on Git Bash and it successfully created a VM for me. I'm guessing it worked because as @manojampalam pointed out, the command runs with some Unix based elements which prevent Windows terminals from executing it.

[bagajjal]

@davehowson - To debug further please share ssh client logs (ssh -vvv user@ip <all_args_for_ur_scenario>) and sshd logs (DEBUG3 enabled). This original issue reported is closed so we need to debug why you are still seeing this issue.

[bardware]

I'm just trying to use docker-toolbox in hyper-v mode and found this discussion. I created an External switch manually and called docker-machine -debug create -d hyperv dockerhyperv as admin in cmd and eventually got

SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255
output  :
Getting to WaitForSSH function...
(dockerhyperv) Calling .GetSSHHostname
(dockerhyperv) DBG | [executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM dockerhyperv ).state
(dockerhyperv) DBG | [stdout =====>] : Running
(dockerhyperv) DBG |
(dockerhyperv) DBG | [stderr =====>] :
(dockerhyperv) DBG | [executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteract
ive (( Hyper-V\Get-VM dockerhyperv ).networkadapters[0]).ipaddresses[0]
(dockerhyperv) DBG | [stdout =====>] : 192.168.0.17
(dockerhyperv) DBG |
(dockerhyperv) DBG | [stderr =====>] :
(dockerhyperv) Calling .GetSSHPort
(dockerhyperv) Calling .GetSSHKeyPath
(dockerhyperv) Calling .GetSSHKeyPath
(dockerhyperv) Calling .GetSSHUsername
Using SSH client type: external
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@192.168.0.17 -o IdentitiesOnly=yes -i C:\Users\bddoeble\.docker\machine\machines\dockerhyperv\id_rsa -p 22] C:\WINDOWS\System32\OpenSSH\ssh.exe <nil>}

I started git bash using start "" "%PROGRAMFILES%\Git\bin\sh.exe" --login, called the exact same command and it worked. I'm on docker toolbox v18.06.1-ce

[bardware]

There's the --native-ssh switch that workes around the issue. Also needed when calling regenerate-certs when using docker-machine in a different network than the one it had been used in first instance.