Certificates aren't properly added to the ssh-agent service in 8.9.1.0

[stefanfritsch]

Please answer the following

"OpenSSH for Windows" version
8.9.1.0

Server OperatingSystem
Ubuntu 20.04

Client OperatingSystem
Windows 11

What is failing

I can't add a certificate to the agent properly. The agent either contains the private key or the certificate (however that works internally) but not both as it should.

Expected output

PS C:\Users\StefanFritsch\.ssh> ssh-add
Enter passphrase for C:\Users\StefanFritsch/.ssh/id_rsa:
Identity added: C:\Users\StefanFritsch/.ssh/id_rsa (C:\Users\StefanFritsch/.ssh/id_rsa)
Certificate added: C:\Users\StefanFritsch/.ssh/id_rsa-cert.pub (stefan.fritsch at 2022-06-15 07:59:21 user key valid for 10h)

PS C:\Users\StefanFritsch\.ssh> ssh-add -l
4096 SHA256:<snip> /home/stefan.fritsch/.ssh/id_rsa (RSA)
4096 SHA256:<snip> /home/stefan.fritsch/.ssh/id_rsa (RSA-CERT)

PS C:\Users\StefanFritsch\.ssh> ssh control-0
Last login: Wed Jun 15 07:57:13 2022 from 80.81.8.105
stefan.fritsch@control-0:~$

Actual output

PS C:\Users\StefanFritsch\.ssh> ssh-add
Enter passphrase for C:\Users\StefanFritsch/.ssh/id_rsa:
Identity added: C:\Users\StefanFritsch/.ssh/id_rsa (C:\Users\StefanFritsch/.ssh/id_rsa)
Certificate added: C:\Users\StefanFritsch/.ssh/id_rsa-cert.pub (stefan.fritsch at 2022-06-15 07:59:21 user key valid for 10h)

PS C:\Users\StefanFritsch\.ssh> ssh-add -l
4096 SHA256:<snip> C:\Users\StefanFritsch/.ssh/id_rsa (RSA-CERT)

PS C:\Users\StefanFritsch\.ssh> ssh control-0
Enter passphrase for key 'C:\Users\StefanFritsch/.ssh/id_rsa':

Context

1. If I first get the certificate and then add the key the above happens, i.e. I have to provide a password to login, but agent forwarding works, i.e. I can then login from control-0 into a different server without problems

2. If I first delete any old certificates, then add the key and then get a new certificate I can log into control-0 but agent forwarding is broken - i.e. the agent simply doesn't contain the certificate.

3. If I delete old certificates, add the key, get a new certificate and then add the key again it's the same as above - i.e. only the certificate shows up in the agent and I can't login with the agent.

4. This used to work in the 8.0.0.1 release on chocolatey but no longer works with the 8.9.1.0 msi from github

5. Restarting the agent doesn't help

6. Re-adding the key doesn't help

7. I tried adding the path with only "/" instead of "" as that was a problem in earlier versions but it didn't make a difference

8. After entering the password it connects just fine and as mentioned above I can then use the key and certificate in the agent (via forwarding) to connect to further servers.

Debugging Output

PS C:\Users\StefanFritsch> get-command ssh

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     ssh.exe                                            8.9.1.0    C:\Program Files\OpenSSH\ssh.exe

PS C:\Users\StefanFritsch> get-command ssh-add

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     ssh-add.exe                                        8.9.1.0    C:\Program Files\OpenSSH\ssh-add.exe

The service is also at the same path.

[crahan]

I can confirm after testing this is happening for me as well (with ed25519 keys).

[weavage]

Happening for me with ed25519 keys also. I compared with a coworkers Mac, and for some reason the Mac and their fancy dancy keyrings add both identities properly. To break this out a bit:

• ssh-add or ssh-add key_file
  • Adding the key with no options only adds the ed25519-cert identity and not the regular identity. This seems to be partially consistent with the man pages as it should try to load certificate signed keys ending in -cert.pub AFTER loading the private key. See here
• ssh-add -k or ssh-add -k key_file
  • This adds private key but not the certificate. I believe this is correct behavior.

It's also worth noting that if you try to load the key in the agent again, it kicks the old key out. For example, running ssh-add -k AFTER already loading the cert key will kick the cert key out. Runningssh-add AFTER loading just the private key will kick the plain private key out. I suspect this has to do with the signatures as both keys report the same signature using ssh-add -l except for the last part in parentheses. Whether or not both types should have the same signature is above my pay grade.

I tried to do a quick test on an Ubuntu 20 system and it showed the same behavior, so maybe this is an upstream bug??? I was running ssh 8.9 on powershell and the Ubuntu box was running 8.2, so hardly a definitive test. Maybe others can add their results.

[youngmit]

I had been banging my head against this for some time now, wondering why I was still getting prompted for passphrase on keys that I had added to the agent. I didn't notice that the private key was being missed until I saw this issue. I even reinstalled win32-openssh a couple of times to make sure I had a clean install and everything. Finally tried removing the certificate and everything works as expected.

Has any progress been made on this? For reference I'm running the latest: OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2

[youngmit]

Also, just found #1333, of which this appears to be a duplicate.

[tomtastic]

This has been frustrating me too, identity and certificate both reported as added, but only the certificate shows with a subsequent ssh-add -l