Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclean remote terminal session causes upload, download and shell modules to hang. #93

Closed
cartesian-theatrics opened this issue Jun 22, 2020 · 11 comments
Labels
bug Something isn't working

Comments

@cartesian-theatrics
Copy link

cartesian-theatrics commented Jun 22, 2020

Hello,

I'm having some troubles making a ssh connection. I believe it's related to using ecdsa keys, although I'm hardly an expert in ssh troubleshooting. Here's the output with --debug-ssh:

Connection established
Remote version string: SSH-2.0-Go
Local version string: SSH-2.0-JSCH-0.1.54
CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
ecdsa-sha2-nistp256 is not available.
ecdsa-sha2-nistp384 is not available.
ecdsa-sha2-nistp521 is not available.
SSH_MSG_KEXINIT sent
SSH_MSG_KEXINIT received
kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
kex: server: ssh-rsa
kex: server: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
kex: server: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
kex: server: none
kex: server: none
kex: server: 
kex: server: 
kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
kex: client: ssh-rsa,ssh-dss
kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-```96,hmac-md5-96
kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
kex: client: none
kex: client: none
kex: client: 
kex: client: 
kex: server->client aes128-ctr hmac-sha1 none
kex: client->server aes128-ctr hmac-sha1 none
SSH_MSG_KEX_ECDH_INIT sent
expecting SSH_MSG_KEX_ECDH_REPLY
ssh_rsa_verify: signature true
Host '<hostname>' is known and matches the RSA host key
SSH_MSG_NEWKEYS sent
SSH_MSG_NEWKEYS received
SSH_MSG_SERVICE_REQUEST sent
SSH_MSG_SERVICE_ACCEPT received
Authentications that can continue: publickey,keyboard-interactive,password
Next authentication method: publickey
Disconnecting from .. <hostname> .. port 2200
Exception in thread "async-dispatch-1" com.jcraft.jsch.JSchException: Auth fail
	at com.jcraft.jsch.Session.connect(Session.java:519)
	at com.jcraft.jsch.Session.connect(Session.java:183)

I'm setting setting the :private and :public-key fields with:

{:private-key (slurp "<home>/.ssh/id_ecdsa")
 :public-key (slurp "<home>/.ssh/id_ecdsa.pub")

I've also tried connecting without setting :private-key and :public-key, but no luck. I am able to connect with plain ssh at the commandline no problem.

Regards,
John C.

@retrogradeorbit
Copy link
Member

see #35

@retrogradeorbit
Copy link
Member

also new style openssh keys dont work: #53

@retrogradeorbit retrogradeorbit added the bug Something isn't working label Jun 22, 2020
@cartesian-theatrics
Copy link
Author

cartesian-theatrics commented Jun 22, 2020

After messing around a bit, I am able to successfully connect (at least when running a regular JVM process). However, the only command I've been able to get working with this connection is get-fact:

(puget/cprint (ssh {:username "..."
                    :hostname "..."
                    :port 2200
                    :private-key (slurp "<home>/.ssh/id_ecdsa")
                    :public-key (slurp "<home>/.ssh/id_ecdsa.pub")
                    :agent-forwarding true
                    :strict-host-key-checking false}
                   (get-fact)))

Everything else I've tried will hang indefinitely after connecting. For example:

(puget/cprint (ssh {:username "..."
                    :hostname "..."
                    :port 2200
                    :private-key (slurp "<home>/.ssh/id_ecdsa")
                    :public-key (slurp "<home>/.ssh/id_ecdsa.pub")
                    :agent-forwarding true
                    :strict-host-key-checking false}
                   (upload {:content "hello world"
                            :dest "/home/brain/hello.txt"})))

Will hang. Output:

Connecting to <hostname> port 2200
Connection established
Remote version string: SSH-2.0-Go
Local version string: SSH-2.0-JSCH-0.1.54
CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
SSH_MSG_KEXINIT sent
SSH_MSG_KEXINIT received
kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
kex: server: ssh-rsa
kex: server: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
kex: server: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
kex: server: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
kex: server: none
kex: server: none
kex: server: 
kex: server: 
kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
kex: client: none
kex: client: none
kex: client: 
kex: client: 
kex: server->client aes128-ctr hmac-sha1 none
kex: client->server aes128-ctr hmac-sha1 none
SSH_MSG_KEX_ECDH_INIT sent
expecting SSH_MSG_KEX_ECDH_REPLY
ssh_rsa_verify: signature true
Host '<hostname>' is known and matches the RSA host key
SSH_MSG_NEWKEYS sent
SSH_MSG_NEWKEYS received
SSH_MSG_SERVICE_REQUEST sent
SSH_MSG_SERVICE_ACCEPT received
Authentications that can continue: publickey,keyboard-interactive,password
Next authentication method: publickey
Authentication succeeded (publickey).
... hangs indefinitely ...

Not sure what's going on. Do you know if there's a way to get more logs out of JSch? Also, for some reason when compiling to native I see these logged at runtime after the "CheckSignatures" line:

ecdsa-sha2-nistp256 is not available.
ecdsa-sha2-nistp384 is not available.
ecdsa-sha2-nistp521 is not available.

@cartesian-theatrics
Copy link
Author

cartesian-theatrics commented Jun 22, 2020

Couple more details:

  1. The format line for id_ecdsa is "-----BEGIN EC PRIVATE KEY-----".
  2. Algorithm in id_ecdsa.pub is "ecdsa-sha2-nistp256".

@retrogradeorbit
Copy link
Member

does it work under the spire binary?

@cartesian-theatrics
Copy link
Author

No, same issue with the spire binary. After trying more commands I'm it seems to be mostly file-transfer commands that fail. get-fact, rm, and stat I know work. upload, download, and shell I know don't work (similar hanging problem). I'm trying to look at the implementation see why in particular shell wouldn't work.

@retrogradeorbit
Copy link
Member

Very interesting. It sounds like it is not the key. For get-fact, rm etc to work it must be connected. Can you give me more information about the target system? OS, version, default shell etc.

upload, download and shell all pipe input to commands on the remote.

Can you confirm that standard scp command works to the remote (not just ssh but scp aswell)?

Also check if there are any echo or printf statements in your shell profile on the remote. Look in files like /etc/profile, /etc/bash.bashrc, ~/.bashrc. Is your remote shell profile clean and free of echo and printf?

@cartesian-theatrics
Copy link
Author

Wow, plain scp has historically worked just fine, but now it's hanging! So sorry I may have wasted your time.

@retrogradeorbit
Copy link
Member

retrogradeorbit commented Jun 24, 2020

You probably have an unclean session. There is something in the ssh shell startup that is echoing characters and interfering with the scp protocol. Here's some info on that. You can try the tty -s trick on your profile startup script and see if it solves the problem. Please report back here if you find the root cause so future travelers will find this ticket!

from: https://web.archive.org/web/20170215184048/https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works

Running scp with talkative shell profiles on the remote side?

People sometimes hit problems with scp while SSH connections continue to work. This is usually a problem with adding echo/printf commands to their shell profile. See two examples.

scp just hangs after the password is entered when I try to copy to the remote side

For example, this can happen if you add this to your shell profile on the remote system:

echo ""

Why it just hangs? That comes from the way how scp in source mode waits for the confirmation of the first protocol message. If it's not binary 0, it expects that it's a notification of a remote problem and waits for more characters to form an error message until the new line arrives. Since you didn't print another new line after the first one, your local scp just stays in a loop, blocked on read(2). In the meantime, after the shell profile was processed on the remote side, scp in sink mode was started, which also blocks on read(2), waiting for a binary zero denoting the start of the data transfer. So, both scp's are blocked on reading, effectively causing a deadlock. In summary, the problems was caused because your remote shell through its profile output "joined" the protocol conversation.

scp just executes my profile and exits if I copy to the remote side

...meaning that scp just prints the 1st message that is printed from user's shell profile and exits. That's because you added for example this into your shell profile:

$ echo 'echo "hi there!"' >> .bashrc

and then run the scp command:

$ cp /etc/passwd localhost:/tmp
hi there!
$ echo $?
1

This is a very similar problem to the one already mentioned. Since the first character received wasn't binary 0 (but character 'h') it assumes a problem, reads up to the next new line character, prints that out as an error message and exits.

There is an easy fix for those problems - just print what you want when you have a terminal, like this:

tty -s && echo "hi there!"

I see protocol error: unexpected message and scp exits

Again, similar to the 1st problem, but you are copying from the remote side. What happened? Your local scp, the data consumer, waits for the protocol message from the producer. However, it gets an empty line immediately followed by a new line character which is a violation of the protocol and your local scp then bails out. If you print more characters in your remote shell profile it is considered an error message (unless it starts with a valid protocol character in which situation the message finally printed before it fails will be even more confusing) and the whole message up to the new line character is printed and scp exits then. Example if I add printf "XXXX" to my profile (remember, printf(1) doesn't automatically add a new line) - the whole output up to the first protocol message ending with the new line is considered an error message:

$ scp localhost:/etc/passwd .
Password: 
XXXXC0644 1135 passwd
$ echo $?
1

And if you mess up with a valid message, for example D with printing this from your remote shell profile: printf "D":

$ scp localhost:/etc/passwd .
Password: 
protocol error: bad mode
$ echo $?
1

Moral of this? Always check the return code of scp.

@retrogradeorbit retrogradeorbit changed the title Ecdsa troubles Unclean remote terminal session causes upload, download and shell modules to hang. Jun 24, 2020
@cartesian-theatrics
Copy link
Author

Thanks for the detailed write up. After some investigating I'm pretty sure the issue I'm having is caused by something in the implementation of the ssh proxy backend.

@retrogradeorbit
Copy link
Member

Theres not much I can do with an unclean terminal session so closing this. Thank's @cartesian-theatrics for reporting as it's here in the archives in case others experience it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants