Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SshOperationTimeoutException disposing stream for connection that was already terminated by server #335

Closed
JasonQG opened this issue Nov 7, 2017 · 18 comments
Assignees
Labels

Comments

@JasonQG
Copy link
Contributor

JasonQG commented Nov 7, 2017

Since upgrading to 2016.1.0, I'm getting the below exception. Right before the stream was disposed, a command was sent to reboot the server, thus terminating the SSH connection on the server side. The same code worked with 2016.0.0.

System.AggregateException: One or more errors occurred. ---> Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.Session.Renci.SshNet.ISession.WaitOnHandle(WaitHandle waitHandle)
   at Renci.SshNet.Channels.Channel.Close()
   at Renci.SshNet.Channels.ChannelSession.Close()
   at Renci.SshNet.Channels.Channel.Dispose(Boolean disposing)
   at Renci.SshNet.Channels.ClientChannel.Dispose(Boolean disposing)
   at Renci.SshNet.Channels.ChannelSession.Dispose(Boolean disposing)
   at Renci.SshNet.Channels.Channel.Dispose()
   at Renci.SshNet.ShellStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
@drieseng
Copy link
Member

drieseng commented Nov 8, 2017

Are you sure you are using 2016.1.0 (RTM), and not one of the betas?
Can you reproduce this issue consistently?

I fixed an issue in Channel in beta4, and improved robustness even more recently (currently in the develop branch, and will be in the upcoming 2017.0.0-beta1 release).
I cannot reproduce this issue using either 2016.0.0-beta4, 2016.0.0 or the develop branch.

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 8, 2017

I started with beta 5, and then I upgraded to the RTM and saw it there too. There must be something else going on with our setup if you're not seeing it, but I'm not sure what at this time. When I can find some time, I'll try to look into it further. For now, I downgraded to avoid the problem.

@drieseng
Copy link
Member

drieseng commented Nov 8, 2017

Would it be possible for you to build SSH.NET from source (the develop branch)?
If you can still reproduce it from the develop branch, I can add extra tracing to allow me to pinpoint the cause.

I'd really like to fix this issue.

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 8, 2017

Thanks for being willing to help out with this. I just downloaded the latest source and was still able to reproduce it. Now what?

@drieseng
Copy link
Member

drieseng commented Nov 9, 2017

I suppose it's because the (SSH) server does not send a SSH_MSG_CHANNEL_CLOSE message before rebooting. What SSH server are you testing against? See this page for instructions on how you can obtain that info.

In previous versions of SSH.NET we were not waiting for the SSH_MSG_CHANNEL_CLOSE message. That explains why you only see this issue with 2016.1.0.

I'll add extra tracing to allow us to get more information, and let you know when I've committed these changes.

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 9, 2017

SSH-2.0-dropbear_2012.55

@drieseng
Copy link
Member

@JasonQG I cannot reproduce this issue using dropbear version 2013.60. Can you share the minimal code you need to reproduce this issue? Do you get the error consistently, or once every x number of times?

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 15, 2017

It happens consistently. There's nothing particularly special about the code. Just a command that resets and then a Dispose(). I suppose maybe the way the system is resetting isn't giving dropbear a chance to send the channel close message maybe?

@drieseng
Copy link
Member

Yes, that was indeed was I was thinking. Is it ok if I just add some Console.WriteLine(...) tracing (which end up in standard output) to the develop branch to allow us to confirm this?

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 16, 2017

Of course. Please go ahead

@drieseng
Copy link
Member

I've added a cwl-tracing branch that - as the name says - adds some Console.WriteLine(...) tracing to allow us to troubleshoot this issue.

When run against OpenSSH_6.6.1p1, I get the following output:

[1] Disposing ShellStream.
[1] Disposing channel from ShellStream.Dispose(bool).
[1] In Close() 0/0.
[1] Sending SSH_MSG_CHANNEL_EOF for channel 0/0.
[1] Sent SSH_MSG_CHANNEL_EOF for channel 0/0.
[1] Sending SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[1] Sent SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[1] Waiting for server to close channel 0/0.
[5] SSH_MSG_CHANNEL_CLOSE received from server 0/0.
[5] Closing channel after receiving SSH_MSG_CHANNEL_CLOSE from server 0/0.
[5] In Close() 0/0.
[1] Finished waiting for server to close channel 0/0.
[1] Disposed channel from ShellStream.Dispose(bool).

Please provide your exact result.

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 18, 2017

This is what I got:

[30] Disposing ShellStream.
[30] Disposing channel from ShellStream.Dispose(bool).
[30] In Close() 0/0.
[30] Sending SSH_MSG_CHANNEL_EOF for channel 0/0.
[30] Sent SSH_MSG_CHANNEL_EOF for channel 0/0.
[30] Sending SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[30] Sent SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[30] Waiting for server to close channel 0/0.
Exception thrown: 'Renci.SshNet.Common.SshOperationTimeoutException' in Renci.SshNet.dll
SshNet.Logging Verbose: 30 : Disconnecting client.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Disposing session.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Disconnecting session.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Sending message 'DisconnectMessage' to server: 'SSH_MSG_DISCONNECT'.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Shutting down socket.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Disposing socket.
SshNet.Logging Verbose: 30 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Disposed socket.
SshNet.Logging Verbose: 22 : [E69D2DFDD3B2FD62AF553A49F1ADB6D81606A244] Raised exception: Renci.SshNet.Common.SshConnectionException: An established connection was aborted by the server.
[30] Exception waiting for server to close channel 0/0: Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 837
at Renci.SshNet.Session.Renci.SshNet.ISession.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 763
at Renci.SshNet.Channels.Channel.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 522
at Renci.SshNet.Channels.Channel.Close() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 574.
[30] Disposed channel from ShellStream.Dispose(bool).
Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in mscorlib.dll
SshNet.Logging Verbose: 45 : [54790C418B85BBBB790989066521950F2AEB82CE] Sending message 'ChannelEofMessage' to server: 'SSH_MSG_CHANNEL_EOF : #0'.
SshNet.Logging Verbose: 45 : [54790C418B85BBBB790989066521950F2AEB82CE] Sending message 'ChannelCloseMessage' to server: 'SSH_MSG_CHANNEL_CLOSE : #0'.
[45] Disposing ShellStream.
[45] Disposing channel from ShellStream.Dispose(bool).
[45] In Close() 0/0.
[45] Sending SSH_MSG_CHANNEL_EOF for channel 0/0.
[45] Sent SSH_MSG_CHANNEL_EOF for channel 0/0.
[45] Sending SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[45] Sent SSH_MSG_CHANNEL_CLOSE for channel 0/0.
[45] Waiting for server to close channel 0/0.
Exception thrown: 'System.Net.Sockets.SocketException' in System.dll
Exception thrown: 'System.Net.Sockets.SocketException' in Renci.SshNet.dll
SshNet.Logging Verbose: 23 : [54790C418B85BBBB790989066521950F2AEB82CE] Raised exception: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Abstractions\SocketAbstraction.cs:line 328
at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1744
at Renci.SshNet.Session.ReceiveMessage() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1010
at Renci.SshNet.Session.MessageListener() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1892
--- End of inner exception stack trace ---
Exception thrown: 'Renci.SshNet.Common.SshConnectionException' in Renci.SshNet.dll
SshNet.Logging Verbose: 23 : [54790C418B85BBBB790989066521950F2AEB82CE] Disconnecting after exception: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Abstractions\SocketAbstraction.cs:line 328
at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1744
at Renci.SshNet.Session.ReceiveMessage() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1010
at Renci.SshNet.Session.MessageListener() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1892
--- End of inner exception stack trace ---
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 825
at Renci.SshNet.Session.Renci.SshNet.ISession.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 763
at Renci.SshNet.Channels.Channel.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 522
at Renci.SshNet.Channels.Channel.Close() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 574
SshNet.Logging Verbose: 23 : [54790C418B85BBBB790989066521950F2AEB82CE] Disposing socket.
SshNet.Logging Verbose: 23 : [54790C418B85BBBB790989066521950F2AEB82CE] Disposed socket.
[45] SSH connection exception waiting for server to close channel 0/0: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Abstractions\SocketAbstraction.cs:line 328
at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1744
at Renci.SshNet.Session.ReceiveMessage() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 1010
at Renci.SshNet.Session.MessageListener() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:lSshNet.Logging Verbose: 45 : Disconnecting client.
SshNet.Logging Verbose: 45 : [54790C418B85BBBB790989066521950F2AEB82CE] Disposing session.
SshNet.Logging Verbose: 45 : [54790C418B85BBBB790989066521950F2AEB82CE] Disconnecting session.
ine 1892
--- End of inner exception stack trace ---
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 825
at Renci.SshNet.Session.Renci.SshNet.ISession.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Session.cs:line 763
at Renci.SshNet.Channels.Channel.WaitOnHandle(WaitHandle waitHandle) in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 522
at Renci.SshNet.Channels.Channel.Close() in C:\jasong\ws_jasong\sit5\cur\Libraries\3rdParty\SshNet\Channels\Channel.cs:line 574.
[45] Disposed channel from ShellStream.Dispose(bool).

@drieseng
Copy link
Member

drieseng commented Nov 27, 2017

@JasonQG This confirms that the server does not always send a SSH_MSG_CHANNEL_CLOSE message before the socket is closed.

I'm adding a ChannelCloseTimeout property on ConnectionInfo that allows you to control how long you want to wait for the server to send a SSH_MSG_CHANNEL_CLOSE. The default value is 1 second.

@drieseng drieseng added this to the 2017.0.0-beta1 milestone Nov 27, 2017
@drieseng drieseng self-assigned this Nov 27, 2017
@drieseng drieseng added the bug label Nov 27, 2017
@drieseng
Copy link
Member

@JasonQG Please build SSH.NET from the develop branch, and confirm that this fixes the problem for you. You can use the ChannelCloseTimeout to reduce or remove the wait completely when closing the channel.

@JasonQG
Copy link
Contributor Author

JasonQG commented Nov 27, 2017

Confirmed that this fixes it for me. Thanks!

@ssoldatenkov
Copy link

Any idea when this fix will land in stable release?

@DashRaju
Copy link

Hi,
I'm using the latest one (2017.0.0-beta1) in one of the application (for transfer of file from external system to internal and from internal to external), but still i'm getting the error ----
Failed to upload file to server ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags) at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) at Renci.SshNet.Session.SocketReadline(TimeSpan timeout) at Renci.SshNet.Session.Connect()...

Any help from anyone who faced similar issue earlier and solved it would be greatly appreciated.

Thanks and Regards,
Dash

@drieseng drieseng modified the milestones: 2017.0.0-beta1, 2020.0.0-beta1 Mar 5, 2020
@jagrut-patel
Copy link

@drieseng I get same error. However, it happens when Shell stream is open and also it happens inconsistently.
Quite opposite of what is discussed in this thread and related fix.

I am posting in this thread to know if it could be a similar issue and that upgrading to 2020.0.0-beta1 would be worth an effort?

I am on v2016.1.0 integrated into a .NET Standard 2.0 library

Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
at Renci.SshNet.Channels.ChannelSession.Open()
at Renci.SshNet.ShellStream..ctor(ISession session, String terminalName, UInt32 columns, UInt32 rows, UInt32 width, UInt32 height, IDictionary2 terminalModeValues, Int32 bufferSize) at Renci.SshNet.ServiceFactory.CreateShellStream(ISession session, String terminalName, UInt32 columns, UInt32 rows, UInt32 width, UInt32 height, IDictionary2 terminalModeValues, Int32 bufferSize)

Thank you! Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants