More than 1 command to Cisco IOS-XE devices fail with empty SessionError exception

[rwobig93]

Describe the bug
When running more than 1 command against a Cisco IOS-XE network device a SessionError exception is raised but is empty (no message or stack), this doesn't occur against NXOS devices or any other Linux host I was able to test against (Ubuntu, RHEL, CentOS) but does occur on both vIOS (using CML/Cisco Modeling Labs) and physical IOS-XE devices

To Reproduce

Steps to reproduce the behavior:

1. Connect to a Cisco IOS-XE device (virtual or physical)
2. Run 2 or more commands (commands are arbitrary as long as they are valid)

Expected behavior
HostOutput object is returned and stdout generator is iterable to get the command output

Actual behaviour
First command succeeds and returns as expected, 2nd command fails with a pssh.exceptions.SessionError that is empty and contains no detail or messages

Code

import logging
from pssh.clients import SSHClient
from pssh.utils import enable_debug_logger

def main():
	logging.basicConfig(encoding='utf-8', level=logging.DEBUG, format='%(asctime)s [%(levelname)s]: %(message)s')

	ssh_client = SSHClient(host="192.168.1.17", user="cisco", password="cisco", pkey=None, port=22)
	logging.getLogger('pssh.host_logger').setLevel(logging.WARNING)
	enable_debug_logger()

	command_output_one = ssh_client.run_command(command="show ip interface brief")
	for line in command_output_one.stdout:
		logging.info(f"Command output: {line}")

	command_output_two = ssh_client.run_command(command="show inventory")
	for line in command_output_two.stdout:
		logging.info(f"Command output: {line}")

	command_output_three = ssh_client.run_command(command="show switch")
	for line in command_output_three.stdout:
		logging.info(f"Command output: {line}")


if __name__ == '__main__':
	main()

Error Output

2024-03-30 23:11:12,197 [DEBUG]: Connecting to 192.168.1.17:22
2024-03-30 23:11:12,332 [DEBUG]: Agent auth failed with AgentConnectionError('Unable to connect to agent') continuing with other authentication methods
2024-03-30 23:11:12,332 [DEBUG]: Trying to authenticate with identity file /home/user/.ssh/id_ed25519
2024-03-30 23:11:14,346 [DEBUG]: Authentication with identity file /home/user/.ssh/id_ed25519 failed with , continuing with other identities
2024-03-30 23:11:14,346 [DEBUG]: Private key auth failed, trying password
2024-03-30 23:11:14,409 [DEBUG]: Authentication completed successfully - setting session to non-blocking mode
2024-03-30 23:11:14,419 DEBUG    pssh.clients.native.single Executing command 'b'show ip interface brief''
2024-03-30 23:11:14,419 [DEBUG]: Executing command 'b'show ip interface brief''
2024-03-30 23:11:14,439 DEBUG    pssh.clients.base.single Reading from stdout buffer, timeout=None
2024-03-30 23:11:14,439 [DEBUG]: Reading from stdout buffer, timeout=None
2024-03-30 23:11:14,440 [INFO]: Command output: 
2024-03-30 23:11:14,440 [INFO]: Command output: 
2024-03-30 23:11:14,452 [INFO]: Command output: 
2024-03-30 23:11:14,452 [INFO]: Command output: Interface                  IP-Address      OK? Method Status                Protocol
2024-03-30 23:11:14,452 [INFO]: Command output: GigabitEthernet0/0         192.168.1.17    YES DHCP   up                    up      
2024-03-30 23:11:14,452 [INFO]: Command output: GigabitEthernet0/1         unassigned      YES unset  administratively down down    
2024-03-30 23:11:14,452 [INFO]: Command output: GigabitEthernet0/2         unassigned      YES unset  administratively down down    
2024-03-30 23:11:14,452 [INFO]: Command output: GigabitEthernet0/3         10.150.1.0      YES TFTP   up                    up      
2024-03-30 23:11:14,567 [INFO]: Command output: Loopback100                10.100.1.10     YES TFTP   up                    up      
Traceback (most recent call last):
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/native/single.py", line 271, in open_session
    chan = self._open_session()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/native/single.py", line 265, in _open_session
    chan = self._eagain(self.session.open_session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/native/single.py", line 349, in _eagain
    return self._eagain_errcode(func, LIBSSH2_ERROR_EAGAIN, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/base/single.py", line 576, in _eagain_errcode
    ret = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "ssh2/session.pyx", line 389, in ssh2.session.Session.open_session
  File "ssh2/utils.pyx", line 214, in ssh2.utils.handle_error_codes
ssh2.exceptions.SocketRecvError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/_general-testing/test.py", line 298, in <module>
    main()
  File "/home/user/_general-testing/test.py", line 288, in main
    command_output_two = ssh_client.run_command(command="show inventory")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/base/single.py", line 555, in run_command
    channel = self.execute(_command, use_pty=use_pty)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/native/single.py", line 297, in execute
    channel = self.open_session() if channel is None else channel
              ^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.10/site-packages/pssh/clients/native/single.py", line 273, in open_session
    raise SessionError(ex)
pssh.exceptions.SessionError

Workaround
I was able to get multiple commands working only if I do a ssh_client.disconnect() then do another full connection to the device

Troubleshooting Steps

• Tried forcing the channel to close using ssh_client.close_channel(command_output_one.channel)
• Tried both SSHClient and ParallelSSHClient with the same behavior
• Tried explicitly defining the Init/Constructor args with low and high values (timeouts, no pty, no shell, no sudo, ect)
• Tried on python3.9, python3.10 and python3.11 with the same result

Additional information
pip freeze output:

$ pip freeze | grep ssh
parallel-ssh==2.12.0
ssh-python==1.0.0
ssh2-python==1.0.0

Tested IOS-XE 15.9 vIOS, IOS-XE 16.3.x physical and IOS-XE 17.03 physical - all with the same failure

This failure is repeatable with both SSHClient and ParallelSSHClient, on ParallelSSHClient I tried also doing stop_on_errors=True on run_command hoping to get more failure detail but didn't see anything different