Deadlock when running commands that spawn background processes

[flupke]

The issue is described in details here: http://bugs.python.org/issue4216#msg77582

The issue is visible for example with git over ssh, when using ssh's ControlMaster option. Basically, it hangs because sh waits to receive an EOF on git's stream, which never happens because ssh inherited git's streams and survives past git's end.

Here's how to reproduce with sh:

import sh
import sys

# Remove control socket so we're sure to create a new ssh process; if you don't
# do this the script will hang the first time it is run, but not the second
# time
sh.rm('-f', '/tmp/git@github.com:22')
# Clone a repository over SSH, this will hang
sh.rm('-rf', 'sh')
sh.git.clone('git@github.com:amoffat/sh.git', _out=sys.stdout, _err=sys.stderr)

You also need to enable persistent ssh connections in your ~/.ssh/config:

Host *
ControlMaster auto
ControlPath /tmp/%r@%h:%p
ControlPersist 72h

I think there should be an option to completely disable sh reading on streams (_no_out and _no_err don't help, neither does redirecting streams to files).

[amoffat]

Thanks for reporting @flupke, I will look into it

[jhrmnn]

I hit this issue as well. A dirty workaround is to call something like subprocess.call(['ssh', host, 'true']) at the beginning of your script, which initiates the ssh connection and doesn't block since it doesn't handle streams at all. Subsequent calls to sh then work as expected. This of course assumes that the connection won't close while your script is running.

[amoffat]

So after some investigation, the way I see it is:

• any process launched through sh has to wait on its stdout/stderr
• any process may launch a subprocess, dup its stdout/stderr, causing the parent process to never finish reading

We could add an option saying "hey, I know this process will launch a subprocess and cause stdout/stderr to never finish, so disable reading on them." but that seems like it requires too much knowledge of what the process does under the hood, and will probably result in a lot of people creating issues, wondering about hanging, and then people try to debug it.

I've opted for a hacky approach (here and here) that requires no knowledge of the process. What it does is, when the process ends, we say "ok, threads that are spinning and trying to read from stdout and stderr, you have X seconds to finish reading from those streams, then you're done." Most processes stdout/err will receive their EOFs immediately, so the timer behavior won't apply to them. The timer behavior will only kick in on processes that have dup'd their fds in another child process, and will therefore never EOF. In that case, the process will hang for half a second on completion, but then terminate correctly

the fix is on the release-1.2 branch and will go out with it

[flupke]

Cool! I tested the branch against my script above, and it doesn't hang anymore. There are error messages though:

$ python test-sh-bug.py                                                                                                                                                                        (10-07 20:59)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/flupke/.virtualenvs/sh-bug/lib/python2.7/site-packages/sh.py", line 1684, in output_thread
    readers.remove(stream)
ValueError: list.remove(x): x not in list

Cloning into 'sh'...
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/flupke/.virtualenvs/sh-bug/lib/python2.7/site-packages/sh.py", line 1684, in output_thread
    readers.remove(stream)
ValueError: list.remove(x): x not in list

[amoffat]

@flupke Weird, works fine for me, but I went ahead and reverted the lines that are causing the errors. Want to give it another try?

[flupke]

Yup, all good! (this was on mac btw)

On Fri, Oct 7, 2016 at 9:07 PM Andrew Moffat notifications@github.com
wrote:

@flupke https://github.com/flupke Weird, works fine for me, but I went
ahead and reverted the lines that are causing the errors. Want to give it
another try?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#310 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALiIlIyMPxBdZqnFmGdpaCE8kwkyxslks5qxphhgaJpZM4ICFhV
.