Error: Couldn't find python binary

[Poil]

Hi,

After installing in root via

scl enable rh-python36
cargo install py-spy"

And running as root via .cargo/bin/py-spy --pid 11792
I have this error py-spy is unable to find the python binary.

I also have tried as the same user as the process with the virtualenvironment loaded but I have the same error message.
My app is in a virtual environment /home/myuser/env36/flexssl/bin/python3
My OS is EL7

I don't understand how the code lookup for the python binary path

Best regards,

[benfred]

I'm not sure what's happening here =( Can you enable logging and paste the output here ?

set RUST_LOG=info
py-spy --pid 123456

[dangast]

I'm also seeing this on EL7, with the OS installation of python 2.7:

host:~$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
host:~$ /usr/bin/python
Python 2.7.5 (default, Nov  6 2016, 00:28:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
host:~$ RUST_LOG=info py-spy -- /usr/bin/python -c "print 'foo'"
 INFO 2018-09-06T17:57:49Z: py_spy: Command line args: ArgMatches { args: {"rate": MatchedArg { occurs: 0, indices: [4], vals: ["1000"] }, "duration": MatchedArg { occurs: 0, indices: [5], vals: ["2"] }, "python_program": MatchedArg { occurs: 3, indices: [1, 2, 3], vals: ["/usr/bin/python", "-c", "print \'foo\'"] }}, subcommand: None, usage: Some("USAGE:\n    py-spy [FLAGS] [OPTIONS] --pid <pid> [python_program]...") }
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 0000000000400000-0000000000401000 r-x /usr/bin/python2.7
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 0000000000600000-0000000000601000 r-- /usr/bin/python2.7
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 0000000000601000-0000000000602000 rw- /usr/bin/python2.7
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 0000000001afc000-0000000001b3e000 rw- [heap]
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa2a6000-00007fdfaa45c000 r-x /usr/lib64/libc-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa45c000-00007fdfaa65c000 --- /usr/lib64/libc-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa65c000-00007fdfaa660000 r-- /usr/lib64/libc-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa660000-00007fdfaa662000 rw- /usr/lib64/libc-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa662000-00007fdfaa667000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa667000-00007fdfaa767000 r-x /usr/lib64/libm-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa767000-00007fdfaa967000 --- /usr/lib64/libm-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa967000-00007fdfaa968000 r-- /usr/lib64/libm-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa968000-00007fdfaa969000 rw- /usr/lib64/libm-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa969000-00007fdfaa96b000 r-x /usr/lib64/libutil-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaa96b000-00007fdfaab6a000 --- /usr/lib64/libutil-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaab6a000-00007fdfaab6b000 r-- /usr/lib64/libutil-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaab6b000-00007fdfaab6c000 rw- /usr/lib64/libutil-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaab6c000-00007fdfaab6e000 r-x /usr/lib64/libdl-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaab6e000-00007fdfaad6e000 --- /usr/lib64/libdl-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaad6e000-00007fdfaad6f000 r-- /usr/lib64/libdl-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaad6f000-00007fdfaad70000 rw- /usr/lib64/libdl-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaad70000-00007fdfaad87000 r-x /usr/lib64/libpthread-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaad87000-00007fdfaaf86000 --- /usr/lib64/libpthread-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaaf86000-00007fdfaaf87000 r-- /usr/lib64/libpthread-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaaf87000-00007fdfaaf88000 rw- /usr/lib64/libpthread-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaaf88000-00007fdfaaf8c000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfaaf8c000-00007fdfab109000 r-x /usr/lib64/libpython2.7.so.1.0
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab109000-00007fdfab308000 --- /usr/lib64/libpython2.7.so.1.0
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab308000-00007fdfab30a000 r-- /usr/lib64/libpython2.7.so.1.0
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab30a000-00007fdfab348000 rw- /usr/lib64/libpython2.7.so.1.0
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab348000-00007fdfab357000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab357000-00007fdfab377000 r-x /usr/lib64/ld-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab524000-00007fdfab56a000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab575000-00007fdfab576000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab576000-00007fdfab577000 r-- /usr/lib64/ld-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab577000-00007fdfab578000 rw- /usr/lib64/ld-2.17.so
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007fdfab578000-00007fdfab579000 rw- 
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007ffd1ee5b000-00007ffd1ee7c000 rw- [stack]
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: 00007ffd1ef58000-00007ffd1ef5a000 r-x [vdso]
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: map: ffffffffff600000-ffffffffff601000 r-x [vsyscall]
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Found python binary @ /usr/bin/python2.7
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Getting version from python binary BSS
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to get version from BSS section: failed to find version string
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Trying to get version from path: /usr/bin/python2.7
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: python version 2.7.0 detected
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to get interp_head from symbols, scanning BSS section from main binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to get interpreter from binary BSS, scanning libpython BSS
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Failed to find a python interpreter in the .data section
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Failed to connect to process, retrying. Error: Couldn't find python binary
 INFO 2018-09-06T17:57:49Z: py_spy::python_spy: Got virtual memory maps from pid 26626:
foo

Error: Couldn't find python binary

host:~$ 

[JasonBenn]

Having a similar problem on Linux 4.15.0 (Ubuntu 18.04). Installed py-spy as root, running a Python web server from within a virtualenv using systemd.

RUST_LOG=info py-spy --pid 10150 > log.txt 2>&1 outputs log.txt.

Thank you for the awesome tool, by the way.

[bobozhengsir]

The similar problem on embedded Python interpreters like uWSGI.

[JasonBenn]

Ah, I'm also using uWSGI ^

[benfred]

I think there are at least two issues being reported here.

The first of which is running on Centos 7 / EL7. I think this is because on this platform python is configured with --enable-shared and the shared library is placed in /usr/lib64, which I wasn't matching. I fixed here 8ea64fa, after which I could run py-spy on centos. @dangast I'm pretty sure this will fix your problem, thanks for the log output! This fix will be in the next release (with a bunch of other fixes, included a fix for glibc 2.18 which you probably also hit).

The second problem is getting py-spy to work with uwsgi, I've created a new issue to track #25. I should have a fix for this next week, it doesn't look too hard to get going.

Thanks everyone for trying this project out!

[benfred]

I pushed v0.1.5 which contains the centos7 fix,