Running inside a docker container with same namespace

[tachang]

Doing:

docker run -it --pid=container:7a688ad0e0b5 --net=container:7a688ad0e0b5 --cap-add sys_admin --cap-add sys_ptrace tachang/py-spy bash

py-spy record -o profile.svg -p 24
Error: No such file or directory (os error 2)

However this works:

py-spy dump -p 24
Process 24: uwsgi --hook-master-start unix_signal:1 gracefully_kill_them_all --ini /example/settings/uwsgi/uwsgi.ini --touch-reload /tmp/reload_uwsgi 
Python v3.7.2 (/usr/local/bin/uwsgi)

Thread 0x7FC5D341CB88 (active): "uWSGIWorker4Core0"

[tachang]

[2019-11-22T18:27:19.802408756Z INFO  remoteprocess::linux] Process 17 appears to be running in a different namespace - setting namespace to match
[2019-11-22T18:27:19.802481087Z WARN  py_spy::python_spy] Failed to set namespace: EINVAL: Invalid argument

[benfred]

Thanks for the bug report! I think I've managed to figure out what is going on here and will have a bug fix out soon.

The setns call is failing with EINVAL because py-spy has multiple threads now with the subprocess profiling code - and setns doesn't seem to work in a multithreaded program: ("EINVAL: The caller is multithreaded and tried to join a new user namespace").

It looks like rbspy reads files from /proc/{pid}/root instead of calling setns: https://github.com/rbspy/rbspy/blob/8945812a8/src/core/address_finder.rs#L316-L319 - I'm going to try that approach instead.

[tachang]

@benfred Thanks! I'm actually working on this right now and I'll put some notes here.
I don't know any Rust at all so it's slow going but I found this post which was really useful:
https://jvns.ca/blog/2018/01/26/spy-container/

My guess right now is that the Error: No such file or directory (os error 2) is referring to this previous line:
[2019-11-24T05:59:14.128555074Z INFO py_spy::python_spy] Found libpython binary @ /usr/local/lib/libpython3.7m.so.1.0

Because I am in this side loaded container it doesn't have that file. The file is definitely available at /proc/17/root/usr/local/lib/libpython3.7m.so.1.0 which is my uwsgi container's filesystem.

So right now I'm trying to figure out how to use this dockerized flag in Rust to maybe instead of setns just prepend /proc/<pid>/ to the calls.

[tachang]

Also for context the reason I need to load another container is I want to inspect this uwsgi that is currently running in a Docker Swarm environment. I would gladly just go ahead and install py-spy inside that running container but Docker swarm doesn't allow you to set capabilities. Because I am using a third party service to deploy this swarm cluster I don't have access to. So py-spy would really come in handy if I can run from another container. Thanks for all the work you put into this!

[benfred]

@tachang In the short term you can get around this with pip install py-spy==0.2.2 - and this should work. I have a fix almost ready to go now, just testing this out

[benfred]

Fix is in 0.3.1