Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

py-spy 0.1.9 fails when started inside docker container #68

Closed
iljau opened this issue Dec 6, 2018 · 6 comments
Closed

py-spy 0.1.9 fails when started inside docker container #68

iljau opened this issue Dec 6, 2018 · 6 comments

Comments

@iljau
Copy link

iljau commented Dec 6, 2018

When py-spy is installed and launched from inside docker container, py-spy==0.1.9 fails, whereas previous version py-spy==0.1.8 worked (SYS_PTRACE capability is set for container).

$ py-spy --version
py-spy 0.1.9

$ RUST_BACKTRACE=1 py-spy -- python3 test.py 

Error: EPERM: Operation not permitted
stack backtrace:
   0:           0x4e7bac - backtrace::backtrace::trace::ha0fbe6521a297b97
   1:           0x4e6682 - backtrace::capture::Backtrace::new_unresolved::ha618912f49ed2d80
   2:           0x4e602f - failure::backtrace::internal::InternalBacktrace::new::h33dcc523c9982708
   3:           0x4e62d1 - <failure::backtrace::Backtrace as core::default::Default>::default::h530f3770400a1ba8
   4:           0x4e6328 - failure::backtrace::Backtrace::new::hfc945c6756648d04
   5:           0x444e30 - py_spy::process::os_impl::Namespace::new::h2c3c0252db472d26
   6:           0x41d75f - py_spy::python_spy::PythonSpy::new::hfbd77dacf6afa4b9
   7:           0x421508 - py_spy::python_spy::PythonSpy::retry_new::hc6dd70655afd2779
   8:           0x41bb35 - py_spy::main::h54ad4b843dae6e20
   9:           0x4416c2 - std::rt::lang_start::{{closure}}::h1e5e01061341fa09
  10:           0x5f1b72 - std::rt::lang_start_internal::{{closure}}::h388a7d34c6d3c7af
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::he4497885895661a8
                        at libstd/panicking.rs:310
  11:           0x6159e9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:103
  12:           0x6012e6 - std::panicking::try::hcf37798695a9b029
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::hdd9654d0e8ee1176
                        at libstd/panic.rs:392
                         - std::rt::lang_start_internal::h4ec5eae1ccf41136
                        at libstd/rt.rs:58
  13:           0x41c533 - main
# test.py
while True:
    pass
@benfred
Copy link
Owner

benfred commented Dec 8, 2018

thanks for the bug report! I've managed to reproduce this - it's caused by some code that was added so that you can profile from the host os into processes running in a docker container. I will have a fix out shortly

benfred added a commit that referenced this issue Dec 8, 2018
We were trying to change the namespace even when it wasn't necessary,
causing issues like #68.

Fix by comparing the target processes namespace to ours by reading the
softlinks appropiately, and also just logging a warning if we failed
to set the namespace if we failed for any reason.
@benfred
Copy link
Owner

benfred commented Dec 8, 2018

fixed in v0.1.10

@benfred benfred closed this as completed Dec 8, 2018
@lovepocky
Copy link

root@dd77dc76b-4ck4h:/opt/app# RUST_BACKTRACE=1 py-spy -p 1
Error: Failed to suspend process
Reason: EPERM: Operation not permitted
stack backtrace:
   0:           0x4eda5d - backtrace::backtrace::trace::h72eb519ea1c30979
   1:           0x4ec4b2 - backtrace::capture::Backtrace::new_unresolved::hcca2671986d02778
   2:           0x4ebe21 - failure::backtrace::internal::InternalBacktrace::new::h8aa243afed4a675f
   3:           0x4ec0d1 - <failure::backtrace::Backtrace as core::default::Default>::default::h9ee3359cb47a172e
   4:           0x4ec129 - failure::backtrace::Backtrace::new::h33aed36859968218
   5:           0x42648b - py_spy::process::os_impl::Lock::new::h19299266bf895b52
   6:           0x40b9f6 - py_spy::python_spy::PythonSpy::get_stack_traces::h562232d9b147cf02
   7:           0x40b627 - py_spy::python_spy::PythonSpy::retry_new::h9464b08b0438a692
   8:           0x44660d - py_spy::main::h2f1f6f64c8aa4e65
   9:           0x422b72 - std::rt::lang_start::{{closure}}::h479c2785e776f3b2
  10:           0x611952 - std::rt::lang_start_internal::{{closure}}::h6b5f7ae7c14f8493
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::he4712eb84191fb3b
                        at libstd/panicking.rs:310
  11:           0x621169 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:102
  12:           0x60412b - std::panicking::try::hd44c69d8635b9942
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::h4424b54e4de01dca
                        at libstd/panic.rs:392
                         - std::rt::lang_start_internal::h76f990c2b2c74932
                        at libstd/rt.rs:58
  13:           0x4482c4 - main
root@dd77dc76b-4ck4h:/opt/app# py-spy --version
py-spy 0.1.10

same issue in kubernetes pod
@benfred

@benfred
Copy link
Owner

benfred commented Dec 20, 2018

@lovepocky did you launch your container with the SYS_PTRACE capability enabled? https://github.com/benfred/py-spy/#how-do-i-run-py-spy-in-kubernetes

If you didn't launch with the SYS_PTRACE capability, you can still trace the process from the host os

@lovepocky
Copy link

I have added capabilities to k8s containers.

root@6ccb596f9d-4lqjd:/opt/app# capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend+eipBounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspendSecurebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked)uid=0(root)gid=0(root)groups=

and get cap_sys_ptrace in container

@CtheSky
Copy link

CtheSky commented Sep 12, 2019

Same here. With cap_sys_ptrace added to k8s containers as documented here. py-spy==0.1.8 works well, but 0.1.9, 0.1.10, 0.1.11 give the same error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants