Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify interaction between panic = abort and backtrace capture #397

Open
yaahc opened this issue Dec 7, 2020 · 25 comments
Open

clarify interaction between panic = abort and backtrace capture #397

yaahc opened this issue Dec 7, 2020 · 25 comments
Labels

Comments

@yaahc
Copy link
Member

yaahc commented Dec 7, 2020

In our project at work we've been seeing issues with backtrace's not capturing the full context of the application state when we panic, where as the builtin backtrace printer for the default provided panic hook has no such issues.

ZcashFoundation/zebra#1471 shows an example of a backtrace that gets cut off, it only shows the frames pushed onto the stack after calling expect, but doesn't include any of our application code's frames otherwise, where as the metadata of the issue captured the location the panic was created from via #[track_caller] just fine.

I'm not convinced I understand what is going on here, or whether this is a bug or expected behaviour, disabling panic = abort fixes the issue though, at least in the playground.

@alexcrichton
Copy link
Member

where as the builtin backtrace printer for the default provided panic hook has no such issues

Hm can you clarify what the default here is? Do you mean that libstd's own backtrace printing prints a full backtrace but this crate doesn't print a backtrace at all? (when the two are "called" from the same context).

I think that panic=abort vs panic=unwind can affect codegen and it may be affecting unwind tables as well. I'm not 100% certain about that though, and it wouldn't explain the difference in this crate vs libstd.

A bit of a long shot, but would it be possible to get some source code I could poke around?

@alexcrichton
Copy link
Member

Hm scratch that

RUST_BACKTRACE=1 CARGO_PROFILE_DEV_PANIC=abort cargo run --example backtrace

in this repository on macOS prints nothing for this crate itself, but libstd prints a backtrace for the panic! I'm perplexed and something weird is going on here, I'll get back to you after investigating!

@yaahc
Copy link
Member Author

yaahc commented Dec 7, 2020

@alexcrichton some background I know that the default panic hook from std provides custom backtrace formatting machinery so that it can avoid needing to allocate, so my guess is that this implementation somehow works with panic = abort but the ones in backtrace::Backtrace and std::backtrace::Backtrace do not. This came up recently in the Backtrace stabilization issue: rust-lang/rust#71706 (comment).

And yea, here's my simple example program

fn main() {
    frame1();
}

fn frame1() {
    frame2();
}

fn frame2() {
    std::env::set_var("RUST_BACKTRACE", "full");
    let backtrace = backtrace::Backtrace::new();
    eprintln!("Backtrace:\n{:?}", backtrace);
    panic!("foo");
}

And the output I get in panic = unwind

Backtrace:
   0: backtrace::frame2
             at examples/backtrace.rs:11:20
   1: backtrace::frame1
             at examples/backtrace.rs:6:4
   2: backtrace::main
             at examples/backtrace.rs:2:4
   3: std::rt::lang_start::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67:33
   4: std::rt::lang_start_internal::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:52:12
      std::panicking::try::do_call
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:305:39
   5: __rust_maybe_catch_panic
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libpanic_unwind/lib.rs:86:7
   6: std::panicking::try
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:281:12
      std::panic::catch_unwind
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panic.rs:394:13
      std::rt::lang_start_internal
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:51:24
   7: std::rt::lang_start
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67:4
   8: main
   9: __libc_start_main
  10: _start

thread 'main' panicked at 'foo', examples/backtrace.rs:13:5
stack backtrace:
   0:     0x55cbd32bfc64 - backtrace::backtrace::libunwind::trace::h90669f559fb267f0
                               at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x55cbd32bfc64 - backtrace::backtrace::trace_unsynchronized::hffde4e353d8f2f9a
                               at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x55cbd32bfc64 - std::sys_common::backtrace::_print_fmt::heaf44068b7eaaa6a
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x55cbd32bfc64 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h88671019cf081de2
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x55cbd32d73fc - core::fmt::write::h4e6a29ee6319c9fd
                               at src/libcore/fmt/mod.rs:1052
   5:     0x55cbd32be197 - std::io::Write::write_fmt::hf06b1c86d898d7d6
                               at src/libstd/io/mod.rs:1426
   6:     0x55cbd32c1d15 - std::sys_common::backtrace::_print::h404ff5f2b50cae09
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x55cbd32c1d15 - std::sys_common::backtrace::print::hcc4377f1f882322e
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x55cbd32c1d15 - std::panicking::default_hook::{{closure}}::hc172eff6f35b7f39
                               at src/libstd/panicking.rs:204
   9:     0x55cbd32c1a01 - std::panicking::default_hook::h7a68887d113f8029
                               at src/libstd/panicking.rs:224
  10:     0x55cbd32c231a - std::panicking::rust_panic_with_hook::hb7ad5693188bdb00
                               at src/libstd/panicking.rs:472
  11:     0x55cbd31db41b - std::panicking::begin_panic::h5b22d6d0d2936270
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:399
  12:     0x55cbd31da55b - backtrace::frame2::h0466262d13bbea09
                               at examples/backtrace.rs:13
  13:     0x55cbd31da456 - backtrace::frame1::h32238f8b98516416
                               at examples/backtrace.rs:6
  14:     0x55cbd31da446 - backtrace::main::h9ada77cd6bbdcc61
                               at examples/backtrace.rs:2
  15:     0x55cbd31da6f0 - std::rt::lang_start::{{closure}}::haed21695f16ce364
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67
  16:     0x55cbd32c1de3 - std::rt::lang_start_internal::{{closure}}::hb26e39676675046f
                               at src/libstd/rt.rs:52
  17:     0x55cbd32c1de3 - std::panicking::try::do_call::he4701ab6e48d80c0
                               at src/libstd/panicking.rs:305
  18:     0x55cbd32c3fe7 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  19:     0x55cbd32c27c0 - std::panicking::try::hd3de25f3cb7024b8
                               at src/libstd/panicking.rs:281
  20:     0x55cbd32c27c0 - std::panic::catch_unwind::h86c02743a24e3d92
                               at src/libstd/panic.rs:394
  21:     0x55cbd32c27c0 - std::rt::lang_start_internal::h9cf8802361ad86c2
                               at src/libstd/rt.rs:51
  22:     0x55cbd31da6c9 - std::rt::lang_start::h527b728873cea05d
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67
  23:     0x55cbd31da59a - main
  24:     0x7f7d210980b3 - __libc_start_main
  25:     0x55cbd31da17e - _start
  26:                0x0 - <unknown>

vs panic = abort

Backtrace:
   0: backtrace::frame2
             at examples/backtrace.rs:11:20
   1: backtrace::frame1
             at examples/backtrace.rs:6:4
   2: backtrace::main
             at examples/backtrace.rs:2:4
   3: std::rt::lang_start::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67:33

thread 'main' panicked at 'foo', examples/backtrace.rs:13:5
stack backtrace:
   0:     0x56514d8b2d44 - backtrace::backtrace::libunwind::trace::h90669f559fb267f0
                               at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x56514d8b2d44 - backtrace::backtrace::trace_unsynchronized::hffde4e353d8f2f9a
                               at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x56514d8b2d44 - std::sys_common::backtrace::_print_fmt::heaf44068b7eaaa6a
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x56514d8b2d44 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h88671019cf081de2
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x56514d8c9e1c - core::fmt::write::h4e6a29ee6319c9fd
                               at src/libcore/fmt/mod.rs:1052
   5:     0x56514d8b1277 - std::io::Write::write_fmt::hf06b1c86d898d7d6
                               at src/libstd/io/mod.rs:1426
   6:     0x56514d8b4da5 - std::sys_common::backtrace::_print::h404ff5f2b50cae09
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x56514d8b4da5 - std::sys_common::backtrace::print::hcc4377f1f882322e
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x56514d8b4da5 - std::panicking::default_hook::{{closure}}::hc172eff6f35b7f39
                               at src/libstd/panicking.rs:204
   9:     0x56514d8b4a91 - std::panicking::default_hook::h7a68887d113f8029
                               at src/libstd/panicking.rs:224
  10:     0x56514d8b53aa - std::panicking::rust_panic_with_hook::hb7ad5693188bdb00
                               at src/libstd/panicking.rs:472
  11:     0x56514d7d63cb - std::panicking::begin_panic::h3e3ab4e410875c09
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:399
  12:     0x56514d7d54c7 - backtrace::frame2::hfd96dced912a452c
                               at examples/backtrace.rs:13
  13:     0x56514d7d53d6 - backtrace::frame1::h5d7c5587ebab67dc
                               at examples/backtrace.rs:6
  14:     0x56514d7d53c6 - backtrace::main::h1f6e5223b7ef4de0
                               at examples/backtrace.rs:2
  15:     0x56514d7d5570 - std::rt::lang_start::{{closure}}::h1a8399b1346eb913
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67
Aborted (core dumped)

@alexcrichton
Copy link
Member

Some quick investigation (heading out now, will dig in more later), is that this is related to how the library that generates the backtrace is itself compiled.

In your code, for example, the backtrace crate is compiled with -Cpanic=abort but the standard library is compiled with -Cpanic=unwind (because that's how we ship it). If you run via:

RUST_BACKTRACE=1 CARGO_PROFILE_DEV_PANIC=abort cargo +nightly run -Z build-std=std,panic_abort --target x86_64-apple-darwin 

(substituting the right target) I think that you'll see that libstd can't generate a backtrace either (or at least that's what I get locally). This makes me think that if the request to unwind comes from code compiled with -Cpanic=abort it doesn't unwind, but if the request to unwind comes from -Cpanic=unwind code then it works.

It looks though like you're getting different output than I am for effectively the same example. You're on Linux, right?

@yaahc
Copy link
Member Author

yaahc commented Dec 8, 2020

It looks though like you're getting different output than I am for effectively the same example. You're on Linux, right?

yea, but this is using stable and I'm not setting that CARGO_PROFILE_DEV_PANIC=abort flag, I was using the cargo.toml profile section to set it.

edit: I see you were using stable before too...

@yaahc
Copy link
Member Author

yaahc commented Dec 8, 2020

(substituting the right target) I think that you'll see that libstd can't generate a backtrace either (or at least that's what I get locally). This makes me think that if the request to unwind comes from code compiled with -Cpanic=abort it doesn't unwind, but if the request to unwind comes from -Cpanic=unwind code then it works.

Just some clarification so I can keep up. "request to unwind", is the backtrace capture requesting an unwind? What does requesting an unwind mean? Is it actually unwinding?

@alexcrichton
Copy link
Member

Oh sorry so to clarify CARGO_PROFILE_DEV_PANIC=abort is just an alternate way of settting the profile in Cargo.toml, they do the same thing. It's just easier to gist snippets using CLI stuff instead of gisting files as well.

Using your example program from above this is what I get:

cargo +stable run - 2 backtraces
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/wat`
Backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/libunwind.rs:90:5
      backtrace::backtrace::trace_unsynchronized
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/capture.rs:140:22
   4: wat::frame2
             at src/main.rs:11:21
   5: wat::frame1
             at src/main.rs:6:5
   6: wat::main
             at src/main.rs:2:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227:5
   8: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:137:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:66:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:381:40
      std::panicking::try
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panic.rs:382:14
      std::rt::lang_start_internal
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:51:25
  11: std::rt::lang_start
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:65:5
  12: _main

thread 'main' panicked at 'foo', src/main.rs:13:5
stack backtrace:
   0:        0x10cf3a814 - std::backtrace_rs::backtrace::libunwind::trace::h6df1416181381e81
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
   1:        0x10cf3a814 - std::backtrace_rs::backtrace::trace_unsynchronized::h792524067a83cded
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/mod.rs:66
   2:        0x10cf3a814 - std::sys_common::backtrace::_print_fmt::he614892186429f3f
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:79
   3:        0x10cf3a814 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hcfc48256a5ab8835
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:58
   4:        0x10cf50ab0 - core::fmt::write::haf3903118f694c48
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/fmt/mod.rs:1080
   5:        0x10cf38506 - std::io::Write::write_fmt::h7385463ac87804ed
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/io/mod.rs:1516
   6:        0x10cf3c23f - std::sys_common::backtrace::_print::ha64a9740ea6183e6
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:61
   7:        0x10cf3c23f - std::sys_common::backtrace::print::h1fad353b070e1eef
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:48
   8:        0x10cf3c23f - std::panicking::default_hook::{{closure}}::h91bd4c58cf71392b
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:208
   9:        0x10cf3bf0d - std::panicking::default_hook::h7bd29c87df967048
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:227
  10:        0x10cf3c80b - std::panicking::rust_panic_with_hook::hae2b05f08a320721
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:577
  11:        0x10cea0544 - std::panicking::begin_panic::{{closure}}::ha475311d882de50e
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:506
  12:        0x10ce88c78 - std::sys_common::backtrace::__rust_end_short_backtrace::h03577718979756e3
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:153
  13:        0x10cf54c92 - std::panicking::begin_panic::he3bed0a1f47791a0
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:505
  14:        0x10ce63aab - wat::frame2::h065882373d0f4cca
                               at /Users/acrichton/code/wat/src/main.rs:13
  15:        0x10ce639c9 - wat::frame1::h0b1c723c6453f077
                               at /Users/acrichton/code/wat/src/main.rs:6
  16:        0x10ce639b9 - wat::main::h020fae008cb58e33
                               at /Users/acrichton/code/wat/src/main.rs:2
  17:        0x10ce63e0e - core::ops::function::FnOnce::call_once::h81c133c675281bfd
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227
  18:        0x10ce63b01 - std::sys_common::backtrace::__rust_begin_short_backtrace::h059c529ce5eeae7e
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:137
  19:        0x10ce63ce4 - std::rt::lang_start::{{closure}}::hed867cdf1941124c
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:66
  20:        0x10cf3cb84 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h4daab3ed47411afb
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:259
  21:        0x10cf3cb84 - std::panicking::try::do_call::h5f7f7b1d8bd7dc86
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:381
  22:        0x10cf3cb84 - std::panicking::try::h29495f6475fc1ef1
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:345
  23:        0x10cf3cb84 - std::panic::catch_unwind::h1ebce9249c33ec53
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panic.rs:382
  24:        0x10cf3cb84 - std::rt::lang_start_internal::hd38bb63f9540b327
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:51
  25:        0x10ce63cc1 - std::rt::lang_start::h9fce25cc1f13d835
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:65
  26:        0x10ce63ae2 - _main
CARGO_PROFILE_DEV_PANIC=abort cargo +stable run - truncated backtraces
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/wat`
Backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/libunwind.rs:90:5
      backtrace::backtrace::trace_unsynchronized
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /Users/acrichton/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.55/src/capture.rs:140:22
   4: wat::frame2
             at src/main.rs:11:21
   5: wat::frame1
             at src/main.rs:6:5
   6: wat::main
             at src/main.rs:2:5

thread 'main' panicked at 'foo', src/main.rs:13:5
stack backtrace:
   0:        0x10465e8d4 - std::backtrace_rs::backtrace::libunwind::trace::h6df1416181381e81
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
   1:        0x10465e8d4 - std::backtrace_rs::backtrace::trace_unsynchronized::h792524067a83cded
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/mod.rs:66
   2:        0x10465e8d4 - std::sys_common::backtrace::_print_fmt::he614892186429f3f
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:79
   3:        0x10465e8d4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hcfc48256a5ab8835
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:58
   4:        0x104674400 - core::fmt::write::haf3903118f694c48
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/fmt/mod.rs:1080
   5:        0x10465c5c6 - std::io::Write::write_fmt::h7385463ac87804ed
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/io/mod.rs:1516
   6:        0x10466026f - std::sys_common::backtrace::_print::ha64a9740ea6183e6
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:61
   7:        0x10466026f - std::sys_common::backtrace::print::h1fad353b070e1eef
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:48
   8:        0x10466026f - std::panicking::default_hook::{{closure}}::h91bd4c58cf71392b
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:208
   9:        0x10465ff3d - std::panicking::default_hook::h7bd29c87df967048
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:227
  10:        0x10466083b - std::panicking::rust_panic_with_hook::hae2b05f08a320721
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:577
  11:        0x1045cf6a7 - std::panicking::begin_panic::{{closure}}::h91da006551d072ed
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:506
  12:        0x104606a18 - std::sys_common::backtrace::__rust_end_short_backtrace::h861cf5a48e5296f2
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:153
  13:        0x1046785d5 - std::panicking::begin_panic::h05aa747b0ee1e6b2
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:505
  14:        0x104594e4f - wat::frame2::h065882373d0f4cca
                               at /Users/acrichton/code/wat/src/main.rs:13
  15:        0x104594d89 - wat::frame1::h0b1c723c6453f077
                               at /Users/acrichton/code/wat/src/main.rs:6
  16:        0x104594d79 - wat::main::h020fae008cb58e33
                               at /Users/acrichton/code/wat/src/main.rs:2
CARGO_PROFILE_DEV_PANIC=abort cargo +nightly run --target x86_64-apple-darwin -Zbuild-std=std,panic_abort - no backtraces
    Finished dev [unoptimized + debuginfo] target(s) in 1.25s
     Running `target/x86_64-apple-darwin/debug/wat`
Backtrace:

thread 'main' panicked at 'foo', src/main.rs:13:5
stack backtrace:

And then running Cargo inside of the backtrace-rs repository iteslf:

cargo +stable run --example backtrace - has a backtrace
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/examples/backtrace`
   0: backtrace::backtrace::libunwind::trace
             at src/backtrace/libunwind.rs:90:5
      backtrace::backtrace::trace_unsynchronized
             at src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at src/capture.rs:140:22
   4: backtrace::main
             at examples/backtrace.rs:4:22
   5: core::ops::function::FnOnce::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227:5
   6: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:137:18
   7: std::rt::lang_start::{{closure}}
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:66:18
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:381:40
      std::panicking::try
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panic.rs:382:14
      std::rt::lang_start_internal
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:51:25
   9: std::rt::lang_start
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/rt.rs:65:5
  10: _main


CARGO_PROFILE_DEV_PANIC=abort cargo +stable run --example backtrace - no backtrace
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/examples/backtrace`

Also to clarify, when I say "request unwind" I mean the function which called _Unwind_Backtrace itself.


In general though I think that there may be room for another rustc flag here. Unwinding on ELF and Mach-O (Unix) platforms primarily goes through DWARF-based unwinding. This varies a bit per platform, but if the unwinding tables aren't present in an executable then the unwinder doesn't know what to do. I think it can sort of get through some functions above by matching standard prologues/epilogues, but honestly I'm not entirely sure why partial backtraces are present sometimes.

In any case with -Cpanic=abort I believe rustc does not generate the uwtable attribute on LLVM functions (unlike how it does so in the -Cpanic=unwind case). That means, I think, that with -Cpanic=abort there is no unwind table information and the unwinder is left to do whatever it can to try to unwind, but it can often fail and not give 100% accurate results.

To answer the original question of this issue, there is no original intention for how panic=abort is supposed to interact with this crate. It seems it doesn't work super well today given how the unwinders we rely on rely on unwind tables and they're not present in panic=abort. That may mean we'll want a rustc flag -Cgenerate-unwind-tables for panic=abort binaries that still want backtraces for unwinds and such.

I think I'm still missing a bit on why the backtraces at least partially work, but I think otherwise this is basically the state of things.

@yaahc
Copy link
Member Author

yaahc commented Dec 9, 2020

Oh okay, that makes a lot of sense, and a -Cgenerate-unwind-tables sounds very helpful. If you want / need any help implementing the feature I can contribute to this as part of my job.

@alexcrichton
Copy link
Member

Oh I'm happy to provide advice/help/review, but unfortunately I don't have time to implement this myself. Feel free to take it on if you'd like!

@bjorn3
Copy link
Member

bjorn3 commented Dec 9, 2020

Could we always have the unwind tables, but just omit the landingpads when using -Cpanic=abort? (Maybe gating it on -Cdebuginfo?)

@yaahc
Copy link
Member Author

yaahc commented Dec 9, 2020

Oh I'm happy to provide advice/help/review, but unfortunately I don't have time to implement this myself. Feel free to take it on if you'd like!

totally understandable. And yea, if you could write up some instructions on where to start and what the steps would be that would help a lot, and I'll be able to start on this right away.

@alexcrichton
Copy link
Member

Sure yeah! It looks like there's a custom-target-spec option called requires_uwtable which presumably forcibly generates unwind tables for a target. I think we'll want to basically promote that to a CLI codegen flag as well (probably something like -Cunwind-tables={true,false}. My recommendation would be to add a method to Session called generate_unwind_tables() which respects the CLI flag if given, and otherwise falls back to the target-specific configuration.

@ghost
Copy link

ghost commented Jan 18, 2021

@rustbot modify labels +easy +good-first-issue +project-error-handling

@rustbot
Copy link

rustbot commented Jan 18, 2021

Error: This repository is not enabled to use triagebot.
Add a triagebot.toml in the root of the master branch to enable it.

Please let @rust-lang/release know if you're having trouble with this bot.

@tmandry
Copy link
Member

tmandry commented Jan 27, 2021

This option exists! -C force-unwind-tables

@yaahc
Copy link
Member Author

yaahc commented Jan 27, 2021

This option exists! -C force-unwind-tables

oh my goddd tylerrrrr ❤️

@alexcrichton
Copy link
Member

Oh wow, thanks @tmandry for pointing that out! I completely forgot that existed... I think that this issue may be mostly done in that case? Although it may be good to perhaps document that in the crate somewhere?

@yaahc
Copy link
Member Author

yaahc commented Feb 2, 2021

Just wanted to go ahead and confirm that this worked for me, after enabling force-unwind-tables + panic = abort I now get this as the backtrace

Backtrace:
   0: panic::frame2
             at src/main.rs:11:21
   1: panic::frame1
             at src/main.rs:6:5
   2: panic::main
             at src/main.rs:2:5
   3: core::ops::function::FnOnce::call_once
             at /home/jlusby/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
   4: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /home/jlusby/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
   5: std::rt::lang_start::{{closure}}
             at /home/jlusby/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:66:18
   6: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/a2f8f6281817d430e20726128b739d3c6708561c/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/a2f8f6281817d430e20726128b739d3c6708561c/library/std/src/panicking.rs:379:40
      std::panicking::try
             at /rustc/a2f8f6281817d430e20726128b739d3c6708561c/library/std/src/panicking.rs:343:19
      std::panic::catch_unwind
             at /rustc/a2f8f6281817d430e20726128b739d3c6708561c/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at /rustc/a2f8f6281817d430e20726128b739d3c6708561c/library/std/src/rt.rs:51:25
   7: std::rt::lang_start
             at /home/jlusby/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:65:5
   8: main
   9: __libc_start_main
  10: _start

instead of the original

Backtrace:
   0: panic::frame2
             at src/main.rs:11:21
   1: panic::frame1
             at src/main.rs:6:5
   2: panic::main
             at src/main.rs:2:5
   3: core::ops::function::FnOnce::call_once
             at /home/jlusby/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5

alexcrichton added a commit that referenced this issue Feb 2, 2021
Following the discussion in #397 seems like this is a missing section of
the documentation! It seems worthwhile to document some pitfalls and
mitigations if possible.
@alexcrichton
Copy link
Member

I've tried to add some documentation for this in #407

@yaahc
Copy link
Member Author

yaahc commented Feb 2, 2021

I've tried to add some documentation for this in #407

awesome, I was thinking we might also want to add a note about force-unwind-tables to various pages of panic = abort related documentation. A cursory google search shows these as possible targets:

@alexcrichton
Copy link
Member

Seems reasonable to me!

alexcrichton added a commit that referenced this issue Feb 2, 2021
* Document unwinding limitations

Following the discussion in #397 seems like this is a missing section of
the documentation! It seems worthwhile to document some pitfalls and
mitigations if possible.

* Update split-debuginfo support to nightly
@stevemk14ebr
Copy link

stevemk14ebr commented May 22, 2021

Hello, this flag does not seem to work in release mode. The unwind table is there by default in debug but never in release even with the force unwind tables flag set to yes:

Cargo.toml

[profile.dev]
panic = "abort"
lto = true
debug = 2
debug-assertions = true

[profile.release]
panic = "abort"
lto = true
debug = 2
debug-assertions = true

.cargo/config

[build]
target = "x86_64-pc-windows-msvc"

rustflags = [
    "-C", "panic=abort",
    "-C", "force-unwind-tables=yes",

    # Pre Link Args
    "-Z", "pre-link-arg=/NOLOGO",
    "-Z", "pre-link-arg=/NXCOMPAT",
    "-Z", "pre-link-arg=/NODEFAULTLIB",
    "-Z", "pre-link-arg=/SUBSYSTEM:NATIVE",
    "-Z", "pre-link-arg=/DRIVER",
    "-Z", "pre-link-arg=/DYNAMICBASE",
    "-Z", "pre-link-arg=/MANIFEST:NO",

    # Post Link Args
    "-C", "link-arg=/OPT:REF,ICF",
    "-C", "link-arg=/ENTRY:driver_entry",
    "-C", "link-arg=/MERGE:.edata=.rdata",
    "-C", "link-arg=/MERGE:.rustc=.data",
    "-C", "link-arg=/INTEGRITYCHECK"
]
Running `rustc --crate-name RSTrace --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type cdylib --emit=dep-info,link -C opt-level=3 -C panic=abort -C lto -C debuginfo=2 -C debug-assertions=on -C overflow-checks=off -C metadata=0783d6da86fe062c --out-dir C:\Users\steve\source\repos\RSTrace\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\steve\source\repos\RSTrace\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\steve\source\repos\RSTrace\target\release\deps --extern ntapi=C:\Users\steve\source\repos\RSTrace\target\x86_64-pc-windows-msvc\release\deps\libntapi-7f7dba2dc6ae5f22.rlib --extern utf16_lit=C:\Users\steve\source\repos\RSTrace\target\release\deps\utf16_lit-bbee57a78fade9e8.dll --extern wasmi=C:\Users\steve\source\repos\RSTrace\target\x86_64-pc-windows-msvc\release\deps\libwasmi-aabf18157cf6d5d2.rlib --extern winapi=C:\Users\steve\source\repos\RSTrace\target\x86_64-pc-windows-msvc\release\deps\libwinapi-0eb57b396aba78ff.rlib -C panic=abort -C force-unwind-tables=yes -Z pre-link-arg=/NOLOGO -Z pre-link-arg=/NXCOMPAT -Z pre-link-arg=/NODEFAULTLIB -Z pre-link-arg=/SUBSYSTEM:NATIVE -Z pre-link-arg=/DRIVER -Z pre-link-arg=/DYNAMICBASE -Z pre-link-arg=/MANIFEST:NO -C link-arg=/OPT:REF,ICF -C link-arg=/ENTRY:driver_entry -C link-arg=/MERGE:.edata=.rdata -C link-arg=/MERGE:.rustc=.data -C link-arg=/INTEGRITYCHECK -L "native=C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\km\x64"`

Then inspecting my binary there is not RUNTIME_FUNCTION entry, which would be expected if the unwind table was generated. This unwind table is strictly necessary for the binary interface i'm interacting with, are there any other techniques to force this table to generate, i only need an entry for one function in particular?

image

@brandonros
Copy link

force-unwind-tables

any way to put this in like... Cargo.toml or .cargo/config.toml?

@bjorn3
Copy link
Member

bjorn3 commented Jun 9, 2022

You can put it in the rustflags field of .cargo/config.toml.

@brandonros
Copy link

for anybody following along

[build]
rustflags = ["-C", "force-unwind-tables"]

emilk added a commit to rerun-io/rerun that referenced this issue Jun 23, 2024
The `backtrace` crate isn't compatible with `panic="abort"` (see
[docs](https://docs.rs/backtrace/latest/backtrace/) and
[issue](rust-lang/backtrace-rs#397)).

One great feature we have in egui is pressing down all modifier keys
on your keyboard and seeing the backtrace to the widget you hover.
This is awesome for when you want to change the code for some
specific part of the UI. But it didn't work, at least not always.

This PR fixes it.

We still use `panic="abort"` in release builds, for the benefit
of smaller binaries.

* See also emilk/egui#4696
emilk added a commit to rerun-io/rerun that referenced this issue Jun 24, 2024
The `backtrace` crate isn't compatible with `panic="abort"` (see
[docs](https://docs.rs/backtrace/latest/backtrace/) and
[issue](rust-lang/backtrace-rs#397)).

One great feature we have in egui is pressing down all modifier keys on
your keyboard and seeing the backtrace to the widget you hover. This is
awesome for when you want to change the code for some specific part of
the UI. But it didn't work, at least not always.

This PR fixes it.

We still use `panic="abort"` in release builds, for the benefit of
smaller binaries.

* See also emilk/egui#4696


https://github.com/rerun-io/rerun/assets/1148717/cff9ddbe-838c-4e67-937a-e7dd97d1537f


### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/6608?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/6608?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!

- [PR Build Summary](https://build.rerun.io/pr/6608)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants