Add API for capturing backtrace #1559

Aaron1011 · 2020-09-22T02:17:21Z

This PR adds two new Miri-defined extern functions:
miri_get_backtrace and miri_resolve_frame, which are documented in
the README. Together, they allow obtaining a backtrace for the currently
executing program.

I've added a test showing how these APIs are used. I've also prepared a
companion PR backtrace-rs, which will allow
backtrace::Backtrace::new() to work automatically under Miri.

Once these two PRs are merged, we will be able to print backtraces from
the normal Rust panic hook (since libstd is now using backtrace-rs).

A few notes:

Resolving the backtrace frames is very slow - you can actually see
each line being printed out one at a time. Some local testing showed
that this is not (primrary) caused by resolving a Span - it seems
to be just Miri being slow.
For the first time, we now interact directly with a user-defined
struct (instead of just executing the user-provided MIR that
manipulates the struct). To allow for future changes, I've added
a 'version' parameter (currently required to be 0). This should allow
us to change the MiriFrame struct should the need ever arise.
I used the approach suggested by @oli-obk - a returned backtrace
pointer consists of a base function allocation, with the 'offset'
used to encode the Span.lo. This allows losslessly reconstructing
the location information in miri_resolve_frame.
There are a few quirks on the backtrace-rs side:
- backtrace-rs calls getcwd() by default to try to simplify
  the filename. This results in an isolation error by default,
  which could be annoying when printing a backtrace from libstd.
- backtrace-rs tries to remove 'internal' frames (everything between
  the call to Backtrace::new() and the internal API call made by
  backtrace-rs) by comparing the returned frame pointer value to
  a Rust function pointer. This doesn't work due to the way we
  construct the frame pointers passed to the caller. We could
  attempt to support this kind of comparison, or just add a
  #[cfg(miri)] and ignore the frames ourselves.

Companion to rust-lang/miri#1559

Aaron1011 · 2020-09-22T02:18:54Z

The backtrace-rs PR is up at rust-lang/backtrace-rs#372

src/shims/foreign_items.rs

tests/run-pass/backtrace-api.rs

RalfJung · 2020-09-22T11:43:28Z

Resolving the backtrace frames is very slow - you can actually see
each line being printed out one at a time. Some local testing showed
that this is not (primrary) caused by resolving a Span - it seems
to be just Miri being slow.

I have seen string formatting being very slow before, so this is likely another case of that.

RalfJung · 2020-09-22T11:51:37Z

backtrace-rs calls getcwd() by default to try to simplify
the filename. This results in an isolation error by default,
which could be annoying when printing a backtrace from libstd.

This is not a new problem, it already leads to isolation error (instead of an empty backtrace) right now.
So this is just an instance of #1034.

RalfJung · 2020-09-22T11:52:10Z

README.md

+    /// and `MiriFrame` should be declared as follows:
+    ///
+    /// ```rust
+    /// struct MiriFrame {


Shouldn't this be repr(C)?

We use mplace_field to write to fields, which ends up using FieldsShape to map the definition-order fields to their actual offsets in memory.

Oh so we exploit that we actually have access to the user-defined type... sneaky.^^

tests/run-pass/backtrace-api.stderr

RalfJung · 2020-09-23T09:56:34Z

README.md

+    ///
+    /// The fields must be declared in exactly the same order as they appear in `MiriFrame` above.
+    /// This function can be called on any thread (not just the one which obtained `frame`)
+    fn miri_resolve_frame(version: u8, frame: *mut ()) -> MiriFrame;


What we are effectively doing here is define something like a syscall interface. I have not seen version numbers in a syscall interface before, but I have seen flag fields -- so maybe it would make sense to instead make this flags: u64? That seems to be a more common way of future-proofing an API.

Also, if we ever need to change the number of arguments, we should probably add a miri_resolve_frame_2 or so. Overall I think things will be least confusing if we can make this work and evolve like a normal C API, not exploiting the fact that Miri can do fancy things like determine how many arguments the caller actually passed (without varargs).

I've added a flags field instead. Currently, I just throw an error if it's not 0. We could return a Result instead, but I don't know if there would ever be a need for code to support multiple Miri versions.

src/shims/foreign_items.rs

README.md

Aaron1011 · 2020-09-24T23:10:44Z

@RalfJung I've addressed your comments

Aaron1011 · 2020-09-28T17:27:26Z

@RalfJung: I've addressed your latest round of comments

RalfJung · 2020-09-28T17:30:35Z

@Aaron1011 yeah I know, I just didn't have time this week-end to look at your PR.

RalfJung · 2020-09-28T17:31:40Z

But I was actually looking at it when you wrote your comment.^^
@bors r+

Thanks for the PR :)

bors · 2020-09-28T17:31:42Z

📌 Commit 2a2a93d has been approved by RalfJung

bors · 2020-09-28T17:31:50Z

⌛ Testing commit 2a2a93d with merge 2c9afae...

Add API for capturing backtrace This PR adds two new Miri-defined extern functions: `miri_get_backtrace` and `miri_resolve_frame`, which are documented in the README. Together, they allow obtaining a backtrace for the currently executing program. I've added a test showing how these APIs are used. I've also prepared a companion PR `backtrace-rs`, which will allow `backtrace::Backtrace::new()` to work automatically under Miri. Once these two PRs are merged, we will be able to print backtraces from the normal Rust panic hook (since libstd is now using backtrace-rs). A few notes: * Resolving the backtrace frames is *very* slow - you can actually see each line being printed out one at a time. Some local testing showed that this is not (primrary) caused by resolving a `Span` - it seems to be just Miri being slow. * For the first time, we now interact directly with a user-defined struct (instead of just executing the user-provided MIR that manipulates the struct). To allow for future changes, I've added a 'version' parameter (currently required to be 0). This should allow us to change the `MiriFrame` struct should the need ever arise. * I used the approach suggested by `@oli-obk` - a returned backtrace pointer consists of a base function allocation, with the 'offset' used to encode the `Span.lo`. This allows losslessly reconstructing the location information in `miri_resolve_frame`. * There are a few quirks on the `backtrace-rs` side: * `backtrace-rs` calls `getcwd()` by default to try to simplify the filename. This results in an isolation error by default, which could be annoying when printing a backtrace from libstd. * `backtrace-rs` tries to remove 'internal' frames (everything between the call to `Backtrace::new()` and the internal API call made by backtrace-rs) by comparing the returned frame pointer value to a Rust function pointer. This doesn't work due to the way we construct the frame pointers passed to the caller. We could attempt to support this kind of comparison, or just add a `#[cfg(miri)]` and ignore the frames ourselves.

README.md

RalfJung · 2020-09-28T17:34:34Z

@bors r+ retry

bors · 2020-09-28T17:34:35Z

📌 Commit 1670f8c has been approved by RalfJung

bors · 2020-09-28T17:34:43Z

⌛ Testing commit 1670f8c with merge e550540...

Add API for capturing backtrace This PR adds two new Miri-defined extern functions: `miri_get_backtrace` and `miri_resolve_frame`, which are documented in the README. Together, they allow obtaining a backtrace for the currently executing program. I've added a test showing how these APIs are used. I've also prepared a companion PR `backtrace-rs`, which will allow `backtrace::Backtrace::new()` to work automatically under Miri. Once these two PRs are merged, we will be able to print backtraces from the normal Rust panic hook (since libstd is now using backtrace-rs). A few notes: * Resolving the backtrace frames is *very* slow - you can actually see each line being printed out one at a time. Some local testing showed that this is not (primrary) caused by resolving a `Span` - it seems to be just Miri being slow. * For the first time, we now interact directly with a user-defined struct (instead of just executing the user-provided MIR that manipulates the struct). To allow for future changes, I've added a 'version' parameter (currently required to be 0). This should allow us to change the `MiriFrame` struct should the need ever arise. * I used the approach suggested by `@oli-obk` - a returned backtrace pointer consists of a base function allocation, with the 'offset' used to encode the `Span.lo`. This allows losslessly reconstructing the location information in `miri_resolve_frame`. * There are a few quirks on the `backtrace-rs` side: * `backtrace-rs` calls `getcwd()` by default to try to simplify the filename. This results in an isolation error by default, which could be annoying when printing a backtrace from libstd. * `backtrace-rs` tries to remove 'internal' frames (everything between the call to `Backtrace::new()` and the internal API call made by backtrace-rs) by comparing the returned frame pointer value to a Rust function pointer. This doesn't work due to the way we construct the frame pointers passed to the caller. We could attempt to support this kind of comparison, or just add a `#[cfg(miri)]` and ignore the frames ourselves.

@oli-obk

This PR adds two new Miri-defined extern functions: `miri_get_backtrace` and `miri_resolve_frame`, which are documented in the README. Together, they allow obtaining a backtrace for the currently executing program. I've added a test showing how these APIs are used. I've also prepared a companion PR `backtrace-rs`, which will allow `backtrace::Backtrace::new()` to work automatically under Miri. Once these two PRs are merged, we will be able to print backtraces from the normal Rust panic hook (since libstd is now using backtrace-rs). A few notes: * Resolving the backtrace frames is *very* slow - you can actually see each line being printed out one at a time. Some local testing showed that this is not (primrary) caused by resolving a `Span` - it seems to be just Miri being slow. * For the first time, we now interact directly with a user-defined struct (instead of just executing the user-provided MIR that manipulates the struct). To allow for future changes, I've added a 'version' parameter (currently required to be 0). This should allow us to change the `MiriFrame` struct should the need ever arise. * I used the approach suggested by @oli-obk - a returned backtrace pointer consists of a base function allocation, with the 'offset' used to encode the `Span.lo`. This allows losslessly reconstructing the location information in `miri_resolve_frame`. * There are a few quirks on the `backtrace-rs` side: * `backtrace-rs` calls `getcwd()` by default to try to simplify the filename. This results in an isolation error by default, which could be annoying when printing a backtrace from libstd. * `backtrace-rs` tries to remove 'internal' frames (everything between the call to `Backtrace::new()` and the internal API call made by backtrace-rs) by comparing the returned frame pointer value to a Rust function pointer. This doesn't work due to the way we construct the frame pointers passed to the caller. We could attempt to support this kind of comparison, or just add a `#[cfg(miri)]` and ignore the frames ourselves.

RalfJung · 2020-09-28T17:44:49Z

Looks like I was right to be worried about test stability... we already have a change in the output:

-RUSTLIB/src/rust/library/std/src/rt.rs:LL:COL (std::rt::lang_start::<()>::{{closure}}#0)
+RUSTLIB/src/rust/library/std/src/rt.rs:LL:COL (std::rt::lang_start::<()>::{closure#0})

Can you adjust normalization to remove more things? We might actually just want to cut off the function name after the first ::<.

bors · 2020-09-28T17:46:06Z

💔 Test failed - checks-travis

Aaron1011 · 2020-09-28T17:48:12Z

This looks like it was caused by https://github.com/rust-lang/rust/pull/76176/files. That PR also caused a lot of changes in rustc UI tests, so I don't think this issue is specific to the way we're printing backtrace frames.

However, I'm fine with normalizing more of the output.

Aaron1011 · 2020-09-28T17:56:50Z

@RalfJung: I've normalized out all turbofished generic arguments (::<>) from the stderr.

Companion to rust-lang/miri#1559

RalfJung · 2020-09-28T19:16:45Z

Thanks!
@bors r+

bors · 2020-09-28T19:16:47Z

📌 Commit 7fba3c2 has been approved by RalfJung

bors · 2020-09-28T19:16:54Z

⌛ Testing commit 7fba3c2 with merge 510e62c...

bors · 2020-09-28T20:13:31Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: RalfJung
Pushing 510e62c to master...

Companion to rust-lang/miri#1559

Aaron1011 added a commit to Aaron1011/backtrace-rs that referenced this pull request Sep 22, 2020

Add support for Miri backtraces

085ee70

Companion to rust-lang/miri#1559

Aaron1011 mentioned this pull request Sep 22, 2020

Add support for Miri backtraces rust-lang/backtrace-rs#372

Merged

Aaron1011 force-pushed the new-miri-backtrace branch from e892458 to 4814f9d Compare September 22, 2020 02:21

oli-obk reviewed Sep 22, 2020

View reviewed changes

src/shims/foreign_items.rs Outdated Show resolved Hide resolved

oli-obk reviewed Sep 22, 2020

View reviewed changes

tests/run-pass/backtrace-api.rs Show resolved Hide resolved

RalfJung reviewed Sep 22, 2020

View reviewed changes

tests/run-pass/backtrace-api.stderr Outdated Show resolved Hide resolved

RalfJung reviewed Sep 23, 2020

View reviewed changes