Reuse cargo's binary selection in cargo miri

[RalfJung]

Currently, cargo miri run will run all the binaries in the project (ouch) and cargo miri test will test all binaries and libraries. This leads to issues like #700 (we run tests even when they are disabled), and it means one cannot run examples with Miri.

We should find a way to re-use cargo's selection of binaries, tests and so on.

[RalfJung]

Related: #1001

[jonhoo]

To add to this, I think it'd be helpful to support all the various target specifiers that cargo build (and cargo test) supports. A lightly edited list:

$ cargo help build
...
OPTIONS:
    -p, --package <SPEC>...          Package to build (see `cargo help pkgid`)
        --workspace                  Build all packages in the workspace
        --exclude <SPEC>...          Exclude packages from the build
        --lib                        Build only this package's library
        --bin <NAME>...              Build only the specified binary
        --bins                       Build all binaries
        --example <NAME>...          Build only the specified example
        --examples                   Build all examples
        --test <NAME>...             Build only the specified test target
        --tests                      Build all tests
        --bench <NAME>...            Build only the specified bench target
        --benches                    Build all benches
        --all-targets                Build all targets
        --release                    Build artifacts in release mode, with optimizations
        --features <FEATURES>...     Space-separated list of features to activate
        --all-features               Activate all available features
        --no-default-features        Do not activate the `default` feature

All packages in the workspace are built if the `--workspace` flag is supplied. The
`--workspace` flag is automatically assumed for a virtual manifest.
Note that `--exclude` has to be specified in conjunction with the `--workspace` flag.

This may just be a matter of being able to pass custom flags to cargo build, which I don't think is currently possible, no matter how many -- you pass :p

[RalfJung]

This may just be a matter of being able to pass custom flags to cargo build, which I don't think is currently possible, no matter how many -- you pass :p

Maybe... historically we used cargo rustc but that changed (fairly) recently, so maybe resolving this issue here is now easier than before. But currently we still use cargo_metadata to parse the Cargo.toml and get a list of all targets ourselves and then iterate over them.

Basically, all flags before the first -- are meant to go to cargo. But how weird would it be for cargo miri run/cargo miri test to support the flags of cargo build (and not cargo run/cargo test)?

[RalfJung]

Hm, but relying on cargo build to do the enumeration could (a) end up running multiple Miri in parallel (I think you saw that with some of your flag combinations), and (b) make #584 much harder.

[CAD97]

one cannot run examples with Miri

I'm.. having the opposite problem right now. cargo +nightly miri test --all-features --all-targets is running both the tests in my examples (intended) and the examples' main (unintended). The invocation without miri, cargo +nightly test --all-features --all-targets does not run the example main. (This is important for this example, because the main is a REPL, which obviously isn't good for miri on CI.)

I'm currently using a cfg to just replace the example main under miri, but it'd be nice if I didn't have to (and still run the examples' tests under miri).

Here's an example run.

[RalfJung]

That's odd, the match here should only cover tests. My best guess is the extra flags are confusing the cargo check call... I don't know what cargo check --bin <name> --profile test --all-features --all-targets does but it is probably weird.^^

[CAD97]

IIUC, --all-targets overrides any narrowing options. IOW, in the presence of none of --lib, --bin <NAME>..., --bins, example <NAME>..., --examples, --test <NAME>..., --tests, --bench <NAME>..., --benches, or --all-targets, the default targets are built (IIUC, --lib if no bins are present, --bin <name> if a single bin is (or default bin is configured), and error if multiple bins are present (and no default bin is configured).) But if any of the above flags are present, the default targets are not included, and each listed target is added to the list of targets to [check|build|test|run].

So cargo check --lib --profile test --all-targets will add the lib target to the check list and then add all of the targets to the check list. That would also explain why I thought I saw tests multiple times -- because the library tests are actually getting added to the list of targets to compile twice.

[CAD97]

OK, this gets weirder 😅

I set up a dead simple playground with

// src/lib.rs
#[test]
fn test_in_lib() {
    println!("====== test_in_lib =====");
}
// src/main.rs
#[test]
fn test_in_bin() {
    println!("====== test_in_bin =====");
}

fn main() {
    println!("====== main_in_bin =====");
}
// examples/example.rs
#[test]
fn test_in_example() {
    println!("====== test_in_example =====");
}

fn main() {
    println!("====== main_in_example =====");
}

Output:

$ cargo miri test --all-targets
# stdout
running 1 test
test test_in_example ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


running 1 test
test test_in_example ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.37s
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.37s

$ cargo miri run --all-targets
# stdout

[empty]

# stderr
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s

Rename examples/example.rs to examples/example.rs.ignored

$ cargo miri test --all-targets
# stdout
running 1 test

running 1 test
test test_in_lib ... ok

test test_in_bin ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


running 1 test

running 1 test
test test_in_lib ... ok
test test_in_bin ... ok


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.42s
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.33s

$ cargo miri run --all-targets
# stdout
====== main_in_bin =====

running 1 test

running 1 test
test test_in_lib ... ok
test test_in_bin ... ok


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished dev [unoptimized + debuginfo] target(s) in 1.36s

[CAD97]

One last note: here's proof that the example's main really is definitely running: it hits a panic I added to that main. I don't know why the smaller playground example doesn't seem to be running the example main.

[RalfJung]

I'm rather surprised we do anything with examples, but for now I can only say that --all-targets should not be used with cargo-miri... it violates some assumptions that code is otherwise making.^^
I didn't even know cargo test --all-targets was a thing...

[RalfJung]

#1512 is another bug caused by using incorrectly reimplementing cargo's binary selection.

@ehuss do you think there is any way that Miri could ask cargo for which binaries it would run with a given cargo run/cargo test invocation? We cannot currently use cargo run/cargo test, not even with a custom RUSTC/RUSTC_WRAPPER, because then cargo will run the binaries which is something we do not want it to do.

[alecmocatta]

I think cargo build --message-format=json for cargo run and cargo test --no-run --message-format=json for cargo test give you the build artifacts for a particular invocation, which you can filter for binaries. For cargo run you'd also have to match against default-run in Cargo.toml and a --bin argument for disambiguation.

[RalfJung]

Oh, I did not know about cargo test --no-run. That could be very useful indeed.

For cargo run, not having to reimplement --bin and default-run would be good, but that seems much less of a hassle than it is for tests. Note that we currently do cargo check for this, not cargo build, because we want check-only builds... but we can always mess with the build flags later so that's not a big difference. (I doubt there is a check-only cargo test --no-run so if we go for that we'll have to implement this flag patching anyway.)

[RalfJung]

Looks like there's more logic than I was even aware of: #1514, #1516. I think it would be rather silly to reimplement all that, we really should find a way to reuse the code from cargo for that.

[ehuss]

Unfortunately I don't think there is a way to determine what cargo run is going to run without running it.

[RalfJung]

Assuming we can get something based on cargo test --no-run to work, would you be open to cargo run --no-run? It sounds a bit silly but I hope this is an not entirely unreasonable usecase.

[ehuss]

I think that should be OK. An alternate solution is to set the runner like CARGO_TARGET_X86_64_APPLE_DARWIN_RUNNER=true cargo run, but a flag should probably be fine.

[RalfJung]

Oh, that "runner" concept sounds very interesting! The only trouble is we'd have to figure out the "target" argument. We could store the build environment in JSON at build time and then actually run Miri when being invoked as the runner. Also presumably this is used for build scripts as well so I suppose we'd have to detect those?

[bjorn3]

Build scripts are compiled for the host and not the target. The syntax is CARGO_TARGET_<TARGET>_RUNNER. This doesn't apply to host dependencies or when you don't pass --target to cargo.

[RalfJung]

Oh so no runner is used for host things? I see.