Do not invoke the test runner when calling cargo bench and harness=false

[lu-zero]

Benchers such as criterion have custom options that are not available to the test runner.

[alexcrichton]

I'm not sure I understand the context for this issue, can you elaboate? All Cargo does is execute a binary, so are you saying Cargo shouldn't execute a binary?

[lu-zero]

Assume you do

# cargo bench -- -s mybaseline

if there are tests and benchmarks in the crate cargo will run the test runner and the bench runner

some example (current rav1e):

cargo bench -- -s newbase
    Finished release [optimized] target(s) in 0.05s
     Running target/release/deps/rav1e-d400d42c32908d29
error: Unrecognized option: 's'
error: bench failed

cargo bench -- --help
    Finished release [optimized] target(s) in 0.05s
     Running target/release/deps/rav1e-d400d42c32908d29
Usage: --help [OPTIONS] [FILTER]

Options:
        --ignored       Run ignored tests
        --test          Run tests and not benchmarks
        --bench         Run benchmarks instead of tests
<snip>

     Running target/release/deps/rav1e-34abbf14cafc2f7c
Usage: --help [OPTIONS] [FILTER]

Options:
        --ignored       Run ignored tests

<snip>

     Running target/release/deps/bench-34ea4bfc7d854e31
Criterion Benchmark

USAGE:
    bench-34ea4bfc7d854e31 [FLAGS] [OPTIONS] [FILTER]

<snip>

[alexcrichton]

@lu-zero can you provide a standalone example? harness = false works locally for me.

[lu-zero]

Sure, here the minimal testcase

[alexcrichton]

@lu-zero I think you're seeing the benchmark for src/lib.rs, you'll want to say bench = false in the [lib] section.

[lu-zero]

I had to sprinkle it around all the [[bin]] sections as well, but did the trick :)

Now, could we make this the default behavior ?

[dwijnand]

What's the default behaviour change? That harness=false under [[bench]] makes cargo bench not run lib and bins benchmarks? If I understood correctly, that doesn't sound right. So maybe I didn't understand correctly. 🙂

[lu-zero]

It is right.

If you are using criterion, you would not expect to have another runner being used by default.

[epage]

It is reasonable to have a mixture of different harnesses, built-in and otherwise. The main issue here is more that there isn't a stable bench harness.

I feel like solving this issue is not aligned with longer term goals (stable bench harness, separating the concerns of harnesses and runners, etc). I'm going to close this. If there is a reason we should re-evaluate this, let us know!

Note that #6945 would help lower the boilerplate involved in this issue until things have improved otherwise.

[lu-zero]

in order to support mix&match we'd need something more complex to pass arguments to the different runners, as long that's available the workaround default behavior suggested stops being needed.