Option to disable "Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 89.5s."

[kevincox]

There is a chance that I am using it wrong but after reading the manual and API docs I can't find a good option to get rid of the warning.

The problem is that I have slow things that I want to benchmark. The tests are on the order of 10s. I could potentially use smaller examples (and I do have some smaller ones in other benchmarks) but I am quite concerned with how long the complex problems take so find it important to have benchmarks for these. Really what I want to do is run 5-10 samples and then use criterion's comparison features to identify changes. Right now my benchmark looks like this:

fn slow(c: &mut criterion::Criterion) {
	let mut group = c.benchmark_group("slow");
	group.sample_size(10);
	group.sampling_mode(criterion::SamplingMode::Flat);

	for path in [...] {
		let data = ...
		group.bench_with_input(
			criterion::BenchmarkId::from_parameter(path),
			&game,
			|b, game| b.iter(|| analyze(data));
	}
}

It seems that the suggestion wants me to tell criterion how long I think my benchmark will take. But I really don't want to bother keeping a runtime estimate up to date. Even worse since I have multiple inputs the duration can be significantly different for each. It seems that I either set it too low and get an annoying warning in my output or I set it too high and further slow down an already slow benchmark.

I understand that with a small number of samples the confidence won't be particularly high but it seems like the best option available without having the benchmark run for an hour. So I just want to turn off the warning to make reading the output easier.

[adsick]

I have the same problem, except it does not complete in reasonable time, the code:

use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion};
use rand::prelude::*;

pub fn lookup(c: &mut Criterion) {
    let mut group = c.benchmark_group("lookup");

    for size in (0..4*1024*1024).step_by(1024*1024) {
        let mut rng = thread_rng();
        let mut data: Vec<u64> = vec![0; size];
        rng.fill(&mut data[..]);

        let mut keys = data.clone();
        keys.shuffle(&mut rng);

        group.bench_with_input(BenchmarkId::new("linear", size), &size, |b, size| {
            b.iter(|| {
                for key in &keys {
                    black_box(data.contains(key));
                }
            });
        });

        let mut rng = thread_rng();
        let mut data: Vec<u64> = vec![0; size];
        rng.fill(&mut data[..]);

        let mut keys = data.clone();
        keys.shuffle(&mut rng);

        group.bench_with_input(BenchmarkId::new("constant", size), &size, |b, size| {
            b.iter(|| {
                for key in &keys {
                    black_box(data.contains(key));
                }
            });
        });
    }
    group.finish();
}

criterion_group!(benches, lookup);
criterion_main!(benches);

I tried to reduce number of samples and confidence level like this: group.confidence_level(0.6).sample_size(12); but it didn't really help.

[adsick]

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 15150.2s, or reduce sample count to 10.
Benchmarking lookup/linear/1048576: Collecting 100 samples in estimated  15150 s (100 iterations)

[johannesloetzsch]

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 30.0s

I had the same issue, this related PR helped me: #806

group.measurement_time(Duration::from_secs(30));