Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] *Display* aggregates only. #664

Closed
LebedevRI opened this issue Aug 29, 2018 · 0 comments · Fixed by #665
Closed

[RFC] *Display* aggregates only. #664

LebedevRI opened this issue Aug 29, 2018 · 0 comments · Fixed by #665

Comments

@LebedevRI
Copy link
Collaborator

There is a flag

DEFINE_bool(benchmark_report_aggregates_only, false,
"Report the result of each benchmark repetitions. When 'true' is "
"specified only the mean, standard deviation, and other statistics "
"are reported for repeated benchmarks.");

and a call
// Specify if each repetition of the benchmark should be reported separately
// or if only the final statistics should be reported. If the benchmark
// is not repeated then the single result is always reported.
Benchmark* ReportAggregatesOnly(bool value = true);

But that affects everything, every reporter, destination:
if (report_aggregates_only) reports.clear();

It would be quite useful to have an ability to be more picky.

More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is great for tooling.

Now the problem. I'm not sure how best to do it.
The most straight-forward solution i can think of is to filter the results here:

benchmark/src/benchmark.cc

Lines 466 to 471 in d9cab61

std::vector<BenchmarkReporter::Run> reports =
RunBenchmark(benchmark, &complexity_reports);
display_reporter->ReportRuns(reports);
if (file_reporter) file_reporter->ReportRuns(reports);
flushStreams(display_reporter);
flushStreams(file_reporter);

So the file reporter will still receive the full std::vector<BenchmarkReporter::Run> reports, while
display_reporter may (depending on the configuration) only receive the aggregates.

Alternative solutions could include modifying the API of BenchmarkReporter,
and every implementation, to skip non-aggregates, depending on some additional config params.

Thoughts?

LebedevRI added a commit that referenced this issue Sep 12, 2018
There is a flag 
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78
and a call
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840
But that affects everything, every reporter, destination:
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316


It would be quite useful to have an ability to be more picky.


More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is **great** for tooling.

Fixes #664
JBakamovic pushed a commit to JBakamovic/benchmark that referenced this issue Dec 6, 2018
There is a flag 
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78
and a call
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840
But that affects everything, every reporter, destination:
https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316


It would be quite useful to have an ability to be more picky.


More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is **great** for tooling.

Fixes google#664
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant