-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
*Display* aggregates only. #665
Conversation
❌ Build benchmark 1402 failed (commit 82c280381b by @LebedevRI) |
After digging a bit more, this is the problematic bit: Lines 566 to 567 in 5159967
If we wanted to test that, we'd need to add some internal debug option to disable this destination rebinding. Reading back from file seems rather fragile.. |
include/benchmark/benchmark.h
Outdated
ARM_ReportAggregatesOnly }; | ||
{ ARM_Unspecified, // The mode has not been manually specified | ||
ARM_Default, // The mode is user-specified as default. | ||
ARM_ReportAggregatesOnly, // All reporters only display aggregates. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does the PR get even simpler if these are renamed to:
ARM_FileReportAggregatesOnly,
ARM_DisplayReportAggregatesOnly,
and then the existing flag sets them both to true, and the new flag sets only the Display one to true?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, kinda.
It gets a bit messier due to the command-line flags.
src/benchmark.cc
Outdated
std::vector<BenchmarkReporter::Run> RunBenchmark( | ||
const benchmark::internal::Benchmark::Instance& b, | ||
std::vector<BenchmarkReporter::Run>* complexity_reports) { | ||
std::pair<std::vector<BenchmarkReporter::Run> /*everything*/, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it worth returning a custom struct to avoid having the comments tracking which member of the pair is which?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
src/benchmark.cc
Outdated
display_aggregates_only); | ||
|
||
const std::vector<BenchmarkReporter::Run>& report_for_display = | ||
display_aggregates_only ? reports.second /*aggregates_only*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would this also be simpler if the return type was 'non-aggregate', 'aggregate' instead of having the first be everything?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, actually it might be.
I didn't do so initially because i thought it was important to print the whole benchmark at once
(for header/context pre-print), but maybe it is too pessimistic..
❌ Build benchmark 1403 failed (commit 658116960e by @LebedevRI) |
src/benchmark.cc
Outdated
(b.aggregation_report_mode == internal::ARM_Unspecified | ||
? FLAGS_benchmark_report_aggregates_only | ||
: b.aggregation_report_mode == internal::ARM_ReportAggregatesOnly); | ||
run_results.display_report_aggregates_only = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do you need to return this? you should be able to (as it was) have a local constant and decide to add to the vector or not based on the value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then i don't understand the previous suggestion, #665 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But to attempt to answer, display_report_aggregates_only
is needed,
because i need to somehow convey back which result (all the reports, or aggregates only)
to report, for both of the reporters. I need to do so because i actually look at the
benchmark::internal::Benchmark::Instance::aggregation_report_mode
,
which is per-instance, not just FLAGS_benchmark_report_aggregates_only
/
FLAGS_benchmark_display_aggregates_only
, which is global.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah right, sorry.
0a62f7b
to
8720574
Compare
❌ Build benchmark 1413 failed (commit ebf7a6f2ad by @LebedevRI) |
Any further thoughts here? :) |
I have nothing further. I've been unable to come up with a satisfactory testing approach without radical replumbing of the file reporter. |
Makes sense.. |
#include <cstdio>
#include <filesystem>
std::filesystem::read_symlink(
std::filesystem::path("/proc/self/fd") / std::to_string(fileno(tmpfile()))
); would at least work on linux :P |
Hmm, actually now that i think about it |
This is so horrible ^^ |
❌ Build benchmark 1415 failed (commit 3580387aea by @LebedevRI) |
… me, but an error in CI?)
❌ Build benchmark 1416 failed (commit b92ecfb4ff by @LebedevRI) |
❌ Build benchmark 1417 failed (commit 45bfd09832 by @LebedevRI) |
test/display_aggregates_only_test.cc
Outdated
|
||
// Read the output back from the file, and delete the file. | ||
std::ifstream tmp_stream(tmp_file_name); | ||
std::string JsonOutput((std::istreambuf_iterator<char>(tmp_stream)), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there's extra parentheses here that i don't think are necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, they are necessary.
https://godbolt.org/z/V6ASzJ
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gah, vexing parse.
nit: s/JsonOutput/json_output/
@dominichamon Thank you for taking a look! |
❌ Build benchmark 1426 failed (commit 38bc928fb7 by @LebedevRI) |
❌ Build benchmark 1427 failed (commit e87c71478b by @LebedevRI) |
@dominichamon thank you for the review! I will merge this soon-ish. |
There is a flag https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78 and a call https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840 But that affects everything, every reporter, destination: https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316 It would be quite useful to have an ability to be more picky. More specifically, i would like to be able to only see the aggregates in the on-screen output, but for the file output to still contain everything. The former is useful in case of a lot of repetition (or even more so if every iteration is reported separately), while the former is **great** for tooling. Fixes google#664
There is a flag
benchmark/src/benchmark.cc
Lines 75 to 78 in d9cab61
and a call
benchmark/include/benchmark/benchmark.h
Lines 837 to 840 in d9cab61
But that affects everything, every reporter, destination:
benchmark/src/benchmark.cc
Line 316 in d9cab61
It would be quite useful to have an ability to be more picky.
More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is great for tooling.
This does work, and is even less intrusive than i thought it will be.
But there is a slight test coverage issue :)
These tests are all testing the first, display, reporter.
So we don't actually check what happens with the second, file, reporter...
This problem isn't a new one, but here i would say it is more severe.
I'll see if i can easily fix it for this particular case..
Fixes #664