Display aggregates only. #665

LebedevRI · 2018-09-02T17:31:40Z

There is a flag

Lines 75 to 78 in d9cab61

    
           DEFINE_bool(benchmark_report_aggregates_only, false, 
        
                       "Report the result of each benchmark repetitions. When 'true' is " 
        
                       "specified only the mean, standard deviation, and other statistics " 
        
                       "are reported for repeated benchmarks.");

and a call

benchmark/include/benchmark/benchmark.h

Lines 837 to 840 in d9cab61

    
           // Specify if each repetition of the benchmark should be reported separately 
        
           // or if only the final statistics should be reported. If the benchmark 
        
           // is not repeated then the single result is always reported. 
        
           Benchmark* ReportAggregatesOnly(bool value = true);

But that affects everything, every reporter, destination:

benchmark/src/benchmark.cc

Line 316 in d9cab61

if (report_aggregates_only) reports.clear();

It would be quite useful to have an ability to be more picky.

More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is great for tooling.

This does work, and is even less intrusive than i thought it will be.
But there is a slight test coverage issue :)
These tests are all testing the first, display, reporter.
So we don't actually check what happens with the second, file, reporter...
This problem isn't a new one, but here i would say it is more severe.
I'll see if i can easily fix it for this particular case..

Fixes #664

coveralls · 2018-09-02T17:47:41Z

Coverage increased (+0.7%) to 88.351% when pulling 4f05864 on LebedevRI:display-aggregates-only into f090141 on google:master.

AppVeyorBot · 2018-09-02T18:28:04Z

❌ Build benchmark 1402 failed (commit 82c280381b by @LebedevRI)

LebedevRI · 2018-09-02T19:51:18Z

After digging a bit more, this is the problematic bit:

benchmark/src/benchmark.cc

Lines 566 to 567 in 5159967

    
           file_reporter->SetOutputStream(&output_file); 
        
           file_reporter->SetErrorStream(&output_file);

If we wanted to test that, we'd need to add some internal debug option to
disable this destination rebinding. Reading back from file seems rather fragile..

dmah42 · 2018-09-03T11:48:16Z

include/benchmark/benchmark.h

-  ARM_ReportAggregatesOnly };
+{ ARM_Unspecified,           // The mode has not been manually specified
+  ARM_Default,               // The mode is user-specified as default.
+  ARM_ReportAggregatesOnly,  // All reporters only display aggregates.


does the PR get even simpler if these are renamed to:

ARM_FileReportAggregatesOnly,
ARM_DisplayReportAggregatesOnly,

and then the existing flag sets them both to true, and the new flag sets only the Display one to true?

Hmm, kinda.
It gets a bit messier due to the command-line flags.

dmah42 · 2018-09-03T11:48:58Z

src/benchmark.cc

-std::vector<BenchmarkReporter::Run> RunBenchmark(
-    const benchmark::internal::Benchmark::Instance& b,
-    std::vector<BenchmarkReporter::Run>* complexity_reports) {
+std::pair<std::vector<BenchmarkReporter::Run> /*everything*/,


is it worth returning a custom struct to avoid having the comments tracking which member of the pair is which?

Makes sense.

dmah42 · 2018-09-03T11:50:24Z

src/benchmark.cc

+                                 display_aggregates_only);
+
+      const std::vector<BenchmarkReporter::Run>& report_for_display =
+          display_aggregates_only ? reports.second /*aggregates_only*/


would this also be simpler if the return type was 'non-aggregate', 'aggregate' instead of having the first be everything?

Hmm, actually it might be.
I didn't do so initially because i thought it was important to print the whole benchmark at once
(for header/context pre-print), but maybe it is too pessimistic..

AppVeyorBot · 2018-09-03T14:34:20Z

❌ Build benchmark 1403 failed (commit 658116960e by @LebedevRI)

src/benchmark.cc

dmah42 · 2018-09-03T18:57:48Z

src/benchmark.cc

-      (b.aggregation_report_mode == internal::ARM_Unspecified
-           ? FLAGS_benchmark_report_aggregates_only
-           : b.aggregation_report_mode == internal::ARM_ReportAggregatesOnly);
+  run_results.display_report_aggregates_only =


why do you need to return this? you should be able to (as it was) have a local constant and decide to add to the vector or not based on the value.

Then i don't understand the previous suggestion, #665 (comment).

But to attempt to answer, display_report_aggregates_only is needed,
because i need to somehow convey back which result (all the reports, or aggregates only)
to report, for both of the reporters. I need to do so because i actually look at the
benchmark::internal::Benchmark::Instance::aggregation_report_mode,
which is per-instance, not just FLAGS_benchmark_report_aggregates_only /
FLAGS_benchmark_display_aggregates_only, which is global.

Ah right, sorry.

… struct.

AppVeyorBot · 2018-09-05T18:37:01Z

❌ Build benchmark 1413 failed (commit ebf7a6f2ad by @LebedevRI)

LebedevRI · 2018-09-05T19:05:04Z

Any further thoughts here? :)
The testing issue is unresolved, but i'm not seeing any clear/clean solution for it..

dmah42 · 2018-09-07T10:06:47Z

I have nothing further. I've been unable to come up with a satisfactory testing approach without radical replumbing of the file reporter.

LebedevRI · 2018-09-07T10:14:32Z

Makes sense..
Would it be unacceptably ugly to simply read back that file and check that?
I guess the troubling part is coming up with the tmp filename.

dmah42 · 2018-09-07T11:14:50Z

#include <cstdio>
#include <filesystem>

std::filesystem::read_symlink(
    std::filesystem::path("/proc/self/fd") / std::to_string(fileno(tmpfile()))
);

would at least work on linux :P

LebedevRI · 2018-09-08T11:00:52Z

Hmm, actually now that i think about it std::tmpnam() might work, too,
and be more cross-platform. (ignoring the fact that std::filesystem is C++17)

LebedevRI · 2018-09-08T11:54:20Z

This is so horrible ^^

AppVeyorBot · 2018-09-08T12:51:17Z

❌ Build benchmark 1415 failed (commit 3580387aea by @LebedevRI)

… me, but an error in CI?)

AppVeyorBot · 2018-09-08T13:49:28Z

❌ Build benchmark 1416 failed (commit b92ecfb4ff by @LebedevRI)

AppVeyorBot · 2018-09-08T14:46:32Z

❌ Build benchmark 1417 failed (commit 45bfd09832 by @LebedevRI)

README.md

dmah42 · 2018-09-12T10:26:02Z

test/display_aggregates_only_test.cc

+
+  // Read the output back from the file, and delete the file.
+  std::ifstream tmp_stream(tmp_file_name);
+  std::string JsonOutput((std::istreambuf_iterator<char>(tmp_stream)),


there's extra parentheses here that i don't think are necessary.

Yes, they are necessary.
https://godbolt.org/z/V6ASzJ

gah, vexing parse.

nit: s/JsonOutput/json_output/

test/report_aggregates_only_test.cc

LebedevRI · 2018-09-12T11:00:30Z

@dominichamon Thank you for taking a look!

AppVeyorBot · 2018-09-12T11:31:45Z

❌ Build benchmark 1426 failed (commit 38bc928fb7 by @LebedevRI)

test/file_reporter_test_helper.h

test/output_test.h

AppVeyorBot · 2018-09-12T12:45:40Z

❌ Build benchmark 1427 failed (commit e87c71478b by @LebedevRI)

LebedevRI · 2018-09-12T12:52:41Z

@dominichamon thank you for the review! I will merge this soon-ish.

There is a flag https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78 and a call https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840 But that affects everything, every reporter, destination: https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316 It would be quite useful to have an ability to be more picky. More specifically, i would like to be able to only see the aggregates in the on-screen output, but for the file output to still contain everything. The former is useful in case of a lot of repetition (or even more so if every iteration is reported separately), while the former is **great** for tooling. Fixes google#664

googlebot added the cla: yes label Sep 2, 2018

dmah42 reviewed Sep 3, 2018

View reviewed changes

LebedevRI added 3 commits September 5, 2018 20:27

*Display* aggregates only.

d029cc5

Make AggregationReportMode a sudo-bitfield, and RunBenchmark() return…

966f0e0

… struct.

Cleanup ternaries

8720574

LebedevRI force-pushed the display-aggregates-only branch from 0a62f7b to 8720574 Compare September 5, 2018 17:27

Add a test for the actual [file] contents of the file reporter

fa6e342

Be more specific about the patters we want to count

d3dbd19

Update docs, and fix compiler warnings (why are they not an error for…

77a285c

… me, but an error in CI?)

dmah42 reviewed Sep 12, 2018

View reviewed changes

README.md Outdated Show resolved Hide resolved

80-column wrap in readme.md

5bf8345

dmah42 reviewed Sep 12, 2018

View reviewed changes

test/report_aggregates_only_test.cc Outdated Show resolved Hide resolved

Share SubstrCnt() in tests

c72a15d

Apply some RAII abstraction goodness

0985914

dmah42 reviewed Sep 12, 2018

View reviewed changes

test/file_reporter_test_helper.h Outdated Show resolved Hide resolved

And make it a simple function

44d5f7a

dmah42 reviewed Sep 12, 2018

View reviewed changes

test/output_test.h Outdated Show resolved Hide resolved

dmah42 approved these changes Sep 12, 2018

View reviewed changes

Be less inline happy

4f05864

LebedevRI changed the title ~~[RFC] *Display* aggregates only.~~ *Display* aggregates only. Sep 12, 2018

LebedevRI merged commit c614dfc into google:master Sep 12, 2018

LebedevRI deleted the display-aggregates-only branch September 12, 2018 13:26

LebedevRI mentioned this pull request Sep 12, 2018

[Tooling] 'display aggregates only' support. #674

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Display aggregates only. #665

Display aggregates only. #665

LebedevRI commented Sep 2, 2018

coveralls commented Sep 2, 2018 •

edited

Loading

AppVeyorBot commented Sep 2, 2018

LebedevRI commented Sep 2, 2018

dmah42 Sep 3, 2018

LebedevRI Sep 3, 2018

dmah42 Sep 3, 2018

LebedevRI Sep 3, 2018

dmah42 Sep 3, 2018

LebedevRI Sep 3, 2018

AppVeyorBot commented Sep 3, 2018

dmah42 Sep 3, 2018

LebedevRI Sep 3, 2018

LebedevRI Sep 3, 2018

dmah42 Sep 3, 2018

AppVeyorBot commented Sep 5, 2018

LebedevRI commented Sep 5, 2018

dmah42 commented Sep 7, 2018

LebedevRI commented Sep 7, 2018

dmah42 commented Sep 7, 2018

LebedevRI commented Sep 8, 2018

LebedevRI commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

dmah42 Sep 12, 2018

LebedevRI Sep 12, 2018

dmah42 Sep 12, 2018

LebedevRI commented Sep 12, 2018

AppVeyorBot commented Sep 12, 2018

AppVeyorBot commented Sep 12, 2018

LebedevRI commented Sep 12, 2018

	DEFINE_bool(benchmark_report_aggregates_only, false,
	"Report the result of each benchmark repetitions. When 'true' is "
	"specified only the mean, standard deviation, and other statistics "
	"are reported for repeated benchmarks.");

	// Specify if each repetition of the benchmark should be reported separately
	// or if only the final statistics should be reported. If the benchmark
	// is not repeated then the single result is always reported.
	Benchmark* ReportAggregatesOnly(bool value = true);

*Display* aggregates only. #665

*Display* aggregates only. #665

Conversation

LebedevRI commented Sep 2, 2018

coveralls commented Sep 2, 2018 • edited Loading

AppVeyorBot commented Sep 2, 2018

LebedevRI commented Sep 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AppVeyorBot commented Sep 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AppVeyorBot commented Sep 5, 2018

LebedevRI commented Sep 5, 2018

dmah42 commented Sep 7, 2018

LebedevRI commented Sep 7, 2018

dmah42 commented Sep 7, 2018

LebedevRI commented Sep 8, 2018

LebedevRI commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

AppVeyorBot commented Sep 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LebedevRI commented Sep 12, 2018

AppVeyorBot commented Sep 12, 2018

AppVeyorBot commented Sep 12, 2018

LebedevRI commented Sep 12, 2018

Display aggregates only. #665

Display aggregates only. #665

coveralls commented Sep 2, 2018 •

edited

Loading