Investigate switch from Hyperfine to Bencher

[jcoo092]

The initial plan with this research was to use Hyperfine to measure the comparative timings of the different programs. It is attractive in that it is:

• language-agnostic
• relatively simple-to-use
• includes useful features out-of-the-box such as some basic statistical profiling and parameterisation for benchmarks

It has some downsides, however, in that it:

• doesn't seem to have any sort of memory usage profiling capabilities right now
• only permits parameterisation over one parameter, whereas for some of the tests at least three different parameters are currently expected to be required, on top of the change in the language used, meaning that everything else must be done in a Makefile or some other external controller
• seems to have a problem with naming output files in the fashion that I want (although, that may be due to some error made by me in the Makefile)

The problem is likely due as much as anything to the fact that Hyperfine probably wasn't really made with this particular purpose in mind. On the other hand, the Bencher system from The Computer Language Benchmarks Game is explicitly designed to compare different implementations of the same program in different languages. Moreover, it doesn't seem to suffer from any of the limitations described above for Hyperfine. Thus, it probably is actually the better choice for this work.

At some point before performing final measurements, Bencher should be investigated to see whether it is a better idea. It looks like it probably is more difficult to use, but overall much more suited towards the task at hand. Whichever option wins out between those two should be used from that point.

[sharkdp]

Developer of hyperfine here. I just found this by chance.

Let us know (on the hyperfine issue tracker), if there is anything to help you better support your use case.

It is attractive in that it is:

• language-agnostic
• relatively simple-to-use

This is one of the design goals. Do you think that there is anything that would make it even simpler to use?

• includes useful features out-of-the-box such as some basic statistical profiling and parameterisation for benchmarks

The default output of hyperfine shows some basic statistical properties. Under the hood, hyperfine performs a few more advanced tests for outliers. Finally, you can use the JSON export and the provided advanced_statistics.py script to analyze the results even further.

It has some downsides, however, in that it:

• doesn't seem to have any sort of memory usage profiling capabilities right now

true. please see sharkdp/hyperfine#86

• only permits parameterisation over one parameter, whereas for some of the tests at least three different parameters are currently expected to be required, on top of the change in the language used, meaning that everything else must be done in a Makefile or some other external controller

see sharkdp/hyperfine#253

• seems to have a problem with naming output files in the fashion that I want (although, that may be due to some error made by me in the Makefile)

I'd love to know more.

[jcoo092]

Hi @sharkdp

First of all, thanks for creating and maintaining Hyperfine! :) It is an excellent tool. To address your points:

Let us know (on the hyperfine issue tracker), if there is anything to help you better support your use case.

When I get the chance, I could indeed raise an issue on the Hyperfine repo explaining my use case if you would like. I suspect that what I want to do is probably actually out-of-scope for what you want to achieve with Hyperfine, though. It is essentially the same idea as what Bencher at the Benchmarks Game is intended to do though, i.e. compare the performance of implementations of the same program(s) in different languages.

This is one of the design goals. Do you think that there is anything that would make it even simpler to use?

No, I can't think of anything else that you could do to simplify usage (with the exception of introducing 'matrix measurements' - see below). Which means that you have already done a good job of making it easy-to-use.

The default output of hyperfine shows some basic statistical properties... you can use the JSON export and the provided advanced_statistics.py script to analyze the results even further.

I had somehow totally missed that (or forgotten about it). Thanks!

See (issues)

Yip, I think it's reasonable to exclude memory analysis. I think you could use perf counters on Linux, and the equivalent on Windows (the name escapes me momentarily), but that would probably be a HUGE amount of work, and I have no idea how to implement it myself.

I had already seen the issue about matrix measurements - I have to admit I was a bit surprised to realise it wasn't already supported. BenchmarkDotNet, where I have used the same functionality in the past, has probably spoiled me. I imagine, however, that BenchmarkDotNet has probably had an order-of-magnitude more contributors and person-hours sunk into it (and has the .NET VM to rely on too). I expect to be performing measurements sooner than the likely introduction of this feature though, and unfortunately, I don't have time to work on this it myself.

I'd love to know more.

For the naming of the output files, please see the use of the benches_record_name variable in the Makefile under the Racket directory in this repo. Essentially, for selecttime, spawn, and montecarlopi programs, I have two variables to measure; and for the linalg and whispers programs, I have three variables to measure. To account for the fact that Hyperfine currently only varies on one of them internally, I use one or two for loops in the shell to vary the other variables by hand and include the value of those variables in the benches_record_name This works well for the two-variable programs, but for some reason with the three-variable programs, one of the variables isn't included in the file name, meaning that as each test completes, it merely overwrites the pre-existing file for the previous test. I would think that this is probably actually an error on my part in writing the Makefile since I would have thought that Hyperfine would just use whatever string is provided to it, but as of yet, I haven't figured out the source of the error one way or another.

[sharkdp]

Thank you for the feedback!

When I get the chance, I could indeed raise an issue on the Hyperfine repo explaining my use case if you would like. I suspect that what I want to do is probably actually out-of-scope for what you want to achieve with Hyperfine, though.

That's certainly possible, yes.

I had already seen the issue about matrix measurements - I have to admit I was a bit surprised to realise it wasn't already supported. BenchmarkDotNet, where I have used the same functionality in the past, has probably spoiled me. I imagine, however, that BenchmarkDotNet has probably had an order-of-magnitude more contributors and person-hours sunk into it (and has the .NET VM to rely on too). I expect to be performing measurements sooner than the likely introduction of this feature though, and unfortunately, I don't have time to work on this it myself.

Haven't heard of this tool. I'll take a look - thank you for the reference.

but for some reason with the three-variable programs, one of the variables isn't included in the file name, meaning that as each test completes, it merely overwrites the pre-existing file for the previous test. I would think that this is probably actually an error on my part in writing the Makefile since I would have thought that Hyperfine would just use whatever string is provided to it

I'd be really surprised if that would be a bug in hyperfine, to be honest. Maybe run echo hyperfine … instead of hyperfine in the Makefile to see what is actually called? Or make VERBOSE=1 (?). Or use strace -f make?

[jcoo092]

Hi @sharkdp

I have some good (though not unexpected) news! It turns out that you were entirely correct that Hyperfine was working correctly with regards to the output filenames - and I only finally fixed it after you suggested echoing the command.

It turns out that I needed to change my way of declaring the output file's name from e.g., bench_linalg : benches_record_name = $(benches_dir)/$@_$$j_$$i to bench_linalg : benches_record_name = $(benches_dir)/$@\_$$j\_$$i. For reasons I'm still not clear on, not using the backslashes meant that the first variable wouldn't appear in the final string passed to Hyperfine. Adding the slashes in means that the whole thing works as desired and expected.

Clearly, it's a shell scripting issue, and I suspect might be specific to Zsh, which I am using.

On a completely separate note, if you're interested in learning about BenchmarkDotNet, one of the lead contributors to it wrote a whole book about it. The book focuses on BDN, but I'm under the impression that quite a lot of it is much more broadly applicable.

Thanks again for Hyperfine and your helpful comments here! :)

[sharkdp]

Adding the slashes in means that the whole thing works as desired and expected.

👍

On a completely separate note, if you're interested in learning about BenchmarkDotNet, one of the lead contributors to it wrote a whole book about it. The book focuses on BDN, but I'm under the impression that quite a lot of it is much more broadly applicabl

Thank you for the reference!