Seemlessly comparing benchmark results across versions

[ahwillia]

In an ideal world, I would love the following to work:

versions = ["v0.0.1", "3743c273f4439833787522242efdcda87951f6d1", "v0.0.2"]
results = Pkg.benchmark("Foo",versions)
@show results # nice printing

@jrevels has pointed out that it is probably not feasible to have Pkg support this functionality in the short-term (see discussion here). But it seems like BenchmarkTools could support this natively, perhaps as a prototype to be taken up by Pkg at a later time.

Let's say we call this function benchmark_pkg(), then my proposal is to support the following:

benchmark_pkg("Foo",:major) # runs benchmarks on all major releases
benchmark_pkg("Foo",:minor) # runs benchmarks on all major & minor releases
benchmark_pkg("Foo",:patch) # runs benchmarks on all releases

# run benchmarks for each version/commit stored in a user-provided list
benchmark_pkg("Foo",::Vector{AbstractString})

How can I help take baby steps towards this vision? My thought is along these lines:

• User defines benchmarks in perf/runtests.jl. (Maybe perf/runbenchmarks.jl would be a better name?)
• pkg_benchmark() finds parameters using tune!, saves them in perf/benchmark_params.jl
• Save the original/starting commit
• For each version specified by the user:
  • Use git checkout ..., to selectively checkout the src/ repository.
  • Run the benchmarks
• After all benchmarks are run, return src/ to its original/starting commit

Is this reasonable? Or more dangerous than I realize? For example, what if user edits code while benchmarks are running and saves the result...

[jrevels]

Seems like this is now being addressed in https://github.com/JuliaCI/PkgBenchmark.jl