Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port perf/benchmark_matrix_ops.jl to benchmark/*.jl #742

Merged
merged 5 commits into from
Feb 26, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions benchmark/Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
PkgBenchmark = "32113eaa-f34f-5b0d-bd6c-c81e245fc73d"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
45 changes: 45 additions & 0 deletions benchmark/bench_matrix_ops.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
module BenchMatrixOps

import Random
using BenchmarkTools
using LinearAlgebra
using StaticArrays

const suite = BenchmarkGroup()
const matrix_sizes = if haskey(ENV, "GITHUB_EVENT_PATH")
(1, 2, 3, 4, 10, 20)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great 👍

If we're already at 15 min, I worry a bit about will happen for a comprehensive benchmark suite 😬

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I guess using the same treatment for QR would probably help with that 15 mins though)

Copy link
Member Author

@tkf tkf Feb 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I have the same worry. I guess we can't do the "full sweep" in CI and be a bit less greedy about the parameters we try.

An option to reduce CI time is to tune benchmarks separately and store the JSON file somewhere. But I think it's easy to get it stale and it's cumbersome to add benchmarks...

BTW, a less ad-hoc way to do this kind of filtering is to use @tagged from BenchmarkToos.jl: https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#indexing-into-a-benchmarkgroup-using-tagged I don't think PkgBenchmark.jl support this though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also wonder how much of this is compile time; some of the algorithms are probably quite terrible for N=20.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, one thing I can't copy easily from perf/*.jl is the compile-time benchmarks. It'd be great if BenchmarkTools reports compile-time while doing the tuning. I don't think it is difficult but just tedious...

else
1:20
end

# Use same arrays across processes (at least with the same Julia version):
Random.seed!(1234)

# Unary operators
for f in [det, inv, exp]
s1 = suite["$f"] = BenchmarkGroup()
for N in matrix_sizes
SA = @SMatrix rand(N, N)
A = Array(SA)
s2 = s1[string(N, pad=2)] = BenchmarkGroup()
s2["SMatrix"] = @benchmarkable $f($SA)
s2["Matrix"] = @benchmarkable $f($A)
end
end

# Binary operators
for f in [*, \]
tkf marked this conversation as resolved.
Show resolved Hide resolved
s1 = suite["$f"] = BenchmarkGroup()
for N in matrix_sizes
SA = @SMatrix rand(N, N)
SB = @SMatrix rand(N, N)
A = Array(SA)
B = Array(SB)
s2 = s1[string(N, pad=2)] = BenchmarkGroup()
s2["SMatrix"] = @benchmarkable $f($SA, $SB)
s2["Matrix"] = @benchmarkable $f($A, $B)
end
end

end # module
BenchMatrixOps.suite