Goal: To accurately measure the impact of changes on the Idris compiler against its exisiting ecosystem of libraries.
Here are all the kinds of benchmarks we want to run:
- benchmarks checking the speed of the compiler at compiling things
- Parsing speed - How fast we parse
- Typechecking speed - How fast we typecheck parsed trees
- Codegen speed - How fast we process typechecked code
- benchmarks checking the speed of the programs generated by the compiler
- With startup
- Without startup
We pull all packages from the latest pack collection, install them
and measure how long it takes to typecheck them. After that, we also
run the tests on each of them to measure the performance of compiled code.
In the current state we only have a coarse view of performance where compiler stages are not separated and startup time is not removed from the measurement.
- Obtain data for each compiler stages during typechecking
- Not sure of the extra logging will degrate performance in a way that makes measurement meaningless
- Measure runtime performance without startup time
- Requires new compiler features
- Run all the benchmarks on a dedicated server in a controlled environement
- Make the benchmarks part of CI
- idris2 self benchmark suite https://github.com/idris-lang/Idris2/tree/main/benchmark
- smallTT benchmarks https://github.com/AndrasKovacs/smalltt
- idris1 benchmarks https://github.com/ziman/idris-benchmarks (would require porting!)