Add a Regularly Updated Collection of Benchmark Results

[ericsnowcurrently]

A public collection of benchmark results would be valuable for the faster-cpython project, as well as the Python community as a whole. The results would need to be produced regularly and uploaded to a publicly accessible location. This would be independent of presentation (e.g. codespeed, which is used by speed.python.org), though some level of interoperability between the collection and codespeed would be nice.

My plan is to first run the benchmarks manually and upload them to this repo. After that I'll look into setting up a cron job and then maybe some further tooling to make life easier. We'll also work on expanding our platform/build matrix.

Key Requirements

• a public location where benchmark results will be stored
• a stable benchmarking host for each platform of interest
• a mechanism by which the benchmarks are run regularly and the results uploaded
• a metadata schema to identify the conditions of each run (pyperf provides some in the results data, which should be enough)

Plan

phase 1: manual runs/uploads

Run benchmarks manually and upload the results. Automation will likely happen well before all the below manual runs have been done.

Only our dedicated linux benchmarking host will be used.

Releases

• 3.8.13 (latest)
• 3.9.12 (latest)
• 3.10.4 (latest)
• 3.11a1
• 3.11a2
• 3.11a3
• 3.11a4
• 3.11a5
• 3.11a6
• 3.11a7
• 3.11b1
• 3.11b2
• 3.11b3
• 3.11b4 (Jun 16)
• 3.11b5 (Jul 9)
• 3.11rc1 (Aug 1)
• 3.11rc2 (Sep 5)
• 3.11 final (Oct 3)

(Also, all future 3.8, 3.9, and 3.10 bug fix releases.)

phase 2: automation

• set up a cron job to do the tasks from phase 1 (may require some rudimentary queueing)
  • Set up a cron job to run benchmarks #386

phase 3

• add other benchmarking hosts (e.g. Windows, Windows ARM, MacOS)
• (maybe) add other build configurations (e.g. clang instead of gcc, debug, without PGO)
• investigate extra tooling

Uploads

location: https://github.com/faster-cpython/ideas/tree/main/benchmark-results/

For now, we will only upload the json file produced by pyperf. The filename will be unique and meaningful to humans:

<impl>-<release>-<commit>-<host>-<compat ID>.json

where <compat ID> is a hash derived from the metadata found in the json file .

Examples:

• example: /benchmark-results/cpython-main-ada6B4cf8d-linux-cc12888f9B4b.json
• example: /benchmark-results/cpython-3.11a5-838b51a079-linux-cc12888f9B4b.json
• example: /benchmark-results/cpython-3.10.2-3571d41577-linux-cc12888f9B4b.json

Open Questions

• how often to benchmark main? (daily? weekly? with every commit?)
• which platforms should be benchmarked?
• should we benchmark earlier releases (e.g. 3.8)? If so, how many releases back?
• what about other build tool chains (e.g. clang instead of gcc)?
• should we add results for other Python implementations?
• do we need more metadata than what pyperf already provides?

[ericsnowcurrently]

Possible Future

tools

• to run the benchmarks and upload the results
• to download from the collection and compare results
• to automate the regular benchmark runs (including cron jobs)
• to add a benchmarking request to a queue (see https://github.com/faster-cpython/tools, especially issues #7 and #8)
• to set up a benchmarking host (or manage a set of them)
• to push results from the collection to speed.python.org (or any codespeed deployment)
• to identify how compatible two benchmark results are

other

• use results from less stable hosts, e.g. VMs, containers, the cloud, personal laptops? (see https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/)
• store results in a sqlite database (in addition to flat files in a repo)?
• use buildbots to run benchmarks?
• a way to request a benchmark run for a given PR or branch (a bot?)
• a GH action that checks a PR for performance regressions (comparing against the collection)
• the whole system could be deployed for other Python community projects

[gvanrossum]

Thanks for writing down a clear plan! I just have one request -- can we make the data available both as JSON and in markdown table form? The latter is easier to review, the former is handy if we want to apply additional tools to it (even pyperf compare_to).

[ericsnowcurrently]

@mdboom, this is likely informative. Most notably, the work to support cron jobs for benchmarking won't be substantial but isn't non-trivial.