The main goal of the benchmarking framework is to detect regressions during development, but one can run benchmarks at any specific commit just to see how it performs, this can be a very useful tool for Aer developers so they can make sure their changes don't introduce important performance regressions.
Our benchmarking framework is based on Airspeed Velocity. We have only implemented benchmarks for the Qiskit Addon, not the standalone mode.
All the benchmarks are under the test/benchmark
directory.
There you'll find a bunch of *_benchmarks.py
files which represent the different type of benchmarks we will run:
- Quantum Volume with different number of qubits and noise models
- Simple one-gate circuits with different number of qubits and noise models.
All prerequisites for building the project need to be installed in the system, take a look at the CONTRIBUTING guide if you don't have them already installed.
Install Airspeed Velocity (ASV
):
$ pip install asv
Move to the test
directory:
$ cd test
And run asv
using the correct configuration file, depending on what O.S. you are executing them:
Linux:
$ asv run --config asv.linux.conf.json
MacOS:
$ asv run --config asv.macos.conf.json
NOTE: We only support Linux and MacOS at the moment
Depending on your system, benchmarks will take a while to complete. After the completion of the tests, you will see the results with a format similar like this:
· Creating environments
· Discovering benchmarks
· Running 3 total benchmarks (1 commit * 1 environment * 3 benchmarks)
[ 0.00%] · For qiskit-aer commit 8b4f4de1 <master>:
[ 0.00%] ·· Benchmarking conda-py3.7
[ 16.67%] ··· Running (quantum_volume_benchmarks.QuantumVolumeTimeSuite.time_quantum_volume--)..
[ 50.00%] ··· Running (simple_benchmarks.SimpleU3TimeSuite.time_simple_u3--).
[ 66.67%] ··· quantum_volume_benchmarks.QuantumVolumeTimeSuite.time_quantum_volume
[ 66.67%] ··· ================= ========== ===================== ============= =============
-- Noise Model
----------------- ------------------------------------------------------------
Quantum Volume No Noise Mixed Unitary Noise Reset Noise Kraus Noise
================= ========== ===================== ============= =============
Num. qubits: 5 123±4ms 124±10ms 154±20ms 145±20ms
Num. qubits: 10 292±10ms 319±50ms 299±10ms 346±2ms
Num. qubits: 15 3.17±2s 2.38±2s 2.19±2s 9.76±2s
================= ========== ===================== ============= =============
[ 83.33%] ··· simple_benchmarks.SimpleCxTimeSuite.time_simple_cx
[ 83.33%] ··· ====================== ========== ===================== ============= =============
-- Noise Model
---------------------- ------------------------------------------------------------
Simple cnot circuits No Noise Mixed Unitary Noise Reset Noise Kraus Noise
====================== ========== ===================== ============= =============
Num. qubits: 5 47.2±2ms 111±40ms 46.8±1ms 21.1±0.2ms
Num. qubits: 10 117±60ms 172±60ms 164±30ms 28.6±0.3ms
Num. qubits: 15 213±30ms 223±80ms 223±90ms 258±5ms
====================== ========== ===================== ============= =============
[100.00%] ··· simple_benchmarks.SimpleU3TimeSuite.time_simple_u3
[100.00%] ··· ==================== ============ ===================== ============= =============
-- Noise Model
-------------------- --------------------------------------------------------------
Simple u3 circuits No Noise Mixed Unitary Noise Reset Noise Kraus Noise
==================== ============ ===================== ============= =============
Num. qubits: 5 20.3±0.3ms 21.3±0.5ms 20.7±0.4ms 21.5±0.1ms
Num. qubits: 10 26.6±0.2ms 27.8±0.3ms 27.6±0.5ms 28.4±0.2ms
Num. qubits: 15 244±40ms 312±70ms 295±100ms 179±100ms
==================== ============ ===================== ============= =============
The output format is pretty self-explanatory, so every row starting with the text: Num. quits:
represents all the benchmarks run for this number of qubits configuration, more precisely, we run 4 benchmarks for every number of qubits configuration, and each of the benchmarks are run with a different noise model, so for example, this line:
Quantum Volume No Noise Mixed Unitary Noise Reset Noise Kraus Noise
================= ========== ===================== ============= =============
Num. qubits: 15 3.17±2s 2.38±2s 2.19±2s 9.76±2s
it's telling us that for our Quantum Volume circuit the time it took to complete was:
- 3.17 seconds with no noise at all
- 2.38 seconds with Mixed unitary noise
- 2.19 seconds with Reset noise
- 9.76 seconds with Kraus noise