Improve benchmarking

BENCHMARKING.md

@@ -0,0 +1,97 @@
+The main goal of the benchmarking framework is to detect regressions during development, but one can run benchmarks at any specific commit just to see how it performs, this can be a very useful tool for Aer developers so they can make sure their changes don't introduce important performance regressions.
+
+Our benchmarking framework is based on [Airspeed Velocity](https://asv.readthedocs.io/).
+We have only implemented benchmarks for the Qiskit Addon, not the standalone mode.
+
+# Where are the benchmarks
+All the benchmarks are under the `test/benchmark` directory.
+There you'll find a bunch of `*_benchmarks.py` files which represent the different type of benchmarks we will run:
+- Quantum Volume with different number of qubits and noise models
+- Simple one-gate circuits with different number of qubits and noise models.
+
+
+# How to run the benchmarks
+All prerequistes for building the project need to be installed in the system, take a look at the [CONTRIBUTING guide](.github/CONTRIBUTING.md) if you don't have them already installed.
+
+Install Airspeed Velocity (`ASV`):
+```
+$ pip install asv
+```
+
+Move to the `test` directory:
+```
+$ cd test
+```
+
+And run `asv` using the correct configuration file, depeding on what O.S. you are executing them:
+Linux:
+```
+$ asv run --config asv.linux.conf.json
+```
+
+MacOS:
+```
+$ asv run --config asv.macos.conf.json
+```
+
+NOTE: We only support Linux and MacOS at the moment
+
+Depending on your system, benchmarks will take a while to complete.
+After the completion of the tests, you will see the results with a format similar like this:
+```
+· Creating environments
+· Discovering benchmarks
+· Running 3 total benchmarks (1 commits * 1 environments * 3 benchmarks)
+[  0.00%] · For qiskit-aer commit 8b4f4de1 <master>:
+[  0.00%] ·· Benchmarking conda-py3.7
+[ 16.67%] ··· Running (quantum_volume_benchmarks.QuantumVolumeTimeSuite.time_quantum_volume--)..
+[ 50.00%] ··· Running (simple_benchmarks.SimpleU3TimeSuite.time_simple_u3--).
+[ 66.67%] ··· quantum_volume_benchmarks.QuantumVolumeTimeSuite.time_quantum_volume
+[ 66.67%] ··· ================= ========== ===================== ============= =============
+              --                                        Noise Model                         
+              ----------------- ------------------------------------------------------------
+                Quantum Volume   No Noise   Mixed Unitary Noise   Reset Noise   Kraus Noise 
+              ================= ========== ===================== ============= =============
+                Num. qubits: 5   123±4ms          124±10ms          154±20ms      145±20ms  
+               Num. qubits: 10   292±10ms         319±50ms          299±10ms      346±2ms   
+               Num. qubits: 15   3.17±2s          2.38±2s           2.19±2s       9.76±2s   
+              ================= ========== ===================== ============= =============
+
+[ 83.33%] ··· simple_benchmarks.SimpleCxTimeSuite.time_simple_cx
+[ 83.33%] ··· ====================== ========== ===================== ============= =============
+              --                                             Noise Model                         
+              ---------------------- ------------------------------------------------------------
+               Simple cnot circuits   No Noise   Mixed Unitary Noise   Reset Noise   Kraus Noise 
+              ====================== ========== ===================== ============= =============
+                  Num. qubits: 5      47.2±2ms         111±40ms          46.8±1ms     21.1±0.2ms 
+                 Num. qubits: 10      117±60ms         172±60ms          164±30ms     28.6±0.3ms 
+                 Num. qubits: 15      213±30ms         223±80ms          223±90ms      258±5ms   
+              ====================== ========== ===================== ============= =============
+
+[100.00%] ··· simple_benchmarks.SimpleU3TimeSuite.time_simple_u3
+[100.00%] ··· ==================== ============ ===================== ============= =============
+              --                                            Noise Model                          
+              -------------------- --------------------------------------------------------------
+               Simple u3 circuits    No Noise    Mixed Unitary Noise   Reset Noise   Kraus Noise 
+              ==================== ============ ===================== ============= =============
+                 Num. qubits: 5     20.3±0.3ms        21.3±0.5ms        20.7±0.4ms    21.5±0.1ms 
+                Num. qubits: 10     26.6±0.2ms        27.8±0.3ms        27.6±0.5ms    28.4±0.2ms 
+                Num. qubits: 15      244±40ms          312±70ms         295±100ms     179±100ms  
+              ==================== ============ ===================== ============= =============
+
+```
+
+# Interpreting the data
+
+The output format is pretty self-explanatory, so every row starting with the text: `Num. quits:` repesents all the benchmarks run for this number of quibits configuration, more precisely, we run 4 benchmarks for every number of qubits configuration, and each of the benchmarks are run with a different noise model, so for example, this line:
+```
+  Quantum Volume   No Noise   Mixed Unitary Noise   Reset Noise   Kraus Noise
+================= ========== ===================== ============= =============
+  Num. qubits: 15   3.17±2s          2.38±2s           2.19±2s       9.76±2s
+```
+it's telling us that for our Quantum Volume circuit the time it took to complete was:
+- 3.17 seconds with no noise at all
+- 2.38 seconds with Mixed unitary noise
+- 2.19 seconds with Reset noise
+- 9.76 seconds with Kraus noise
+

test/asv.linux.conf.json

@@ -0,0 +1,185 @@
+// To use this configuration for running the benchmarks, we have to run asv like this:
+// $ asv --connfig asv.linux.conf.json
+{
+    // The version of the config file format.  Do not change, unless
+    // you know what you are doing.
+    "version": 1,
+
+    // The name of the project being benchmarked
+    "project": "qiskit-aer",
+
+    // The project's homepage
+    "project_url": "http://qiskit.org/aer",
+
+    // The URL or local path of the source code repository for the
+    // project being benchmarked
+    "repo": "../",
+
+    // The Python project's subdirectory in your repo.  If missing or
+    // the empty string, the project is assumed to be located at the root
+    // of the repository.
+    // "repo_subdir": "",
+
+    // Customizable commands for building, installing, and
+    // uninstalling the project. See asv.conf.json documentation.
+    //
+    // "install_command": ["python -mpip install {wheel_file}"],
+    // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
+    // "build_command": [
+    //     "python setup.py build",
+    //     "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
+    // ],
+
+    "install_command": [
+        "python -c \"import shutil; shutil.rmtree('{build_dir}/qiskit')\"",
+        "python -c \"import shutil; shutil.rmtree('{build_dir}/qiskit_aer.egg-info')\"",
+        "python -mpip install {wheel_file}"
+    ],
+    "uninstall_command": [
+        "return-code=any python -mpip uninstall -y qiskit-terra",
+        "return-code=any python -mpip uninstall -y {project}"
+    ],
+    "build_command": [
+        "python -mpip install -U scikit-build cython",
+        "python setup.py bdist_wheel --dist-dir={build_cache_dir} -- -DCMAKE_CXX_COMPILER=g++ -DSTATIC_LINKING=True -- -j8",
+        "python -mpip install https://github.com/QISKit/qiskit-terra/archive/master.zip"
+    ],
+
+    // List of branches to benchmark. If not provided, defaults to "master"
+    // (for git) or "default" (for mercurial).
+    // "branches": ["master"], // for git
+    // "branches": ["default"],    // for mercurial
+
+    // The DVCS being used.  If not set, it will be automatically
+    // determined from "repo" by looking at the protocol in the URL
+    // (if remote), or by looking for special directories, such as
+    // ".git" (if local).
+    // "dvcs": "git",
+
+    // The tool to use to create environments.  May be "conda",
+    // "virtualenv" or other value depending on the plugins in use.
+    // If missing or the empty string, the tool will be automatically
+    // determined by looking for tools on the PATH environment
+    // variable.
+    "environment_type": "conda",
+
+    // timeout in seconds for installing any dependencies in environment
+    // defaults to 10 min
+    //"install_timeout": 600,
+
+    // the base URL to show a commit for the project.
+    // "show_commit_url": "http://github.com/owner/project/commit/",
+
+    // The Pythons you'd like to test against.  If not provided, defaults
+    // to the current version of Python used to run `asv`.
+    // "pythons": ["2.7", "3.6"],
+
+    // The list of conda channel names to be searched for benchmark
+    // dependency packages in the specified order
+    // "conda_channels": ["conda-forge", "defaults"]
+
+    // The matrix of dependencies to test.  Each key is the name of a
+    // package (in PyPI) and the values are version numbers.  An empty
+    // list or empty string indicates to just test against the default
+    // (latest) version. null indicates that the package is to not be
+    // installed. If the package to be tested is only available from
+    // PyPi, and the 'environment_type' is conda, then you can preface
+    // the package name by 'pip+', and the package will be installed via
+    // pip (with all the conda available packages installed first,
+    // followed by the pip installed packages).
+    //
+    // "matrix": {
+    //     "numpy": ["1.6", "1.7"],
+    //     "six": ["", null],        // test with and without six installed
+    //     "pip+emcee": [""],   // emcee is only available for install with pip.
+    // },
+
+    //"matrix": {
+    //    "pip+qiskit-terra": [""],
+    //},
+
+    // Combinations of libraries/python versions can be excluded/included
+    // from the set to test. Each entry is a dictionary containing additional
+    // key-value pairs to include/exclude.
+    //
+    // An exclude entry excludes entries where all values match. The
+    // values are regexps that should match the whole string.
+    //
+    // An include entry adds an environment. Only the packages listed
+    // are installed. The 'python' key is required. The exclude rules
+    // do not apply to includes.
+    //
+    // In addition to package names, the following keys are available:
+    //
+    // - python
+    //     Python version, as in the *pythons* variable above.
+    // - environment_type
+    //     Environment type, as above.
+    // - sys_platform
+    //     Platform, as in sys.platform. Possible values for the common
+    //     cases: 'linux2', 'win32', 'cygwin', 'darwin'.
+    //
+    // "exclude": [
+    //     {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
+    //     {"environment_type": "conda", "six": null}, // don't run without six on conda
+    // ],
+    //
+    // "include": [
+    //     // additional env for python2.7
+    //     {"python": "2.7", "numpy": "1.8"},
+    //     // additional env if run on windows+conda
+    //     {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
+    // ],
+
+    // The directory (relative to the current directory) that benchmarks are
+    // stored in.  If not provided, defaults to "benchmarks"
+    "benchmark_dir": "benchmark",
+
+
+    // The directory (relative to the current directory) to cache the Python
+    // environments in.  If not provided, defaults to "env"
+    // "env_dir": "env",
+    "env_dir": ".asv/envs",
+
+    // The directory (relative to the current directory) that raw benchmark
+    // results are stored in.  If not provided, defaults to "results".
+    // "results_dir": "results",
+    "results_dir": ".asv/results",
+
+    // The directory (relative to the current directory) that the html tree
+    // should be written to.  If not provided, defaults to "html".
+    // "html_dir": "html",
+    "html_dir": ".asv/html"
+
+    // The number of characters to retain in the commit hashes.
+    // "hash_length": 8,
+
+    // `asv` will cache results of the recent builds in each
+    // environment, making them faster to install next time.  This is
+    // the number of builds to keep, per environment.
+    // "build_cache_size": 2,
+
+    // The commits after which the regression search in `asv publish`
+    // should start looking for regressions. Dictionary whose keys are
+    // regexps matching to benchmark names, and values corresponding to
+    // the commit (exclusive) after which to start looking for
+    // regressions.  The default is to start from the first commit
+    // with results. If the commit is `null`, regression detection is
+    // skipped for the matching benchmark.
+    //
+    // "regressions_first_commits": {
+    //    "some_benchmark": "352cdf",  // Consider regressions only after this commit
+    //    "another_benchmark": null,   // Skip regression detection altogether
+    // },
+
+    // The thresholds for relative change in results, after which `asv
+    // publish` starts reporting regressions. Dictionary of the same
+    // form as in ``regressions_first_commits``, with values
+    // indicating the thresholds.  If multiple entries match, the
+    // maximum is taken. If no entry matches, the default is 5%.
+    //
+    // "regressions_thresholds": {
+    //    "some_benchmark": 0.01,     // Threshold of 1%
+    //    "another_benchmark": 0.5,   // Threshold of 50%
+    // },
+}

test/asv.conf.json → test/asv.macos.conf.json

@@ -1,3 +1,5 @@
+// To use this configuration for running the benchmarks, we have to run asv like this:
+// $ asv --connfig asv.macos.conf.json run
 {
     // The version of the config file format.  Do not change, unless
     // you know what you are doing.
@@ -39,8 +41,8 @@
     ],
     "build_command": [
         "pip install -U scikit-build cython",
-        "python setup.py bdist_wheel --dist-dir={build_cache_dir} -- -DCMAKE_CXX_COMPILER=g++-7 -DSTATIC_LINKING=True -- -j8",
-        "pip install https://github.com/QISKit/qiskit-terra/archive/master.zip"
+        "pip install https://github.com/QISKit/qiskit-terra/archive/master.zip",
+        "VERBOSE=1 python setup.py bdist_wheel --plat-name macosx-10.9-x86_64 --dist-dir={build_cache_dir} -- -DSTATIC_LINKING=True -- -j8"
     ],
 
     // List of branches to benchmark. If not provided, defaults to "master"

test/benchmark/__init__.py

@@ -1 +1,15 @@
+from qiskit.qobj import Qobj
+from qiskit.providers.aer.noise import NoiseModel
 
+def qobj_repr_hook(self):
+    """ This is needed for ASV to beauty-printing reports """
+    return "Num. qubits: {0}".format(self.config.n_qubits)
+
+Qobj.__repr__ = qobj_repr_hook
+
+
+def noise_model_repr_hook(self):
+    """ This is needed for ASV to beauty-printing reports """
+    return self.__class__.__name__.replace("_", " ").capitalize()
+
+NoiseModel.__repr__ = noise_model_repr_hook

test/benchmark/quantum_volume_benchmarks.py

@@ -5,11 +5,8 @@
 import qiskit as Terra
 from qiskit import QiskitError
 from qiskit.providers.aer import QasmSimulator
-from .tools import quantum_volume_circuit
-from .tools import mixed_unitary_noise_model
-from .tools import reset_noise_model
-from .tools import kraus_noise_model
-
+from .tools import quantum_volume_circuit, mixed_unitary_noise_model, \
+                   reset_noise_model, kraus_noise_model, no_noise
 
 class QuantumVolumeTimeSuite:
     """
@@ -40,29 +37,29 @@ def __init__(self):
         self.timeout = 60 * 20
         self.qv_circuits = []
         self.backend = QasmSimulator()
-        for num_qubits in (16, ):
+        for num_qubits in (5, 10, 15):
             for depth in (10, ):
                 # We want always the same seed, as we want always the same circuits
                 # for the same value pairs of qubits,depth
                 circ = quantum_volume_circuit(num_qubits, depth, seed=1)
                 self.qv_circuits.append(
                     Terra.compile(
                         circ, self.backend, shots=1, basis_gates=['u3', 'cx']))
-        self.param_names = ["Quantum Volume (16qubits 10depth)", "Noise Model"]
+        self.param_names = ["Quantum Volume", "Noise Model"]
         # This will run every benchmark for one of the combinations we have here:
         # bench(qv_circuits, None) => bench(qv_circuits, mixed()) =>
         # bench(qv_circuits, reset) => bench(qv_circuits, kraus())
         self.params = (self.qv_circuits, [
-            None,
+            no_noise(),
             mixed_unitary_noise_model(),
             reset_noise_model(),
             kraus_noise_model()
         ])
 
-    def setup(self, qobj):
+    def setup(self, qobj, noise_model_wrapper):
         pass
 
-    def time_quantum_volume(self, qobj, noise_model):
-        result = self.backend.run(qobj, noise_model=noise_model).result()
+    def time_quantum_volume(self, qobj, noise_model_wrapper):
+        result = self.backend.run(qobj, noise_model=noise_model_wrapper()).result()
         if result.status != 'COMPLETED':
             raise QiskitError("Simulation failed. Status: " + result.status)