Example PyO3 project with automated test coverage for Rust and Python
This repository shows how to set up a continuous-integration job for measuring
coverage over a project using PyO3. Based on
the CI for PyO3's own tests,
this example provides a simpler starting point for new projects. Coverage is
computed using cargo-llvm-cov. The
coverage is measured over both the Python and Rust sections by taking advantage
of the ability of cargo-llvm-cov to measure coverage on any binary.
The repository contains 3 areas: (1) Rust only, (2) Rust with PyO3 bindings, and (3) Python only.
├── python
│ └── foobar
│ └── __init__.py # Python code
├── src
│ └── lib.rs # Rust code, PyO3 bindings, Rust tests
└── tests
└── test_foobar.py # Python tests, also covering PyO3/Rust
Each area defines a simple function for adding two numbers:
rust_sum- Rust onlypyo3_sum- Rust with PyO3 bindingspy_sum- Python only
In order to get full test coverage of all of the functions, both the Python and Rust tests need to be run. We'll show the process step by step.
Before getting started, install cargo-llvm-cov:
rustup component add llvm-tools-preview
cargo install cargo-llvm-cov
The Python tests require a virtual environment to isolate the packages from the
system. This installs pytest, which is able to measure the coverage of the
Python only sections and also exercise the PyO3 bound Rust code.
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
The Rust tests use cargo test. To measure the full Rust coverage from Python
tests, cargo-llvm-cov provides a set of environment variables that cause the
Rust binaries to include instrumentation coverage
that will save profile files when they are used. This means that any program
that exercises the binaries can be measured for the effect it has on coverage.
$ cargo llvm-cov show-env --export-prefix
export RUSTFLAGS=" -C instrument-coverage --cfg coverage --cfg trybuild_no_target"
export LLVM_PROFILE_FILE="/home/.../rust-python-coverage/target/rust-python-coverage-%m.profraw"
export CARGO_INCREMENTAL="0"
export CARGO_LLVM_COV_TARGET_DIR="/home/.../rust-python-coverage/target"
The profile files generated can be inspected in CARGO_LLVM_COV_TARGET_DIR.
In addition to these environment variables, it can be helpful to use incremental
compilation. For larger projects with nested directories and multiple Cargo.toml
files, it is necessary to centralize the CARGO_TARGET_DIR by setting it to the
CARGO_LLVM_COV_TARGET_DIR that is generated.
source <(cargo llvm-cov show-env --export-prefix)
export CARGO_TARGET_DIR=$CARGO_LLVM_COV_TARGET_DIR
export CARGO_INCREMENTAL=1
With these environment variables set up, we are ready to run the coverage measurements.
cargo llvm-cov clean --workspace
cargo test
maturin develop
pytest tests --cov=foobar --cov-report xml
cargo llvm-cov --no-run --lcov --output-path coverage.lcov
First the cargo llvm-cov clean command removes any previous profiling
information. We then run the regular tests, which builds the Rust version
of the library. We use maturin develop to build and install the Python
wheel for the project. Then the pytest command is used to exercise the
Python tests, while also generating a coverage report for the Python code.
The last step of this process happens in the CI, where we upload both coverage files to CodeCov. Merging reports is an automatic feature of CodeCov, so the final view shows the combined view.