Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test coverage documentation #4064

Merged
merged 1 commit into from
Aug 12, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 38 additions & 17 deletions guide/src/wasm-bindgen-test/coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,11 @@ Currently it is particularly difficult to [deliver compile-line arguments to pro

Make sure you are using `RUSTFLAGS=-Cinstrument-coverage -Zno-profiler-runtime`.

Due to the current limitation of `llvm-cov`, we can't collect profiling symbols from the generated `.wasm` files. Instead, we can grab them from the LLVM IR with `--emit=llvm-ir` by using Clang. Additionally, the emitted LLVM IR files by Rust contain invalid code that can't be parsed by Clang, so they need to be adjusted. Clang must use the same LLVM version that Rustc is using, which can be checkd by calling `rustc +nightly -vV`.
Due to the current limitation of `llvm-cov`, we can't collect profiling symbols from the generated `.wasm` files. Instead, we can grab them from the LLVM IR with `--emit=llvm-ir` by using Clang. Additionally, the emitted LLVM IR files by Rust contain invalid code that can't be parsed by Clang, so they need to be adjusted.

At the time of writing Rust Nightly uses LLVM v19, however [minicov] only supports LLVM v18. Usage of Clang or any LLVM tools must match the version used by [minicov].

[minicov]: https://crates.io/crates/minicov

### Arguments to the test runner

Expand All @@ -43,6 +47,8 @@ This feature relies on the [minicov] crate, which provides a profiling runtime f

### Example

This adapts code taken from the [Rustc book], see that for more examples and general information on test coverage as well.

```sh
# Run the tests:
# - `CARGO_HOST_RUSTFLAGS` to pass the configuration to `wasm-bindgen-macro`.
Expand All @@ -53,26 +59,41 @@ RUSTFLAGS="-Cinstrument-coverage -Zno-profiler-runtime --emit=llvm-ir --cfg=wasm
CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner \
cargo +nightly test -Ztarget-applies-to-host -Zhost-config --tests
# Adjust the LLVM IR and compile to object files:
# - Replaces every function body with `unreachable`.
# - Removes Rust-specific `range` annotations from function signatures.
name=name_of_the_tested_crate_in_snake_case; \
for file in `ls target/wasm32-unknown-unknown/debug/deps/$name-*.ll`; \
do \
perl -i -p0e 's/(^define.*?$).*?^}/$1\nstart:\n unreachable\n}/gms' $file && \
perl -i -p0e 's/(?<=noundef) range\(.*?\)//g' $file && \
clang $file -Wno-override-module -c; \
# - Extract a list of compiled artifacts from Cargo and filter them with `jq`.
# - Figure out the path to the LLVM IR file corresponding to an artifact.
# - Replace every function body with `unreachable`.
# - Remove Rust-specific `range` annotations from function signatures.
# - Compile to object file with Clang and store for later usage with `llvm-cov`.
crate_name=name_of_the_tested_crate_in_snake_case
objects=()
IFS=$'\n'
for file in $(
CARGO_HOST_RUSTFLAGS=--cfg=wasm_bindgen_unstable_test_coverage \
RUSTFLAGS="-Cinstrument-coverage -Zno-profiler-runtime --emit=llvm-ir --cfg=wasm_bindgen_unstable_test_coverage" \
cargo +nightly test -Ztarget-applies-to-host -Zhost-config --tests --no-run --message-format=json | \
jq -r "select(.reason == \"compiler-artifact\") | (select(.target.kind == [\"test\"]) // select(.target.name == \"$crate_name\")) | .filenames[0]"
)
do
if [[ ${file##*.} == "rlib" ]]; then
base=$(basename $file .rlib)
file=$(dirname $file)/${base#"lib"}.ll
else
file=$(dirname $file)/$(basename $file .wasm).ll
fi

perl -i -p0e 's/(^define.*?$).*?^}/$1\nstart:\n unreachable\n}/gms' $file
perl -i -p0e 's/(^define( [^ ]+)*) range\(.*?\)/$1/gm' $file

output = $(basename $file .ll).o
clang-18 $input -Wno-override-module -c -o $output
objects+=(-object $output)
done
# Merge all generated raw profiling data.
# This uses `cargo-binutils` which uses LLVM tools shipped by Rust to make sure there is no LLVM version discrepancy.
# But `llvm-profdata` can be used directly as well.
# See <https://crates.io/crates/cargo-binutils>.
rust-profdata merge -sparse ./*.profraw -o coverage.profdata
# Produce test coverage data in the HTML format.
rust-cov show --instr-profile=coverage.profdata --object ./*.o --format=html --Xdemangler=rust-demangler --sources src --output-dir coverage
llvm-profdata-18 merge -sparse *.profraw -o coverage.profdata
# Produce test coverage data in the HTML format and pass the object files we generated earlier.
llvm-cov-18 show -show-instantiations=false -Xdemangler=rustfilt -output-dir coverage -format=html -instr-profile=coverage.profdata ${objects[@]} -sources src
```

The [rustc book] has a lot more exapmles and information on test coverage as well.

[rustc book]: https://doc.rust-lang.org/nightly/rustc/instrument-coverage.html

## Attribution
Expand Down