Skip to content

Commit

Permalink
Update documentation for coverage tests
Browse files Browse the repository at this point in the history
  • Loading branch information
Zalathar committed Nov 8, 2023
1 parent 77dbe57 commit d1fd1d3
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 25 deletions.
24 changes: 13 additions & 11 deletions src/llvm-coverage-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,20 +273,22 @@ since it will not be called), and adds a new `FunctionCoverage`, with

## Testing LLVM Coverage

[(See also the compiletest documentation for the `tests/coverage-map` and
`tests/run-coverage` test suites.)](./tests/compiletest.md#coverage-tests)
[(See also the compiletest documentation for the `tests/coverage`
test suite.)](./tests/compiletest.md#coverage-tests)

Coverage instrumentation in the MIR is validated by a `mir-opt` test:
[`tests/mir-opt/instrument_coverage.rs`].

Coverage instrumentation in LLVM IR is validated by the [`tests/coverage-map`]
test suite. These tests compile a test program to LLVM IR assembly, and then
Coverage instrumentation in LLVM IR is validated by the [`tests/coverage`]
test suite in `coverage-map` mode.
These tests compile a test program to LLVM IR assembly, and then
use the [`src/tools/coverage-dump`] tool to extract and pretty-print the
coverage mappings that would be embedded in the final binary.

End-to-end testing of coverage instrumentation and coverage reporting is
performed by the [`tests/run-coverage`] and [`tests/run-coverage-rustdoc`]
test suites. These tests compile and run a test program with coverage
performed by the [`tests/coverage`] test suite in `coverage-run` mode,
and by the [`tests/coverage-run-rustdoc`] test suite.
These tests compile and run a test program with coverage
instrumentation, then use LLVM tools to convert the coverage data into a
human-readable coverage report.

Expand All @@ -296,19 +298,19 @@ expected LLVM IR instructions and structured data for a coverage-enabled
program, including various checks for Coverage Map-related metadata and the LLVM
intrinsic calls to increment the runtime counters.

Expected results for the `coverage-map`, `run-coverage`, `run-coverage-rustdoc`,
Expected results for the `coverage`, `coverage-run-rustdoc`,
and `mir-opt` tests can be refreshed by running:

```shell
./x test tests/*coverage* --bless
./x test coverage --bless
./x test coverage-run-rustdoc --bless
./x test tests/mir-opt --bless
```

[`tests/mir-opt/instrument_coverage.rs`]: https://github.com/rust-lang/rust/blob/master/tests/mir-opt/instrument_coverage.rs
[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map
[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage
[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump
[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage
[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc
[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc
[`coverage-llvmir`]: https://github.com/rust-lang/rust/tree/master/tests/run-make/coverage-llvmir

## Implementation Details of the `InstrumentCoverage` MIR Pass
Expand Down
47 changes: 33 additions & 14 deletions src/tests/compiletest.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,8 @@ The following test suites are available, with links for more information:
- `run-make-fulldeps``run-make` tests which require a linkable build of `rustc`,
or the rust demangler
- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
- [`coverage-map`](#coverage-tests) - tests for coverage maps produced by
coverage instrumentation
- [`run-coverage`](#coverage-tests) - tests that run an instrumented program
and check its coverage report
- [`run-coverage-rustdoc`](#coverage-tests) - coverage tests that also run
- [`coverage`](#coverage-tests) - tests for coverage instrumentation
- [`coverage-run-rustdoc`](#coverage-tests) - coverage tests that also run
instrumented doctests
- [Rustdoc tests](../rustdoc.md#tests):
- `rustdoc` — tests for rustdoc, making sure that the generated files
Expand Down Expand Up @@ -402,7 +399,28 @@ These may be removed in the future.

### Coverage tests

The tests in [`tests/coverage-map`] test the mappings between source code
The tests in [`tests/coverage`] are shared by multiple test modes that test
coverage instrumentation in different ways.
Running the `coverage` test suite will automatically run each test in all of
the different coverage modes.

Each mode also has an alias to run the coverage tests in just that mode:

```bash
./x test coverage # runs all of tests/coverage in all coverage modes
./x test tests/coverage # same as above

./x test tests/coverage/if.rs # runs the specified test in all coverage modes

./x test coverage-map # runs all of tests/coverage in "coverage-map" mode only
./x test coverage-run # runs all of tests/coverage in "coverage-run" mode only

./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
```

---

In `coverage-map` mode, these tests verify the mappings between source code
regions and coverage counters that are emitted by LLVM.
They compile the test with `--emit=llvm-ir`,
then use a custom tool ([`src/tools/coverage-dump`])
Expand All @@ -416,18 +434,18 @@ coverage reports.

As a rule of thumb, any PR that doesn't change coverage-specific
code should **feel free to re-bless** the `coverage-map` tests as necessary,
without worrying about the actual changes, as long as the `run-coverage` tests
without worrying about the actual changes, as long as the `coverage-run` tests
still pass.

---

The tests in [`tests/run-coverage`] perform an end-to-end test of coverage reporting.
In `coverage-run` mode, these tests perform an end-to-end test of coverage reporting.
They compile a test program with coverage instrumentation, run that program to
produce raw coverage data, and then use LLVM tools to process that data into a
human-readable code coverage report.

Instrumented binaries need to be linked against the LLVM profiler runtime,
so `run-coverage` tests are **automatically skipped**
so `coverage-run` tests are **automatically skipped**
unless the profiler runtime is enabled in `config.toml`:

```toml
Expand All @@ -439,14 +457,15 @@ profiler = true
This also means that they typically don't run in PR CI jobs,
though they do run as part of the full set of CI jobs used for merging.

The tests in [`tests/run-coverage-rustdoc`] also run instrumented doctests and
---

The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and
include them in the coverage report. This avoids having to build rustdoc when
only running the main `run-coverage` suite.
only running the main `coverage` suite.

[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map
[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage
[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump
[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage
[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc
[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc


## Building auxiliary crates
Expand Down

0 comments on commit d1fd1d3

Please sign in to comment.