coverage: Unify tests/coverage-map and tests/run-coverage into tests/coverage

[Zalathar]

Ever since the introduction of the coverage-map suite, it's been awkward to have to manage two separate coverage test directories containing dozens of mostly-identical files.

However, those two suites were separate for good reasons. They have very different requirements (since only one of them requires actually running the test program), running only one suite is noticeably faster than running both, and having separate suites allows them to be blessed separately if desired. So while unifying them was an obvious idea, actually doing so was non-trivial.

---

Nevertheless, this PR finds a way to merge the two suites into one directory while retaining almost all of the developer-experience benefits of having two suites. This required non-trivial implementations of Step, but the end result works very smoothly.

---

The first 5 commits are a copy of #117340, which has been closed in favour of this PR.

[rustbot]

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

[Zalathar]

Some other approaches that I considered but ultimately rejected:

• The original idea was to just declare two different test suites in bootstrap that point to the same directory. However, bootstrap contains some assumptions that aren't compatible with this, so the easy approach doesn't actually work.
  • One of those assumptions is fixed by one of the patches in this PR, but there were other remaining difficulties with path handling. This PR is in many ways an improved variant of that original approach.
• An obvious idea is to fully unify the two suites/modes into one compiletest mode that runs both sets of checks. This carries several drawbacks:
  • A key advantage of the coverage-map tests is that they don't require the profiler runtime, which lets them run in more CI configurations (e.g. for PRs) and makes them easier to run locally. There's no way to reconcile that with the fact that run-coverage tests do require the profiler runtime, which is a major reason why the two suites were separate in the first place.
  • During development, it's important to be able to run only one of the modes if desired, both to make relevant test rebuilds/runs faster, and to allow selective blessing of only one mode. So while I did have a mostly-working implementation of this approach, I discarded it because the developer-experience regression would have been unacceptable.
• A related idea I had was to hook into the compiletest code that splits a test into multiple revisions, and have it also split run-coverage tests into multiple modes.
  • This would have required adding several new features to compiletest to preserve important properties of the existing developer experience.
  • I did have a mostly-working implementation of this, but once I figured out how to do a similar thing at the bootstrap level with aliases, the bootstrap approach seemed like a clearly superior solution.

[Zalathar]

I've also been thinking about renaming the run-coverage alias/mode to coverage-run, but I think that's too much scope creep for this PR. Better to keep the old name for now, and worry about changing it in a separate PR.

[Zalathar]

Found and fixed an additional papercut: Because compiletest was using the same output directory for both modes, they would interfere with each others' timestamp files.

I've fixed this by adding a special case to compiletest's output path for coverage test modes (diff).

[Zalathar]

@rustbot label +A-code-coverage

[Zalathar]

Oh, I notice there's already a Mode::disambiguator method that serves a similar purpose; perhaps I should hook into that instead.

EDIT: Done. The existing method wasn't quite what I wanted, but I used it as inspiration for a new method.

[Mark-Simulacrum]

@bors r+ p=1

[bors]

📌 Commit 523997b has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

⌛ Testing commit 523997b with merge 08c3545...

[bors]

💔 Test failed - checks-actions

[Zalathar]

Looks like output normalization is missing paths on Windows.

Perhaps because we’re using suite_path in --src-base, which has a forward slash.

[cjgillot]

@bors r+ rollup=never

[bors]

📌 Commit 4e6f438 has been approved by cjgillot

It is now in the queue for this repository.

[bors]

⌛ Testing commit 4e6f438 with merge 91cfcb0...

[bors]

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 91cfcb0 to master...

[rust-timer]

Finished benchmarking commit (91cfcb0): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.5%, 0.5%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.5%	[0.5%, 0.5%]	1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.4%, 1.7%]	11
Regressions ❌ (secondary)	3.2%	[2.1%, 4.4%]	2
Improvements ✅ (primary)	-0.6%	[-1.0%, -0.5%]	5
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.4%	[-1.0%, 1.7%]	16

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 662.798s -> 664.133s (0.20%)
Artifact size: 308.75 MiB -> 308.79 MiB (0.01%)