cargo test renders device unresponsive due to MSVC linker RAM usage

[djeedai]

Problem

Invoking cargo t on an i7-13700K Win11 with 32GB RAM makes the whole device unresponsive.

CPU is around 20-30%, RAM climbs at 100% and is likely paging heavily, process explorer shows 1 instance of the MSVC linker (link.exe) per core (24), each consuming 1+ GB of RAM.

Steps

1. Get a project with more doc tests than physical cores, and than GB or RAM. A good example is https://github.com/bevyengine/bevy, my own test is from https://github.com/djeedai/bevy_hanabi.
2. Run cargo t on Windows 11 (not sure version matters; use of MSVC is required though)

Possible Solution(s)

• Limit the number of threads used when cargo test runs rustdoc on Windows, due to the large RAM footprint of the MSVC linker (about 1 GB per test running / link.exe instance). This is likely not ideal because RAM footprint probably depends on crate(s) linked. In the example used (Bevy) the project is very large (300+ packages), so the linker very busy.
• Allow per-project configuring in advance the number of doc tests, for example in Cargo.toml (Enable specifying --test-threads in Cargo.toml #8430)

Notes

Confirmed that cargo t -- --test-threads=8 yields a lot smoother experience, as RAM usage never goes above 60% and therefore CPU usage is much higher (almost CPU bound).

This affects only doc tests, not regular tests (which don't link I guess). Solution ideally shouldn't limit the number of parallel tests for regular tests, only limit parallel linker instances / doc tests.

Somewhat related to #8430 but for different reason.

Version

cargo 1.72.1 (103a7ff2e 2023-08-15)
release: 1.72.1
commit-hash: 103a7ff2ee7678d34f34d778614c5eb2525ae9de
commit-date: 2023-08-15
host: x86_64-pc-windows-msvc
libgit2: 1.6.4 (sys:0.17.2 vendored)
libcurl: 8.1.2-DEV (sys:0.4.63+curl-8.1.2 vendored ssl:Schannel)
os: Windows 10.0.22621 (Windows 11 Professional) [64-bit]

[epage]

We're tracking CPU/memory pressure during compilation at #12912. doctests adds an interesting twist to this problem.

[djeedai]

Thanks @epage. I'd like to highlight that the pressure of non-doc tests has never been a problem with the mentioned projects as far as I know. Doctest ones though have started to be a consistent issue locally for several months, possibly more.

[weihanglo]

Specifically to linker RAM usage, they might be attributed to these issue:

• Stop emitting .debug_pubnames and .debug_pubtypes into Linux object files and binaries rust#48762
• https://rust-lang.zulipchat.com/#narrow/stream/317568-t-compiler.2Fwg-debugging/topic/investigating.20debuginfo.20size

At least on Linux, the debug_pubtypes and debug_pubnames take the largest portion in a binary. Doc tests do suffer from this as well from my experience.