Improve large workspace build speeds by automatically using a separate CARGO_TARGET_DIR per workspace crate (opt-in)

[chinedufn]

Describe the problem you are trying to solve

The problem that I'd like to solve is having crates within a large workspace re-built when their source code hasn't changed, simply because you built another crate within your workspace (which I'm assuming tainted the cache in some sort of way but I'm not entirely familiar with the caching strategies).

To illustrate:

1. You have a large cargo workspace

2. Crates within your workspace depend on other crates within your workspace. In some cases multiple crates depend on the same crate within the workspace.

3. For example, Crate A and B both make use of crate C. Crate A uses certain feature flags, crate B uses certain feature flags.

4. cargo build crate-a && cargo build crate-b && cargo build crate-a && cargo build crate-b. I'd expect the second build of each crate to be skipped because nothing has changed, but in this example the second build of each crate might take a few seconds or more.

Describe the solution you'd like

The gist of the solution that I'd like is a very easy way to make every crate in the workspace use a separate CARGO_TARGET_DIR when built.

This would mean trading disk space for compile times (opt-in).

Right now I manually specify a CARGO_TARGET_DIR for some of my worst cases of "Aww .. things are getting re-built if nothing changed..." scenarios. Such as when building a few different distribution binaries in a CI job.

# CI (example)
# In most cases most of these crates haven't changed since the last CI build, so ideally nearly no time is spent
# on those build commands (right now I just set a separate CARGO_TARGET_DIR per command. That works great!
cargo build --release crate-1
cargo build --release crate-2
cargo build --release crate-3

But I haven't manually added CARGO_TARGET_DIR to other build scenarios yet - mainly because it's a (fairly minor) hassle to have to sprinkle the env var in everywhere that I build.

I'd prefer to say something such as

export CARGO_PER_WORKSPACE_CRATE_CARGO_TARGET_DIR=1

And from then on anytime cargo builds a crate it automatically uses target/MY_CRATE_NAME.

This would be very impactful for me when running my test suite where one of the commands is cargo test --all in a workspace that haas a couple dozen crates.

Of course I could take the time to write out all of my crate tests (or write a script that enumerates them) as CARGO_TARGET_DIR=... cargo test -p crate.

And I'll eventually probably do that - but I think that having Cargo offer an easy way to handle this instead of leaving it to user-land makes more sense since you end up saving quite a bit of compile time (if you're willing to trade away disk space) when you're working in a large workspace hopping around between different crates.

Notes

This is a quick sketch of one potential solution to one small aspect of the "reduce build times" problem.

I'd imagine that a more robust solution might look differently than this initial proposal.

Feel free to point out flaws and bad ideas!

[ehuss]

Building different crates in a workspace like that shouldn't trigger a rebuild. Each variant of each dependency is stored in the deps directory with a different filename to avoid rebuilds. There is one notable exception to that, and that is if you change the RUSTFLAGS environment variable between builds.

Is this a public project that I can look at? Or do you think you can reduce it to a reproduction?

[chinedufn]

Ah! So I just tried my example that I laid out in my issue comment and it doesn't trigger a re-build.

So apologies, that means that I'm not actually sure what is leading to me re-building more than I'd expect.

I do use CARGO_PROFILE_RELEASE_LTO=true in some places, not sure if that's similar to the RUSTFLAGS exception that you mention.

Alright - I'll close this and whenever I run into what's really going on I'll open a new issue.

Thanks!

[ehuss]

Changing LTO shouldn't cause a rebuild, except on a few platforms (macOS, Windows MSVC, and wasm32) where it will trigger re-linking the final executable. I think dylib and cdylib targets will also need to be rebuilt. Custom named profiles (#6988) can help with those cases if necessary.

[chinedufn]

I also have different build scripts across different workspace crates so maybe they’re contributing.

I don’t know - but I’ll pay attention to exactly when I see unexpected build times and see if I can open a more focused issue (or likely just fix something that I’m doing incorrectly).

Thanks for the link - had never heard of that proposal - will read up!

[chinedufn]

Hey, could my unexpected re-building have anything to do with the fact that I use cargo test --all (for example.. I've previously had issues with feature flags getting combined and having to use cargo test --all --exclude ... to exclude crates that were enabling features that other crates needed to be off)

Or is me using --all irrelevant here?

Changing LTO shouldn't cause a rebuild, except on a few platforms (macOS,

Hmm, I use macOS

I think dylib and cdylib targets will also need to be rebuilt

I also have a couple of cdylib's in the workspace.

---

I'm noticing that the re-building crates seem to be somewhat related to my cdylib (dependencies of my cdylib, or dependencies of its dependencies etc) so when I get around to trying #6988 I hope to be in good shape.

Thanks!