Hard-linking incremental files can be expensive.

[ehuss]

Doing some profiling to determine why incremental Cargo builds are slow, I noticed that hard-linking the incremental files takes a nontrivial amount of time. This issue is to ask those in-the-know if it may be possible to improve it.

On my system (macos, apfs, an extremely fast ssd), it takes between 500 to 750ms of a 7s incremental build (7% to 10%) just to hard-link the incremental files. The incremental build of cargo's lib generates about 1000 files, and it appears the incremental system links them 3 times (once from incremental to working, then working to deps, and then deps back to incremental), for a total of about 3,000 hard links.

I see two things here:

1. Can incremental produce fewer files?
2. Can incremental not hard-link the same files 3 times?

[nagisa]

Producing fewer files for incremental compilation would reduce reuse/hit ratio. Not sure whether number of links can be reduced, but OS taking 0.2ms on average to increment a link count and add inode to a directory list seems so excessive that I suspect there might be something else going as well (e.g. perhaps we’re saturating the I/O interface with other ops?)

[pnkfelix]

(@wesleywiser notes that this might also be a big issue on windows, since we probably aren't using junction points there, and thus we may be copying files back and forth there, which would be ... bad with respect to time and space usage...)

[pnkfelix]

One idea also posed by @wesleywiser : we could consider switching from leaning on FS features like linking, and instead use e.g. a text file storing the linking info. The latter might be much faster.

[pnkfelix]

Discussed in today's meeting. Next steps here may be to evaluate which files are involved in the sym-linking (is it just serialized dep-graph info, or is it also .o files that will be passed to linker invocations).

The reason for the question is to determine how cross-cutting a change here would be (to introduce an auxiliary file as discussed in my previous comment).

[ehuss]

@pnkfelix I'm not too familiar with how incremental works, but the files are of the form: target/debug/incremental/cargo-15l3bj1wk5kkj/s-ftbw7tdi60-872ygq-10c9jr4cx1rwy/2y4saqot8nvtvbxb.o.

I just checked, and it looks like rustc is hard-linking fewer files than it used to. For Cargo's library, in the latest 1.50 nightly, it hard-links about 259 .o files (3 times). In 1.37 (when this issue was filed), it hard-linked 890 .o files (3 times). A lot has changed in the past year, a wild guess would be changes in partitioning?