Apply BOLT optimizations without rebuilding LLVM

[Kobzol]

This PR adds an explicit BOLT bootstrap step which applies BOLT on the fly when LLVM artifacts are copied to a sysroot (it only does this once per bootstrap invocation, the result is cached). This avoids one LLVM rebuild in the Linux CI dist build.

r? @jyn514

[Kobzol]

@bors try

[bors]

⌛ Trying commit e686fc13c0b16035839e374ead4ff9fef0d68cc1 with merge 5866a9a1a511dcca9566a367b0ccdf69e2b0aedf...

[jyn514]

I don't have time for reviews right now.

r? @Mark-Simulacrum cc @nikic

[bors]

☀️ Try build successful - checks-actions
Build commit: 5866a9a1a511dcca9566a367b0ccdf69e2b0aedf (5866a9a1a511dcca9566a367b0ccdf69e2b0aedf)

[Kobzol]

Hmm, the build has worked, but BOLT was executed multiple times and the whole build wasn't very fast. We should probably add information about the individual bootstrap step durations into the CI timer first.

[jyn514]

We should probably add information about the individual bootstrap step durations into the CI timer first.

Doesn't that already exist? I remember seeing RUSTC-TIMER log output or something like that

[Kobzol]

Yes, it's printed during the build, and stored into metrics.json, but it's not parsed by the Python PGO script yet, so we cannot see the aggregated information in the nice timing table at the end. I'll send a PR soon that will fix that.

[Kobzol]

@bors try

[bors]

⌛ Trying commit 250528f2590671e8d865d663c38aab0620a66916 with merge 6b1d08dcfe231d72d8f2310c341771dd724d43fa...

[bors]

☀️ Try build successful - checks-actions
Build commit: 6b1d08dcfe231d72d8f2310c341771dd724d43fa (6b1d08dcfe231d72d8f2310c341771dd724d43fa)

[bors]

☀️ Try build successful - checks-actions
Build commit: 6b1d08dcfe231d72d8f2310c341771dd724d43fa (6b1d08dcfe231d72d8f2310c341771dd724d43fa)

[Kobzol]

It seems that with this optimization (however it's implemented in the end), we can get to ~2h 5m Linux dist time. I'll check if perf hasn't regressed first though.

@rust-timer build 6b1d08dcfe231d72d8f2310c341771dd724d43fa

[rust-timer]

Finished benchmarking commit (6b1d08dcfe231d72d8f2310c341771dd724d43fa): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.0%	[0.4%, 1.4%]	3
Regressions ❌ (secondary)	3.2%	[1.7%, 4.3%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.0%	[0.4%, 1.4%]	3

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.1%	[-3.1%, -3.1%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-3.1%	[-3.1%, -3.1%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.1%	[-2.1%, -2.0%]	2
All ❌✅ (primary)	-	-	0

[lqd]

these benchmarks are currently noisy

[nikic]

Any thoughts on the variant described in #107521 (comment)? I think this approach (and #107521 as well) is going to be something of a dead end, because it leaves behind some problems that can't really be addressed while keeping the general approach of reusing LLVM artifacts from a previous build. Apart from the unnecessary final rustc rebuild (which is at least fairly fast), we also do the previous rustc build with a BOLT-instrumented LLVM, so that build ends up being slow. The other consideration is how this is going to generalize to optimizing rustc itself with BOLT, where sharing artifacts from previous builds would be less straightforward.

[Kobzol]

I discussed your idea with @jyn514 (https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/bootstrap.20LLVM.20postprocess.20step), and based on that discussion, I implemented a BOLT bootstrap step (I pushed it now) that applies BOLT changes on-the-fly when LLVM is copied to sysroot. But I'm not sure if it's exactly what you had in mind.

Apart from the unnecessary final rustc rebuild (which is at least fairly fast), we also do the previous rustc build with a BOLT-instrumented LLVM, so that build ends up being slow.

This could be solved simply by only performing the BOLT steps in stage2/dist, right? We want to apply BOLT when we copy LLVM to stage2 sysroot, but not before.

The other consideration is how this is going to generalize to optimizing rustc itself with BOLT, where sharing artifacts from previous builds would be less straightforward.

BOLT for rustc has been quite lackluster for now, so I wouldn't worry about it at the moment, first I'd like to focus on LLVM to reduce the CI time as soon as possible. But I suppose that for rustc it wouldn't be that different, we can apply the BOLT step after stage2 build finishes, we'll just need to make sure that we will bust the cache before the next build.

[Kobzol]

@bors try @rust-timer queue

[nikic]

This looks reasonable to me. Thanks for working on it!

[Mark-Simulacrum]

r=me unless these cleanups seem like improvements worth making

[Kobzol]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 9aad2ad with merge 341579aa2fdf71f726bc9f49d02b0160e27a9edf...

[bors]

☀️ Try build successful - checks-actions
Build commit: 341579aa2fdf71f726bc9f49d02b0160e27a9edf (341579aa2fdf71f726bc9f49d02b0160e27a9edf)

[rust-timer]

Finished benchmarking commit (341579aa2fdf71f726bc9f49d02b0160e27a9edf): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

[Kobzol]

Perf. looks good. Rebased on master and fixed nits.

@rustbot ready

[Mark-Simulacrum]

@bors r+

Thanks!

[bors]

📌 Commit 9aad2ad has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

⌛ Testing commit 9aad2ad with merge 35636f9...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 35636f9 to master...

[rust-timer]

Finished benchmarking commit (35636f9): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[3.9%, 3.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.8%, 0.8%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[0.8%, 0.8%]	1