rustc_metadata: Switch all decoder methods from vectors to iterators #92245

petrochenkov · 2021-12-24T07:41:34Z

To avoid allocations in some cases.

Also remove unnecessary is_proc_macro_crate checks from decoder, currently the general strategy is to shift all the work to the encoder and assume that all the encoded data is correct and can be decoded unconditionally in the decoder.

rust-highfive · 2021-12-24T07:41:37Z

r? @nagisa

(rust-highfive has picked a reviewer for you, use r? to override)

joshtriplett · 2021-12-24T09:27:12Z

@bors try @rust-timer queue

rust-timer · 2021-12-24T09:27:13Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2021-12-24T09:27:19Z

⌛ Trying commit 2a074cf61105ad9a668719faa3a240e7d1f78c7e with merge a1b48ab4b3ea7e5795cb5b843b52edd696feaaf3...

bors · 2021-12-24T11:01:14Z

☀️ Try build successful - checks-actions
Build commit: a1b48ab4b3ea7e5795cb5b843b52edd696feaaf3 (a1b48ab4b3ea7e5795cb5b843b52edd696feaaf3)

rust-timer · 2021-12-24T11:01:16Z

Queued a1b48ab4b3ea7e5795cb5b843b52edd696feaaf3 with parent 59337cd, future comparison URL.

rust-timer · 2021-12-24T12:29:28Z

Finished benchmarking commit (a1b48ab4b3ea7e5795cb5b843b52edd696feaaf3): comparison url.

Summary: This change led to small relevant mixed results 🤷 in compiler performance.

Small improvement in instruction counts (up to -0.4% on full builds of await-call-tree)
Small regression in instruction counts (up to 0.7% on incr-unchanged builds of helloworld)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

petrochenkov · 2021-12-24T14:37:32Z

No idea what could cause the regressions here, maybe some unlucky new inlining choices, I'll check.

petrochenkov · 2021-12-26T01:58:49Z

Blocked on #92277.

rustc_metadata: Stop passing `CrateMetadataRef` by reference (step 1) It's already a (fat) reference. Double referencing it creates lifetime issues for its methods that want to return iterators. --- Extracted from rust-lang#92245 for a perf run. The PR changes a lot of symbol names due to function signature changes, so it's hard to do differential profiling, let's spend some machine time instead.

petrochenkov · 2022-01-14T14:09:22Z

@bors try @rust-timer queue

rust-timer · 2022-01-14T14:09:24Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2022-01-14T14:41:12Z

⌛ Trying commit 14d56df404eb124a3599698edc6a53be9f30dca0 with merge 565e74827743aed951f73a4d339f5e9379afef9e...

bors · 2022-01-14T15:58:00Z

☀️ Try build successful - checks-actions
Build commit: 565e74827743aed951f73a4d339f5e9379afef9e (565e74827743aed951f73a4d339f5e9379afef9e)

rust-timer · 2022-01-14T15:58:01Z

Queued 565e74827743aed951f73a4d339f5e9379afef9e with parent 86f7f78, future comparison URL.

rust-timer · 2022-01-14T17:28:47Z

Finished benchmarking commit (565e74827743aed951f73a4d339f5e9379afef9e): comparison url.

Summary: This change led to small relevant improvements 🎉 in compiler performance.

Small improvement in instruction counts (up to -0.8% on incr-unchanged builds of cranelift-codegen)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf -perf-regression

nagisa · 2022-01-15T19:05:56Z

compiler/rustc_metadata/src/rmeta/decoder.rs

-            self.root.native_libraries.decode((self, sess)).collect()
-        }
+    fn get_native_libraries(self, sess: &'a Session) -> impl Iterator<Item = NativeLib> + 'a {
+        self.root.native_libraries.decode((self, sess))


Hm, it isn't entirely obvious to me that native_libraries will always decode to an empty list for is_proc_macro_crate() == true, though I'll admit that I didn't spend a long time spelunking the relevant code trying to figure out if there's any code elsewhere that ensures this property holds.

Is there any way it would make sense to add an assertion or something along the lines here?

For proc macro crates encoder now encodes only a specific set of metadata fields related to proc macros, everything else is empty.
It wasn't always like that, so decoder still has many is_proc_macro conditions.
We should generally cleanup it and remove them. But I guess they can be replaced with asserts instead, that would be more reliable.

Looks like we are not checking the is_proc_macro property in majority of cases.
This is how it looks if the asserts are added - petrochenkov@b026841
For Lazy<[T]> it looks like we cannot even check whether it's empty without decoding, that's why some checks are commented out.

So I think we should just drop the is_proc_macro checks without adding asserts, and assume that encoder encodes everything correctly.

For Lazy<[T]> it looks like we cannot even check whether it's empty without decoding, that's why some checks are commented out.

I think you can check the .meta field on a Lazy<[T]>.

I see.
IMO, it's still simpler to assume that we are just decoding whatever the encoder put there, and make the decoder "fast and dumb", I've updated the PR description.

I'm fine with not adding any of the asserts too, if we are sufficiently confident lack of them will not really cause any issues. Just wanted to make sure we at least considered it.

Also remove unnecessary `is_proc_macro_crate` checks from decoder

nagisa · 2022-01-16T14:03:56Z

@bors r+ rollup=never

bors · 2022-01-16T14:03:58Z

📌 Commit 4549b13 has been approved by nagisa

bors · 2022-01-16T14:25:28Z

⌛ Testing commit 4549b13 with merge 48e89b0...

bors · 2022-01-16T17:22:51Z

☀️ Test successful - checks-actions
Approved by: nagisa
Pushing 48e89b0 to master...

rust-timer · 2022-01-16T21:45:39Z

Finished benchmarking commit (48e89b0): comparison url.

Summary: This benchmark run did not return any relevant changes.

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

rust-highfive assigned nagisa Dec 24, 2021

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Dec 24, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 24, 2021

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 24, 2021

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Dec 24, 2021

petrochenkov mentioned this pull request Dec 24, 2021

[WIP] rustc_metadata: Switch each_child_of_item from callback to iterator #92251

Closed

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 24, 2021

petrochenkov force-pushed the cmrval branch from 2a074cf to 922447e Compare December 25, 2021 05:35

petrochenkov mentioned this pull request Dec 25, 2021

rustc_metadata: Stop passing CrateMetadataRef by reference (step 1) #92277

Merged

petrochenkov changed the title ~~rustc_metadata: Stop passing CrateMetadataRef by reference~~ rustc_metadata: Stop passing CrateMetadataRef by reference (step 2) Dec 26, 2021

petrochenkov added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 26, 2021

This comment has been minimized.

Sign in to view

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Jan 14, 2022

petrochenkov force-pushed the cmrval branch from 922447e to 14d56df Compare January 14, 2022 14:07

petrochenkov changed the title ~~rustc_metadata: Stop passing CrateMetadataRef by reference (step 2)~~ rustc_metadata: Switch all decoder methods from vectors to iterators Jan 14, 2022

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 14, 2022

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. perf-regression Performance regression. labels Jan 14, 2022

petrochenkov removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jan 14, 2022

nagisa reviewed Jan 15, 2022

View reviewed changes

rustc_metadata: Switch all decoder methods from vectors to iterators

4549b13

Also remove unnecessary `is_proc_macro_crate` checks from decoder

petrochenkov force-pushed the cmrval branch from 14d56df to 4549b13 Compare January 16, 2022 06:52

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 16, 2022

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 16, 2022

bors merged commit 48e89b0 into rust-lang:master Jan 16, 2022

rustbot added this to the 1.60.0 milestone Jan 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc_metadata: Switch all decoder methods from vectors to iterators #92245

rustc_metadata: Switch all decoder methods from vectors to iterators #92245

petrochenkov commented Dec 24, 2021 •

edited

Loading

rust-highfive commented Dec 24, 2021

joshtriplett commented Dec 24, 2021

rust-timer commented Dec 24, 2021

bors commented Dec 24, 2021

bors commented Dec 24, 2021

rust-timer commented Dec 24, 2021

rust-timer commented Dec 24, 2021

petrochenkov commented Dec 24, 2021

petrochenkov commented Dec 26, 2021

This comment has been minimized.

petrochenkov commented Jan 14, 2022

rust-timer commented Jan 14, 2022

bors commented Jan 14, 2022

bors commented Jan 14, 2022

rust-timer commented Jan 14, 2022

rust-timer commented Jan 14, 2022

nagisa Jan 15, 2022

petrochenkov Jan 16, 2022

petrochenkov Jan 16, 2022

bjorn3 Jan 16, 2022

petrochenkov Jan 16, 2022

nagisa Jan 16, 2022

nagisa commented Jan 16, 2022

bors commented Jan 16, 2022

bors commented Jan 16, 2022

bors commented Jan 16, 2022

rust-timer commented Jan 16, 2022

rustc_metadata: Switch all decoder methods from vectors to iterators #92245

rustc_metadata: Switch all decoder methods from vectors to iterators #92245

Conversation

petrochenkov commented Dec 24, 2021 • edited Loading

rust-highfive commented Dec 24, 2021

joshtriplett commented Dec 24, 2021

rust-timer commented Dec 24, 2021

bors commented Dec 24, 2021

bors commented Dec 24, 2021

rust-timer commented Dec 24, 2021

rust-timer commented Dec 24, 2021

petrochenkov commented Dec 24, 2021

petrochenkov commented Dec 26, 2021

This comment has been minimized.

petrochenkov commented Jan 14, 2022

rust-timer commented Jan 14, 2022

bors commented Jan 14, 2022

bors commented Jan 14, 2022

rust-timer commented Jan 14, 2022

rust-timer commented Jan 14, 2022

nagisa Jan 15, 2022

Choose a reason for hiding this comment

petrochenkov Jan 16, 2022

Choose a reason for hiding this comment

petrochenkov Jan 16, 2022

Choose a reason for hiding this comment

bjorn3 Jan 16, 2022

Choose a reason for hiding this comment

petrochenkov Jan 16, 2022

Choose a reason for hiding this comment

nagisa Jan 16, 2022

Choose a reason for hiding this comment

nagisa commented Jan 16, 2022

bors commented Jan 16, 2022

bors commented Jan 16, 2022

bors commented Jan 16, 2022

rust-timer commented Jan 16, 2022

petrochenkov commented Dec 24, 2021 •

edited

Loading