[WIP] cstore: Add a new caching layer for data decoded from resolver #89059

petrochenkov · 2021-09-18T00:21:18Z

Implement something to address https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/.22Querifying.22.20early.20cstore.20accesses/near/253737343

One obvious further optimization would be to use some kind of arena to avoid cloning heavy decoded data like vectors.

rust-highfive · 2021-09-18T00:21:21Z

r? @oli-obk

(rust-highfive has picked a reviewer for you, use r? to override)

petrochenkov · 2021-09-18T00:21:48Z

@bors try @rust-timer queue

rust-timer · 2021-09-18T00:21:49Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2021-09-18T00:21:55Z

⌛ Trying commit 0d937e9d79af47aeae19877210cc568285eae27a with merge 4dd94872d389336d4dbbb73ba19e70bcd9473a01...

bors · 2021-09-18T01:44:31Z

☀️ Try build successful - checks-actions
Build commit: 4dd94872d389336d4dbbb73ba19e70bcd9473a01 (4dd94872d389336d4dbbb73ba19e70bcd9473a01)

rust-timer · 2021-09-18T01:44:33Z

Queued 4dd94872d389336d4dbbb73ba19e70bcd9473a01 with parent a58db2e, future comparison URL.

bjorn3 · 2021-09-18T05:59:02Z

compiler/rustc_metadata/src/rmeta/decoder.rs

+                        self.root.name,
+                        self.cnum,
+                    )
+                },


DefKind is very cheap to decode I think.

rust/compiler/rustc_hir/src/def.rs

Lines 55 to 119 in c7dbe7a

pub enum DefKind {

// Type namespace

Mod,

/// Refers to the struct itself, [`DefKind::Ctor`] refers to its constructor if it exists.

Struct,

Union,

Enum,

/// Refers to the variant itself, [`DefKind::Ctor`] refers to its constructor if it exists.

Variant,

Trait,

/// Type alias: `type Foo = Bar;`

TyAlias,

/// Type from an `extern` block.

ForeignTy,

/// Trait alias: `trait IntIterator = Iterator<Item = i32>;`

TraitAlias,

/// Associated type: `trait MyTrait { type Assoc; }`

AssocTy,

/// Type parameter: the `T` in `struct Vec<T> { ... }`

TyParam,

// Value namespace

Fn,

Const,

/// Constant generic parameter: `struct Foo<const N: usize> { ... }`

ConstParam,

Static,

/// Refers to the struct or enum variant's constructor.

///

/// The reason `Ctor` exists in addition to [`DefKind::Struct`] and

/// [`DefKind::Variant`] is because structs and enum variants exist

/// in the *type* namespace, whereas struct and enum variant *constructors*

/// exist in the *value* namespace.

///

/// You may wonder why enum variants exist in the type namespace as opposed

/// to the value namespace. Check out [RFC 2593] for intuition on why that is.

///

/// [RFC 2593]: https://github.com/rust-lang/rfcs/pull/2593

Ctor(CtorOf, CtorKind),

/// Associated function: `impl MyStruct { fn associated() {} }`

AssocFn,

/// Associated constant: `trait MyTrait { const ASSOC: usize; }`

AssocConst,

// Macro namespace

Macro(MacroKind),

// Not namespaced (or they are, but we don't treat them so)

ExternCrate,

Use,

/// An `extern` block.

ForeignMod,

/// Anonymous constant, e.g. the `1 + 2` in `[u8; 1 + 2]`, or `const { 1 + 2}`

AnonConst,

/// Opaque type, aka `impl Trait`.

OpaqueTy,

Field,

/// Lifetime parameter: the `'a` in `struct Foo<'a> { ... }`

LifetimeParam,

/// A use of [`global_asm!`].

GlobalAsm,

Impl,

Closure,

Generator,

}

rust-timer · 2021-09-18T07:37:41Z

Finished benchmarking commit (4dd94872d389336d4dbbb73ba19e70bcd9473a01): comparison url.

Summary: This change led to large relevant mixed results 🤷 in compiler performance.

Moderate improvement in instruction counts (up to -1.2% on full builds of coercions)
Large regression in instruction counts (up to 4.8% on full builds of deeply-nested-closures)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

petrochenkov · 2021-09-18T11:49:02Z

@bors try @rust-timer queue

rust-timer · 2021-09-18T11:49:03Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2021-09-18T11:49:09Z

⌛ Trying commit 3c24f7cb3f1a0d4e420033e9cc962121ecb32667 with merge f764f6372fa6071b7d65617b2fc04123b516bdbf...

bors · 2021-09-18T13:08:24Z

☀️ Try build successful - checks-actions
Build commit: f764f6372fa6071b7d65617b2fc04123b516bdbf (f764f6372fa6071b7d65617b2fc04123b516bdbf)

rust-timer · 2021-09-18T13:08:25Z

Queued f764f6372fa6071b7d65617b2fc04123b516bdbf with parent 5438ee4, future comparison URL.

rust-timer · 2021-09-18T15:43:22Z

Finished benchmarking commit (f764f6372fa6071b7d65617b2fc04123b516bdbf): comparison url.

Summary: This change led to large relevant regressions 😿 in compiler performance.

Large regression in instruction counts (up to 4.4% on full builds of deeply-nested)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

bors · 2021-09-21T01:46:03Z

☀️ Try build successful - checks-actions
Build commit: 6cc1b9a099c33502cb7d0d6f0df626a37b5997f9 (6cc1b9a099c33502cb7d0d6f0df626a37b5997f9)

rust-timer · 2021-09-21T01:46:05Z

Queued 6cc1b9a099c33502cb7d0d6f0df626a37b5997f9 with parent 60e70cc, future comparison URL.

rust-timer · 2021-09-21T09:29:27Z

Finished benchmarking commit (6cc1b9a099c33502cb7d0d6f0df626a37b5997f9): comparison url.

Summary: This change led to moderate relevant mixed results 🤷 in compiler performance.

Moderate improvement in instruction counts (up to -1.2% on incr-unchanged builds of externs)
Moderate regression in instruction counts (up to 1.6% on full builds of unify-linearly)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

petrochenkov · 2021-12-21T12:17:03Z

I'm not going to pursue this general mechanism anymore, given the results.
All the useful parts were extracted to #92156, #92159 and #92161.

rustc_metadata: Merge `get_ctor_def_id` and `get_ctor_kind` Also avoid decoding the whole `ty::AssocItem` to get a `has_self` flag. A small optimization and cleanup extracted from rust-lang#89059.

resolve: Minor miscellaneous cleanups from rust-lang#89059 `@bors` rollup=always

…askrgr Rollup of 7 pull requests Successful merges: - rust-lang#90586 (Relax priv-in-pub lint on generic bounds and where clauses of trait impls.) - rust-lang#92112 (Fix the error of checking `base_expr` twice in type_changing_struct_update) - rust-lang#92147 (rustc_builtin_macros: make asm mod public for rustfmt) - rust-lang#92161 (resolve: Minor miscellaneous cleanups from rust-lang#89059) - rust-lang#92264 (Remove `maybe_uninit_extra` feature from Vec docs) - rust-lang#92303 (Add test cases for issue rust-lang#26186) - rust-lang#92307 (Fix minor typos) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

rust-highfive assigned oli-obk Sep 18, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 18, 2021

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 18, 2021

bjorn3 reviewed Sep 18, 2021

View reviewed changes

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Sep 18, 2021

petrochenkov force-pushed the cachestore branch from 0d937e9 to 3c24f7c Compare September 18, 2021 11:48

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 18, 2021

petrochenkov mentioned this pull request Sep 18, 2021

rustdoc: Pre-calculate traits that are in scope for doc links #88679

Merged

petrochenkov removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 18, 2021

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Sep 18, 2021

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 18, 2021

petrochenkov force-pushed the cachestore branch from 3c24f7c to 89eb72e Compare September 18, 2021 19:30

petrochenkov removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Sep 21, 2021

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Sep 21, 2021

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 21, 2021

crlf0710 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 9, 2021

apiraino added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Oct 14, 2021

JohnCSimon added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 31, 2021

JohnCSimon added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 5, 2021

petrochenkov mentioned this pull request Dec 21, 2021

rustc_metadata: Merge get_ctor_def_id and get_ctor_kind #92156

Merged

petrochenkov closed this Dec 21, 2021

petrochenkov added a commit to petrochenkov/rust that referenced this pull request Dec 21, 2021

resolve: Minor miscellaneous cleanups from rust-lang#89059

1324100

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 27, 2021

Rollup merge of rust-lang#92161 - petrochenkov:misclean, r=cjgillot

a11414d

resolve: Minor miscellaneous cleanups from rust-lang#89059 `@bors` rollup=always

matthiaskrgr mentioned this pull request Dec 27, 2021

Rollup of 7 pull requests #92329

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] cstore: Add a new caching layer for data decoded from resolver #89059

[WIP] cstore: Add a new caching layer for data decoded from resolver #89059

petrochenkov commented Sep 18, 2021

rust-highfive commented Sep 18, 2021

petrochenkov commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 18, 2021

bors commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bjorn3 Sep 18, 2021

rust-timer commented Sep 18, 2021

petrochenkov commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 18, 2021

bors commented Sep 18, 2021

rust-timer commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 21, 2021

rust-timer commented Sep 21, 2021

rust-timer commented Sep 21, 2021

petrochenkov commented Dec 21, 2021

	pub enum DefKind {
	// Type namespace
	Mod,
	/// Refers to the struct itself, [`DefKind::Ctor`] refers to its constructor if it exists.
	Struct,
	Union,
	Enum,
	/// Refers to the variant itself, [`DefKind::Ctor`] refers to its constructor if it exists.
	Variant,
	Trait,
	/// Type alias: `type Foo = Bar;`
	TyAlias,
	/// Type from an `extern` block.
	ForeignTy,
	/// Trait alias: `trait IntIterator = Iterator<Item = i32>;`
	TraitAlias,
	/// Associated type: `trait MyTrait { type Assoc; }`
	AssocTy,
	/// Type parameter: the `T` in `struct Vec<T> { ... }`
	TyParam,

	// Value namespace
	Fn,
	Const,
	/// Constant generic parameter: `struct Foo<const N: usize> { ... }`
	ConstParam,
	Static,
	/// Refers to the struct or enum variant's constructor.
	///
	/// The reason `Ctor` exists in addition to [`DefKind::Struct`] and
	/// [`DefKind::Variant`] is because structs and enum variants exist
	/// in the type namespace, whereas struct and enum variant constructors
	/// exist in the value namespace.
	///
	/// You may wonder why enum variants exist in the type namespace as opposed
	/// to the value namespace. Check out [RFC 2593] for intuition on why that is.
	///
	/// [RFC 2593]: https://github.com/rust-lang/rfcs/pull/2593
	Ctor(CtorOf, CtorKind),
	/// Associated function: `impl MyStruct { fn associated() {} }`
	AssocFn,
	/// Associated constant: `trait MyTrait { const ASSOC: usize; }`
	AssocConst,

	// Macro namespace
	Macro(MacroKind),

	// Not namespaced (or they are, but we don't treat them so)
	ExternCrate,
	Use,
	/// An `extern` block.
	ForeignMod,
	/// Anonymous constant, e.g. the `1 + 2` in `[u8; 1 + 2]`, or `const { 1 + 2}`
	AnonConst,
	/// Opaque type, aka `impl Trait`.
	OpaqueTy,
	Field,
	/// Lifetime parameter: the `'a` in `struct Foo<'a> { ... }`
	LifetimeParam,
	/// A use of [`global_asm!`].
	GlobalAsm,
	Impl,
	Closure,
	Generator,
	}

[WIP] cstore: Add a new caching layer for data decoded from resolver #89059

[WIP] cstore: Add a new caching layer for data decoded from resolver #89059

Conversation

petrochenkov commented Sep 18, 2021

rust-highfive commented Sep 18, 2021

petrochenkov commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 18, 2021

bors commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bjorn3 Sep 18, 2021

Choose a reason for hiding this comment

rust-timer commented Sep 18, 2021

petrochenkov commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 18, 2021

bors commented Sep 18, 2021

rust-timer commented Sep 18, 2021

rust-timer commented Sep 18, 2021

bors commented Sep 21, 2021

rust-timer commented Sep 21, 2021

rust-timer commented Sep 21, 2021

petrochenkov commented Dec 21, 2021