Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace most of rustc_metadata::schema with something based on queries. #65407

Open
eddyb opened this issue Oct 14, 2019 · 0 comments
Open

Replace most of rustc_metadata::schema with something based on queries. #65407

eddyb opened this issue Oct 14, 2019 · 0 comments
Labels
A-metadata Area: Crate metadata C-cleanup Category: PRs that clean code up or issues documenting cleanup. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@eddyb
Copy link
Member

eddyb commented Oct 14, 2019

Right now (cross-crate) "metadata" is encoded as an ad-hoc heterogeneous tree, described in rustc_metadata::schema, with Lazy<T> acting as indirection (as in "pointer to T", inside the "metadata" blob) and letting the user choose whether to decode of the T value.

There is also a random-access array (called "table" in #59953), which is currently only used for Entry.

This cross-crate system predates the on-demand/incremental query system, and we have accumulated a lot of data in the schema which is similar (but not always identical) to certain queries, and additional code to present that information through queries.

The disadvantages I see with the current approach are:

  • a lot of schema/encoder/decoder boilerplate for everything
    • most of which isn't documented well, perpetuating the ad-hoc-ness
  • somewhat inconsistent organization
    • e.g. predicates in Entry vs super_predicates in TraitData
  • decoding more than is needed, even if mostly Lazy pointers

In #59953, the table of Entrys is replaced by a table for everything that used to be in an Entry field.
For example, the predicates_of query would then perform predicates[i].decode() instead of entries[i].decode().predicates.decode() (irrelevant details elided).

This is effectively a trade-off:

However, we can go further - #59953 doesn't touch EntryKind, which is still a sprawling enum with even two levels of Lazy indirection in places.


Ultimately, we could have "cross-crate metadata" be one table per query in most cases. This would accentuate the trade-off from #59953 further, but it would also allow simplifying rustc_metadata and unifying it further with incremental save&restore.

One of the queries that would benefit most from this is def_kind, which could be stored as a fully-populated table of bytes, much more compact and cheaper to decode than EntryKind today.

@jonas-schievink jonas-schievink added A-metadata Area: Crate metadata C-cleanup Category: PRs that clean code up or issues documenting cleanup. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Oct 14, 2019
JohnTitor added a commit to JohnTitor/rust that referenced this issue Oct 22, 2019
…ables, r=michaelwoerister

rustc_metadata: use a table for super_predicates, fn_sig, impl_trait_ref.

This is an attempt at a part of rust-lang#65407, i.e. moving parts of cross-crate "metadata" into tables that match queries more closely.

Three new tables should be enough to see some perf/metadata size changes.
(need to do something similar to rust-lang#59953 (comment))

There are other bits of data that could be made into tables, but they can be more compact so the impact would likely be not as bad, and they're also more work to set up.
Centril added a commit to Centril/rust that referenced this issue Oct 23, 2019
…ables, r=michaelwoerister

rustc_metadata: use a table for super_predicates, fn_sig, impl_trait_ref.

This is an attempt at a part of rust-lang#65407, i.e. moving parts of cross-crate "metadata" into tables that match queries more closely.

Three new tables should be enough to see some perf/metadata size changes.
(need to do something similar to rust-lang#59953 (comment))

There are other bits of data that could be made into tables, but they can be more compact so the impact would likely be not as bad, and they're also more work to set up.
Centril added a commit to Centril/rust that referenced this issue Oct 23, 2019
…ables, r=michaelwoerister

rustc_metadata: use a table for super_predicates, fn_sig, impl_trait_ref.

This is an attempt at a part of rust-lang#65407, i.e. moving parts of cross-crate "metadata" into tables that match queries more closely.

Three new tables should be enough to see some perf/metadata size changes.
(need to do something similar to rust-lang#59953 (comment))

There are other bits of data that could be made into tables, but they can be more compact so the impact would likely be not as bad, and they're also more work to set up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-metadata Area: Crate metadata C-cleanup Category: PRs that clean code up or issues documenting cleanup. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

2 participants