Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode MIR metadata by iterating on DefId instead of traversing the HIR tree #81215

Merged
merged 6 commits into from
Feb 5, 2021

Conversation

cjgillot
Copy link
Contributor

Split out of #80347.

This part only traverses mir_keys and encodes MIR according to the def kind.

r? @oli-obk

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 20, 2021
(true, false)
}
// Closures and functions
DefKind::Generator | DefKind::Closure | DefKind::AssocFn | DefKind::Fn => {
Copy link
Contributor

@tmiasko tmiasko Jan 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently optimized MIR for generators is always required. I will have to revisit the tests if this implementation manages to pass them. EDIT: It doesn't.

@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-llvm-9 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
failures:

---- [ui] ui/consts/const-int-arithmetic.rs stdout ----

error: test compilation failed although it shouldn't!
status: exit code: 1
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/consts/const-int-arithmetic.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zemit-future-incompat-report" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/const-int-arithmetic/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/const-int-arithmetic/auxiliary"
------------------------------------------

------------------------------------------
stderr:
stderr:
------------------------------------------
error: any use of this value will cause an error
   |
   |
LL |                 const $label: $ty = $expr;
...
...
LL |         C41: 100i8.wrapping_div(10), 10;
   |              ^^^^^^^^^^^^^^^^^^^^^^ "calling extern function `core::num::<impl i8>::wrapping_div`" needs an rfc before being allowed inside constants
   |
   = note: `#[deny(const_err)]` on by default

error: any use of this value will cause an error
   |
   |
LL |                 const $label: $ty = $expr;
...
...
LL |         C42: (-128i8).wrapping_div(-1), -128;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^ "calling extern function `core::num::<impl i8>::wrapping_div`" needs an rfc before being allowed inside constants

error: any use of this value will cause an error
   |
   |
LL |                 const $label: $ty = $expr;
...
...
LL |         C43: 100i8.wrapping_rem(10), 0;
   |              ^^^^^^^^^^^^^^^^^^^^^^ "calling extern function `core::num::<impl i8>::wrapping_rem`" needs an rfc before being allowed inside constants

error: any use of this value will cause an error
   |
   |
LL |                 const $label: $ty = $expr;
...
...
LL |         C44: (-128i8).wrapping_rem(-1), 0;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^ "calling extern function `core::num::<impl i8>::wrapping_rem`" needs an rfc before being allowed inside constants
error: aborting due to 4 previous errors


------------------------------------------
------------------------------------------


---- [ui] ui/consts/const-size_of_val-align_of_val.rs stdout ----

error: test compilation failed although it shouldn't!
status: exit code: 1
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/consts/const-size_of_val-align_of_val.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zemit-future-incompat-report" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/const-size_of_val-align_of_val/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/const-size_of_val-align_of_val/auxiliary"
------------------------------------------

------------------------------------------
stderr:
stderr:
------------------------------------------
error: any use of this value will cause an error
   |
   |
LL | const SIZE_OF_DANGLING: usize = unsafe { mem::size_of_val_raw(0x100 as *const i32) };
   |                                          |
   |                                          |
   |                                          "calling extern function `size_of_val_raw`" needs an rfc before being allowed inside constants
   |
   = note: `#[deny(const_err)]` on by default

error: any use of this value will cause an error
   |
   |
LL | const ALIGN_OF_DANGLING: usize = unsafe { mem::align_of_val_raw(0x100 as *const i16) };
   |                                           |
   |                                           |
   |                                           "calling extern function `align_of_val_raw`" needs an rfc before being allowed inside constants
error: aborting due to 2 previous errors


------------------------------------------
------------------------------------------


---- [ui] ui/consts/ptr_is_null.rs stdout ----

error: test compilation failed although it shouldn't!
status: exit code: 1
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/consts/ptr_is_null.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zemit-future-incompat-report" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/ptr_is_null" "-A" "unused" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--crate-type=lib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/consts/ptr_is_null/auxiliary"
------------------------------------------

------------------------------------------
stderr:
stderr:
------------------------------------------
error: any use of this value will cause an error
   |
   |
LL | pub const _: () = assert!(!(FOO as *const usize).is_null());
   |                            |
   |                            |
   |                            "calling extern function `ptr::const_ptr::<impl *const T>::is_null`" needs an rfc before being allowed inside constants
   |
   = note: `#[deny(const_err)]` on by default

error: any use of this value will cause an error
   |
   |
LL | pub const _: () = assert!(!(42 as *const usize).is_null());
   |                            |
   |                            |
   |                            "calling extern function `ptr::const_ptr::<impl *const T>::is_null`" needs an rfc before being allowed inside constants

error: any use of this value will cause an error
   |
   |
LL | pub const _: () = assert!((0 as *const usize).is_null());
   |                           |
   |                           |
   |                           "calling extern function `ptr::const_ptr::<impl *const T>::is_null`" needs an rfc before being allowed inside constants

error: any use of this value will cause an error
   |
   |
LL | pub const _: () = assert!(std::ptr::null::<usize>().is_null());
   |                           |
   |                           |
   |                           "calling extern function `ptr::const_ptr::<impl *const T>::is_null`" needs an rfc before being allowed inside constants

error: any use of this value will cause an error
   |
   |
LL | pub const _: () = assert!(!("foo" as *const str).is_null());
   |                            |
   |                            |
   |                            "calling extern function `ptr::const_ptr::<impl *const T>::is_null`" needs an rfc before being allowed inside constants
error: aborting due to 5 previous errors


------------------------------------------
------------------------------------------


---- [ui] ui/generator/metadata-sufficient-for-layout.rs stdout ----

error: test compilation failed although it shouldn't!
status: exit code: 101
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/generator/metadata-sufficient-for-layout.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zemit-future-incompat-report" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/generator/metadata-sufficient-for-layout" "-A" "unused" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/generator/metadata-sufficient-for-layout/auxiliary"
------------------------------------------

------------------------------------------
stderr:
stderr:
------------------------------------------
error: internal compiler error: compiler/rustc_metadata/src/rmeta/decoder.rs:1183:17: get_optimized_mir: missing MIR for `DefId(18:7 ~ metadata_sufficient_for_layout[8787]::g::{closure#0})`
thread 'rustc' panicked at 'Box<Any>', compiler/rustc_errors/src/lib.rs:958:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

note: the compiler unexpectedly panicked. this is a bug.
note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.51.0-nightly (0566609f2 2021-01-20) running on x86_64-unknown-linux-gnu


note: compiler flags: -Z threads=1 -Z ui-testing -Z deduplicate-diagnostics=no -Z emit-future-incompat-report -Z unstable-options -C prefer-dynamic -C rpath -C debuginfo=0
query stack during panic:
query stack during panic:
#0 [optimized_mir] optimizing MIR for `metadata_sufficient_for_layout::g::{closure#0}`
#1 [layout_raw] computing layout of `[generator@metadata_sufficient_for_layout::g::{closure#0} {()}]`
end of query stack


------------------------------------------

---

Some tests failed in compiletest suite=ui mode=ui host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--suite" "ui" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-9/bin/FileCheck" "--nodejs" "/usr/bin/node" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python3" "--lldb-python" "/usr/bin/python3" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "9.0.0" "--llvm-components" "aarch64 aarch64asmparser aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets amdgpu amdgpuasmparser amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitstreamreader bitwriter bpf bpfasmparser bpfcodegen bpfdesc bpfdisassembler bpfinfo codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfogsym debuginfomsf debuginfopdb demangle dlltooldriver engine executionengine fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader jitlink lanai lanaiasmparser lanaicodegen lanaidesc lanaidisassembler lanaiinfo libdriver lineeditor linker lto mc mca mcdisassembler mcjit mcparser mips mipsasmparser mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmparser msp430codegen msp430desc msp430disassembler msp430info native nativecodegen nvptx nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcjit passes perfjitevents powerpc powerpcasmparser powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata remarks riscv riscvasmparser riscvcodegen riscvdesc riscvdisassembler riscvinfo riscvutils runtimedyld scalaropts selectiondag sparc sparcasmparser sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target textapi transformutils vectorize webassembly webassemblyasmparser webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info x86utils xcore xcorecodegen xcoredesc xcoredisassembler xcoreinfo xray" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap --stage 2 test --exclude src/tools/tidy
Build completed unsuccessfully in 0:16:01

@oli-obk
Copy link
Contributor

oli-obk commented Jan 21, 2021

I don't have time to review before mid next week. @tmiasko do you want to take this one?

Copy link
Contributor

@tmiasko tmiasko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I really like that now we have a single function responsible for deciding what to encode that can be reused between prefetcher & encoder. Additionally with MIR being encoded separately, we could easily include its size in statistics.

compiler/rustc_metadata/src/rmeta/encoder.rs Outdated Show resolved Hide resolved
@cjgillot
Copy link
Contributor Author

The change in diagnostics is probably due to the iteration order in mir_keys. Should I sort the mir_keys before encoding, or is the regression acceptable?

@oli-obk
Copy link
Contributor

oli-obk commented Jan 21, 2021

If the iteration order were stable, then the exact order would be irrelevant. But I'm guessing this order depends on a hash map. We would get spurious changes in ui tests failing all the time if we are dependent on a hashmap for diagnostics ordering.

@cjgillot
Copy link
Contributor Author

Last commit sorts the hash map to avoid the unstable iteration in diagnostics.

@bors
Copy link
Contributor

bors commented Jan 24, 2021

☔ The latest upstream changes (presumably #80919) made this pull request unmergeable. Please resolve the merge conflicts.

@cjgillot
Copy link
Contributor Author

Rebased with the test back-and-forth removed.

@oli-obk
Copy link
Contributor

oli-obk commented Feb 3, 2021

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 3, 2021
@bors
Copy link
Contributor

bors commented Feb 3, 2021

⌛ Trying commit 5a60f0a with merge f7291779fd3275697027f6eefb933af7f154c92b...

@bors
Copy link
Contributor

bors commented Feb 3, 2021

☀️ Try build successful - checks-actions
Build commit: f7291779fd3275697027f6eefb933af7f154c92b (f7291779fd3275697027f6eefb933af7f154c92b)

@rust-timer
Copy link
Collaborator

Queued f7291779fd3275697027f6eefb933af7f154c92b with parent b593389, future comparison URL.

@rust-timer
Copy link
Collaborator

Finished benchmarking try commit (f7291779fd3275697027f6eefb933af7f154c92b): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 3, 2021
@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 3, 2021
@cjgillot
Copy link
Contributor Author

cjgillot commented Feb 3, 2021

Similar results.

@oli-obk
Copy link
Contributor

oli-obk commented Feb 4, 2021

Well... the CTFE stress test regression is completely gone. So that's good.

Now we're seeing a clap regression which has 90% more is_const_fn_raw invocations and 400% more mir_abstract_const invocations.

@oli-obk
Copy link
Contributor

oli-obk commented Feb 4, 2021

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 4, 2021
@bors
Copy link
Contributor

bors commented Feb 4, 2021

⌛ Trying commit 09ac459 with merge d015793ce5cbbc0e7555778a818058a8d558a7ab...

@bors
Copy link
Contributor

bors commented Feb 4, 2021

☀️ Try build successful - checks-actions
Build commit: d015793ce5cbbc0e7555778a818058a8d558a7ab (d015793ce5cbbc0e7555778a818058a8d558a7ab)

@rust-timer
Copy link
Collaborator

Queued d015793ce5cbbc0e7555778a818058a8d558a7ab with parent e708cbd, future comparison URL.

@rust-timer
Copy link
Collaborator

Finished benchmarking try commit (d015793ce5cbbc0e7555778a818058a8d558a7ab): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 4, 2021
@cjgillot
Copy link
Contributor Author

cjgillot commented Feb 4, 2021

The clap regression does not appear any more.

@tmiasko
Copy link
Contributor

tmiasko commented Feb 4, 2021

In wg-grammar some constructors are no longer queried in mir_for_ctfe & optimized_mir (don't know if this is a good thing or a bad thing):

2,90d1
< rust_grammar.parse-_C-Expr__0-{constructor#0}.-------.mir_map.0.mir
< rust_grammar.parse-_C-Expr__10__0-{constructor#0}.-------.mir_map.0.mir
< rust_grammar.parse-_C-Expr__10__1-{constructor#0}.-------.mir_map.0.mir
...
< rust_grammar.parse-_C-TOKEN_TREE-{constructor#0}.-------.mir_map.0.mir
17913,17997d17823
< rust_grammar.parse-ListHead-Nil-{constructor#0}.-------.mir_map.0.mir
< rust_grammar.parse-_P-_0-{constructor#0}.-------.mir_map.0.mir
...
< rust_grammar.parse-_P-_9-{constructor#0}.-------.mir_map.0.mir
18088d17913
< rust_grammar.parse-ParseError-NoParse-{constructor#0}.-------.mir_map.0.mir

@tmiasko
Copy link
Contributor

tmiasko commented Feb 4, 2021

It turns out we encoded MIR for fictive constructors previously.

@oli-obk
Copy link
Contributor

oli-obk commented Feb 5, 2021

Ok, I went over the leftover "regressions" (in the 0.x % range), and all of them seem to be because there's an additional call to mir_keys, which is actually below millisecond timing and thus only occurs on perf tests that themselves just take a few milliseconds.

@bors r+

@bors
Copy link
Contributor

bors commented Feb 5, 2021

📌 Commit 09ac459 has been approved by oli-obk

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 5, 2021
@bors
Copy link
Contributor

bors commented Feb 5, 2021

⌛ Testing commit 09ac459 with merge 23adf9f...

@bors
Copy link
Contributor

bors commented Feb 5, 2021

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 23adf9f to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Feb 5, 2021
@bors bors merged commit 23adf9f into rust-lang:master Feb 5, 2021
@rustbot rustbot added this to the 1.51.0 milestone Feb 5, 2021
@cjgillot cjgillot deleted the defkey-mir branch February 5, 2021 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants