incr.comp.: Switch to red/green change tracking, remove legacy system. #44901

michaelwoerister · 2017-09-28T14:48:02Z

This PR finally switches incremental compilation to red/green tracking and completely removes the legacy dependency graph implementation -- which includes a few quite costly passes that are simply not needed with the new system anymore.

There's still some documentation to be done and there's certainly still lots of optimizing and tuning ahead -- but the foundation for red/green is in place with this PR. This has been in the making for a long time :)

r? @nikomatsakis
cc @alexcrichton, @rust-lang/compiler

alexcrichton · 2017-09-28T14:53:36Z

src/librustc_trans/back/symbol_export.rs

@@ -79,9 +79,7 @@ pub fn provide_local(providers: &mut Providers) {
    providers.is_exported_symbol = |tcx, id| {
        // FIXME(#42293) needs red/green to not break a bunch of incremental
        // tests
-        tcx.dep_graph.with_ignore(|| {


:D

(you can probably delete the FIXME above this as well)

alexcrichton · 2017-09-28T14:54:15Z

🎉 🎉 🎉 🎉 🎉 🎉 🌮 🎉 🎉 🎉 🎉 🎉 🎉 🎉

kennytm · 2017-09-28T16:21:26Z

src/librustc/ty/maps/plumbing.rs

+
+                profq_msg!(tcx,
+                    ProfileQueriesMsg::QueryBegin(
+                        span.clone(),


Should be span.data(). Why needs the clone()?

[00:06:30] error[E0308]: mismatched types [00:06:30] --> /checkout/src/librustc/ty/maps/plumbing.rs:654:25 [00:06:30] | [00:06:30] 654 | span.clone(), [00:06:30] | ^^^^^^^^^^^^ expected struct `syntax_pos::SpanData`, found struct `syntax_pos::Span` [00:06:30] ... [00:06:30] 710 | DepKind::RegionScopeTree => { force!(region_scope_tree, def_id!()); } [00:06:30] | ------------------------------------- in this macro invocation [00:06:30] | [00:06:30] = note: expected type `syntax_pos::SpanData` [00:06:30] found type `syntax_pos::Span` [00:06:30] = help: here are some functions which might fulfill your needs: [00:06:30] - .data() ... [00:06:32] error: aborting due to 110 previous errors [00:06:32] [00:06:32] error: Could not compile `rustc`.

Looks like this needs a rebase...

sgrif · 2017-09-28T21:11:41Z

Speaking of red/green

that's a lot of both. ;)

Mark-Simulacrum · 2017-09-29T02:16:34Z

Let me know if you want to check perf (try build + ping me).

michaelwoerister · 2017-09-29T09:19:07Z

@Mark-Simulacrum Awesome, thank you!

michaelwoerister · 2017-09-29T12:13:11Z

@bors try

bors · 2017-09-29T12:13:21Z

⌛ Trying commit 762b735 with merge b669ce1...

@nikomatsakis

incr.comp.: Switch to red/green change tracking, remove legacy system. This PR finally switches incremental compilation to [red/green tracking](#42293) and completely removes the legacy dependency graph implementation -- which includes a few quite costly passes that are simply not needed with the new system anymore. There's still some documentation to be done and there's certainly still lots of optimizing and tuning ahead -- but the foundation for red/green is in place with this PR. This has been in the making for a long time `:)` r? @nikomatsakis cc @alexcrichton, @rust-lang/compiler

bors · 2017-09-29T13:39:52Z

☀️ Test successful - status-travis
State: approved= try=True

Mark-Simulacrum · 2017-09-29T17:48:40Z

Looks like wins across the board:

http://perf.rust-lang.org/compare.html?commit_a=0253d98382fa351d53880605a346d0ea2e941a07&commit_b=b669ce1863e5451453545f892d1014eb16b9a878&stat=instructions%3Au

michaelwoerister · 2017-09-29T20:53:28Z

Thanks, @Mark-Simulacrum! The link does not seem to work for me though at the moment (I see no results listed).

Mark-Simulacrum · 2017-09-29T22:21:10Z

Should work now. I think perf.rlo was just not updating for some reason...

michaelwoerister · 2017-10-02T09:28:07Z

Thanks, @Mark-Simulacrum! Looks promising.

…ind out their color.

nikomatsakis · 2017-10-03T09:34:26Z

src/librustc/dep_graph/graph.rs

+                DepNodeColor::Red
+            };
+
+            assert!(data.colors.borrow_mut().insert(key, color).is_none());


Nit: Although I am often tempted to write an assert! like this, I tend to think that putting crucial side-effect operations inside of an assert! is bad-style. Too easy to overlook. I think I'd prefer to see something like:

let old_value = data.colors.borrow_mut().insert(key, color); assert!(old_value.is_none(), "key {:?} executed twice", key);

nikomatsakis

r=me with comments and nits addressed

nikomatsakis · 2017-10-03T09:48:05Z

src/librustc/ty/maps/plumbing.rs

+                                                                        f)
+                    }
+
+                    if let Some(dep_node_index) = tcx.dep_graph.try_mark_green(tcx, &dep_node) {


I'm curious, why not make try_mark_green check the cache, so we can collapse these two ifs into one? They seem sort of like fundamentally the same case (marked green already vs can mark green now)

Wouldn't that mean that we always load the result from the cache, even if it is never asked for?

nikomatsakis · 2017-10-03T09:54:23Z

src/librustc/ty/maps/plumbing.rs

+                debug_assert!(tcx.dep_graph.is_green(dep_node_index));
+
+                // We don't do any caching yet, so recompute
+                let (result, diagnostics) = tcx.cycle_check(span, Query::$name(key), || {


Nit: feels like this maybe wants a subroutine, kind of duplicates the code above.

this happens later, I saw, sorry :)

nikomatsakis · 2017-10-03T09:58:08Z

src/librustc/ty/maps/plumbing.rs

+    // We should never get into the situation of having to force this from the DepNode.
+    // Since we cannot reconstruct the query key, we would always end up having to evaluate
+    // the first caller of this query that *is* reconstructible. This might very well be
+    // compile_codegen_unit() in which case we'd lose all re-use.


I find this comment pretty unclear. What is "this", in the first sentence, referring to? What makes CodegenUnit worse or different from other kinds of depnodes?

Hm, yeah, I seem to have moved this comment out of context without updating it. CodegenUnit is different because there's no way to reconstruct the query-key from the DepNode, so the "parent query" would have to be re-done. That parent query is compile_codegen_unit(), so we would always have to recompile the codegen unit just to find out that it would not have been necessary.

I'll update the comment and try to make it clearer.

nikomatsakis · 2017-10-03T10:00:54Z

src/librustc/ty/maps/plumbing.rs

+    match dep_node.kind {
+        // These are inputs that are expected to be pre-allocated and that
+        // should therefore always be red or green already
+        DepKind::AllLocalTraitImpls |


This match feels mildly unfortunate -- i.e., it feels like we ought to be able to derive this information from other macros and not require it to be "re-entered" here. But oh well, I haven't thought about how to actually do that. Might be nice to file a FIXME issue about it though.

Yeah, it's definitely not pretty. But I didn't want spend too much time on making this macro generated before we have a proof of concept implementation of red/green. I'll add a FIXME.

nikomatsakis · 2017-10-03T10:13:54Z

src/librustc/dep_graph/graph.rs

@@ -644,15 +664,26 @@ pub(super) struct CurrentDepGraph {
    edges: IndexVec<DepNodeIndexNew, Vec<DepNodeIndexNew>>,
    node_to_node_index: FxHashMap<DepNode, DepNodeIndexNew>,

+    anon_id_seed: Fingerprint,


this feels like it merits a nice comment

Agreed :)

michaelwoerister · 2017-10-03T12:05:46Z

Thanks for the review, @nikomatsakis!

michaelwoerister · 2017-10-04T10:38:12Z

@bors r=nikomatsakis

bors · 2017-10-04T10:38:13Z

📌 Commit 0454a41 has been approved by nikomatsakis

bors · 2017-10-04T19:14:48Z

⌛ Testing commit 0454a41 with merge d7e73e4...

@nikomatsakis

incr.comp.: Switch to red/green change tracking, remove legacy system. This PR finally switches incremental compilation to [red/green tracking](#42293) and completely removes the legacy dependency graph implementation -- which includes a few quite costly passes that are simply not needed with the new system anymore. There's still some documentation to be done and there's certainly still lots of optimizing and tuning ahead -- but the foundation for red/green is in place with this PR. This has been in the making for a long time `:)` r? @nikomatsakis cc @alexcrichton, @rust-lang/compiler

bors · 2017-10-04T23:09:27Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nikomatsakis
Pushing d7e73e4 to master...

alexcrichton · 2017-10-05T20:00:32Z

@michaelwoerister

I think we've recovered on all perf benchmarks except two. Everything else either improved since the original regression or was within 5% that I figured it would come out in the wash.

Interestingly I think those two benchmarks are the "from scratch build up incremental first time" benchmark, right?

michaelwoerister · 2017-10-06T08:03:16Z

perf.rlo is showing me only flat lines right now :/

I think I'll write up a tracking issue for possible performance improvements.

…-recursion, r=eddyb incr.comp.: Fix infinite recursion in Debug implementation of DepNode Small bug fix. Depends on rust-lang#44901 to land first.

rust-highfive assigned nikomatsakis Sep 28, 2017

alexcrichton reviewed Sep 28, 2017

View reviewed changes

kennytm reviewed Sep 28, 2017

View reviewed changes

alexcrichton added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 28, 2017

michaelwoerister force-pushed the on-demand-eval branch from a1d7f6c to b88c694 Compare September 29, 2017 09:13

michaelwoerister force-pushed the on-demand-eval branch from b88c694 to 762b735 Compare September 29, 2017 10:37

michaelwoerister mentioned this pull request Sep 29, 2017

Tracking Issue for incr. comp. red/green testing #44716

Closed

3 tasks

michaelwoerister mentioned this pull request Sep 29, 2017

incr.comp.: Update fingerprint-based auto tests for red/green tracking. #44924

Closed

27 tasks

michaelwoerister mentioned this pull request Oct 2, 2017

rustc: Implement ThinLTO #44841

Merged

michaelwoerister added 9 commits October 2, 2017 15:36

incr.comp.: Determine red/green state of every new node.

a948be8

incr.comp.: Add minimal version of try_mark_green procedure.

6db27d9

incr.comp.: Re-execute queries during red/green marking in order to f…

f0bbf4e

…ind out their color.

incr.comp.: Make some DepNodes non-anonymous.

e1994bd

incr.comp.: Add some logging to DepGraph::try_mark_green().

c9a17ef

incr.comp.: Use red/green tracking for CGU re-use.

e6badfd

incr.comp.: Build DepGraphQuery from new dep-graph impl.

272c324

incr.comp.: Remove saving and loading of legacy dep-graph.

171c020

incr.comp.: Remove legacy dep-graph runtime.

df06a7e

incr.comp.: Do some cleanup.

dbab705

michaelwoerister force-pushed the on-demand-eval branch from 762b735 to 563d30e Compare October 2, 2017 14:22

nikomatsakis reviewed Oct 3, 2017

View reviewed changes

nikomatsakis approved these changes Oct 3, 2017

View reviewed changes

incr.comp.: Fix some merge fallout.

c96d0bf

michaelwoerister mentioned this pull request Oct 4, 2017

incr.comp.: Implement force_from_dep_node() in a macro. #45015

Closed

incr.comp.: Address review comments.

0454a41

michaelwoerister force-pushed the on-demand-eval branch from 563d30e to 0454a41 Compare October 4, 2017 10:36

michaelwoerister mentioned this pull request Oct 4, 2017

incr.comp.: Fix infinite recursion in Debug implementation of DepNode #45018

Merged

bors merged commit 0454a41 into rust-lang:master Oct 4, 2017

michaelwoerister mentioned this pull request Oct 23, 2017

incr.comp.: Move Metadata-hash loading to DepGraph. #44702

Closed

incr.comp.: Switch to red/green change tracking, remove legacy system. #44901

incr.comp.: Switch to red/green change tracking, remove legacy system. #44901

Conversation

michaelwoerister commented Sep 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Sep 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgrif commented Sep 28, 2017

Mark-Simulacrum commented Sep 29, 2017

michaelwoerister commented Sep 29, 2017

michaelwoerister commented Sep 29, 2017

bors commented Sep 29, 2017

bors commented Sep 29, 2017

Mark-Simulacrum commented Sep 29, 2017

michaelwoerister commented Sep 29, 2017

Mark-Simulacrum commented Sep 29, 2017

michaelwoerister commented Oct 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelwoerister commented Oct 3, 2017

michaelwoerister commented Oct 4, 2017

bors commented Oct 4, 2017

bors commented Oct 4, 2017

bors commented Oct 4, 2017

alexcrichton commented Oct 5, 2017

michaelwoerister commented Oct 6, 2017