Enable reuse of `.o` files if nothing has changed #34956

nikomatsakis · 2016-07-21T16:59:46Z

This PR completes a first "spike" for incremental compilation by enabling us to reuse .o files when nothing has changed. When in incr. mode, we will save .o files into the temporary directory, then copy them back out again if they are still valid. The code is still a bit rough but it does seem to work. =)

r? @michaelwoerister

Fixes #34036
Fixes #34037
Fixes #34038

michaelwoerister · 2016-07-21T17:35:38Z

Excellent :) I'll take a look at it first thing tomorrow.

jonas-schievink · 2016-07-21T20:09:38Z

I just tried to build glium with this and I'm hitting multiple ICEs about local/foreign items:

https://gist.github.com/jonas-schievink/536a1ad7aa065193e4954242c9885cbd

(I've set RUSTFLAGS=-Zincremental=target/INCREMENTAL before running cargo)

michaelwoerister · 2016-07-22T07:55:50Z

src/librustc/dep_graph/dep_node.rs

+/// them even in the absence of a tcx.)
+#[derive(Clone, Debug, PartialEq, Eq, Hash, RustcEncodable, RustcDecodable)]
+pub enum WorkProductId {
+    PartitionObjectFile(String), // see (*TransPartition) below


Nit: Our nomenclature around "partitions", "codegen units", "translation units", etc is a bit inconsistent. We are using these names more or less interchangeably and it would probably be good to decide on the canonical term.

michaelwoerister · 2016-07-22T10:16:33Z

OK, I've done a first pass over this. It's so nice to see all the work from the last 10 months finally coming together! (I mean, not that we are anywhere near being done yet, of course :))

The PR looks very good. The only thing I'm not so sure about is the factoring of the WorkProduct functionality. It seems a bit strange to have the partitioning-hashes in there, something which will only be used for object files (and maybe .bc files), but not for something like MIR. At the same time the WorkProductId does not scale to things with a DefId. It seems to me that there is room for improvement. However, I won't block the PR on this.

Another thing that isn't quite clear to me is the relationship between the "temp" directory and the "incr. comp.". Are they the same when incr. comp. is enabled? Or do we copy back and forth between them? I guess, we'll clarify these questions when we tackle #34957, #34955, and #32754.

nikomatsakis · 2016-07-22T14:28:13Z

The only thing I'm not so sure about is the factoring of the WorkProduct functionality. It seems a bit strange to have the partitioning-hashes in there, something which will only be used for object files (and maybe .bc files), but not for something like MIR. At the same time the WorkProductId does not scale to things with a DefId. It seems to me that there is room for improvement.

As I said on IRC, I also felt like the WorkProduct thing wasn't as general as I originally thought it would be. I suspect it might be better to just rip it out and change all that stuff to be specific to object files -- though I guess what is there could work just as well for bc files as it does for o files, so it's at least general enough for that.

(OTOH, it won't work for MIR, since we'd want something with a def-id as key for that; they also probably don't need the auxiliary hash info.)

I'm inclined to leave it now but just expect it to evolve, but that's mostly because I'm lazy. I'd also be happy to rip it out.

michaelwoerister · 2016-07-22T15:02:39Z

I'm inclined to leave it now but just expect it to evolve, but that's mostly because I'm lazy. I'd also be happy to rip it out.

I'm fine with leaving it as it is in the PR for now. If we had a clear vision of what it ought to look like, that would be different.

bors · 2016-07-22T17:26:44Z

☔ The latest upstream changes (presumably #34917) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · 2016-07-22T20:11:12Z

@bors r=mw

bors · 2016-07-22T20:11:13Z

📌 Commit 5c56fca has been approved by mw

bors · 2016-07-23T22:55:26Z

⌛ Testing commit 5c56fca with merge f62c12b...

bors · 2016-07-24T01:12:25Z

💔 Test failed - auto-linux-musl-64-opt

nikomatsakis · 2016-07-25T22:40:19Z

@michaelwoerister care to review the final commit?

bors · 2016-07-26T12:23:15Z

💔 Test failed - auto-win-msvc-64-opt-rustbuild

alexcrichton · 2016-07-26T17:21:44Z

@bors: retry

On Tue, Jul 26, 2016 at 5:23 AM, bors notifications@github.com wrote:

💔 Test failed - auto-win-msvc-64-opt-rustbuild
https://buildbot.rust-lang.org/builders/auto-win-msvc-64-opt-rustbuild/builds/1979

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#34956 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD95JZkQsxuMpO50qTI2ocLXsY7R55Nks5qZfw2gaJpZM4JSAbm
.

bors · 2016-07-28T00:38:26Z

☔ The latest upstream changes (presumably #34907) made this pull request unmergeable. Please resolve the merge conflicts.

A work product right now is just a `.o` file. In the future it probably includes other kinds of files, such as `.bc` files saving the unoptimized LLVM IR. However, because WorkProductIds must be independent of DefIds, so that they don't need translation, this system may not be suitable *as is* for storing fine-grained information (such as the MIR for individual defs), as it was originally intended. We will want to refactor some for that.

We used to use `Name`, but the session outlives the tokenizer, which means that attempts to read this field after trans has complete otherwise panic. All reads want an `InternedString` anyhow.

Work products are deleted if any of their inputs are dirty.

This checks the `previous_work_products` data from the dep-graph and tries to simply copy a `.o` file if possible. We also add new work-products into the dep-graph, and create edges to/from the dep-node for a work-product.

In the older version, a `.o` and ` .bc` file were separate work-products. This newer version keeps, for each codegen-unit, a set of files of different kinds. We assume that if any kinds are available then all the kinds we need are available, since the precise set of switches will depend on attributes and command-line switches. Should probably test this: the effect of changing attributes in particular might not be successfully tracked?

nikomatsakis · 2016-07-28T16:33:41Z

@bors r=mw

bors · 2016-07-28T16:33:42Z

📌 Commit 2f9fff2 has been approved by mw

nikomatsakis · 2016-07-28T18:10:25Z

@bors r-

seeing make check errors locally :(

actually we shouldn't even hash nested items at all, but that is addressed in a followup PR

nikomatsakis · 2016-07-28T19:43:56Z

@bors r=mw

bors · 2016-07-28T19:43:57Z

📌 Commit 42cd5d4 has been approved by mw

bors · 2016-07-28T21:17:55Z

⌛ Testing commit 42cd5d4 with merge 54c0dcf...

@michaelwoerister

Enable reuse of `.o` files if nothing has changed This PR completes a first "spike" for incremental compilation by enabling us to reuse `.o` files when nothing has changed. When in incr. mode, we will save `.o` files into the temporary directory, then copy them back out again if they are still valid. The code is still a bit rough but it does seem to work. =) r? @michaelwoerister Fixes #34036 Fixes #34037 Fixes #34038

bors · 2016-07-29T00:17:01Z

WiSaGaN · 2016-08-05T08:18:53Z

Will this reduce the time of cargo check if only one of several files changes?

nikomatsakis · 2016-08-10T21:29:45Z

@WiSaGaN it should, yes, but we are still fixing some known shortcomings there.

rust-highfive assigned michaelwoerister Jul 21, 2016

nikomatsakis mentioned this pull request Jul 21, 2016

Avoid modifying incr-compilation directory unless compilation completes normally #34957

Closed

michaelwoerister reviewed Jul 22, 2016
View reviewed changes

nikomatsakis force-pushed the incr-comp-o-files branch from 3293bac to 5c56fca Compare July 22, 2016 20:11

nikomatsakis mentioned this pull request Jul 23, 2016

ICEs with incremental compilation building glium #34991

Closed

nikomatsakis added 7 commits July 28, 2016 12:05

Store crate_disambiguator as an InternedString

ffc13b2

We used to use `Name`, but the session outlives the tokenizer, which means that attempts to read this field after trans has complete otherwise panic. All reads want an `InternedString` anyhow.

Code to save/load the work-products map from disk

cca4804

Work products are deleted if any of their inputs are dirty.

Modify trans to skip generating .o files

58d4b8e

This checks the `previous_work_products` data from the dep-graph and tries to simply copy a `.o` file if possible. We also add new work-products into the dep-graph, and create edges to/from the dep-node for a work-product.

Add a testing mechanism and a simple spike test

08a72d2

Address mw nits

ceeb158

nikomatsakis force-pushed the incr-comp-o-files branch from ea13edb to 2f9fff2 Compare July 28, 2016 16:31

nikomatsakis added 2 commits July 28, 2016 15:39

hash def-path's better

e359147

actually we shouldn't even hash nested items at all, but that is addressed in a followup PR

make it possible to track where hash diverges

42cd5d4

bors merged commit 42cd5d4 into rust-lang:master Jul 29, 2016

This was referenced Jul 29, 2016

Various improvements to the SVH #35079

Merged

intravisit: Fold functionality of IdVisitor into the regular Visitor. #35090

Merged

LLVM upgrade #34743

Merged

steveklabnik mentioned this pull request Aug 18, 2016

Incremental recompilation #2369

Closed

nikomatsakis deleted the incr-comp-o-files branch October 3, 2016 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable reuse of `.o` files if nothing has changed #34956

Enable reuse of `.o` files if nothing has changed #34956

nikomatsakis commented Jul 21, 2016 •

edited

Loading

michaelwoerister commented Jul 21, 2016

jonas-schievink commented Jul 21, 2016

michaelwoerister Jul 22, 2016

michaelwoerister commented Jul 22, 2016

nikomatsakis commented Jul 22, 2016

michaelwoerister commented Jul 22, 2016

bors commented Jul 22, 2016

nikomatsakis commented Jul 22, 2016

bors commented Jul 22, 2016

bors commented Jul 23, 2016

bors commented Jul 24, 2016

nikomatsakis commented Jul 25, 2016

bors commented Jul 26, 2016

alexcrichton commented Jul 26, 2016

bors commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

bors commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

bors commented Jul 28, 2016

bors commented Jul 28, 2016

bors commented Jul 29, 2016

WiSaGaN commented Aug 5, 2016

nikomatsakis commented Aug 10, 2016

Enable reuse of .o files if nothing has changed #34956

Enable reuse of .o files if nothing has changed #34956

Conversation

nikomatsakis commented Jul 21, 2016 • edited Loading

michaelwoerister commented Jul 21, 2016

jonas-schievink commented Jul 21, 2016

michaelwoerister Jul 22, 2016

Choose a reason for hiding this comment

michaelwoerister commented Jul 22, 2016

nikomatsakis commented Jul 22, 2016

michaelwoerister commented Jul 22, 2016

bors commented Jul 22, 2016

nikomatsakis commented Jul 22, 2016

bors commented Jul 22, 2016

bors commented Jul 23, 2016

bors commented Jul 24, 2016

nikomatsakis commented Jul 25, 2016

bors commented Jul 26, 2016

alexcrichton commented Jul 26, 2016

bors commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

bors commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

nikomatsakis commented Jul 28, 2016

bors commented Jul 28, 2016

bors commented Jul 28, 2016

bors commented Jul 29, 2016

WiSaGaN commented Aug 5, 2016

nikomatsakis commented Aug 10, 2016

Enable reuse of `.o` files if nothing has changed #34956

Enable reuse of `.o` files if nothing has changed #34956

nikomatsakis commented Jul 21, 2016 •

edited

Loading