Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustbuild: Fail the build if we build Cargo twice #49053

Merged
merged 1 commit into from
Mar 27, 2018

Conversation

alexcrichton
Copy link
Member

This commit updates the ToolBuild step to stream Cargo's JSON messages, parse
them, and record all libraries built. If we build anything twice (aka Cargo)
it'll most likely happen due to dependencies being recompiled which is caught by
this check.

@rust-highfive
Copy link
Collaborator

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 15, 2018
@alexcrichton
Copy link
Member Author

r? @kennytm or @Mark-Simulacrum

Note that this currently fails to build because Cargo's getting compiled twice on CI, but I hope to fix that with rust-lang/cargo#5188 by the time this is approved and otherwise ready to land.

@kennytm kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 15, 2018
@alexcrichton
Copy link
Member Author

With this PR the build currently fails with:

$ ./x.py build src/tools/{rls,cargo}
...
duplicate artfacts found when compiling a tool, this typically means that something was recompiled because a transitive dependency has different features activated than in a previous build:

  num-traits 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)
    `cargo` enabled features [] at "libnum_traits-58f093f238cd6fd3.rlib"
    `rls` enabled features ["default", "std"] at "libnum_traits-3b34cffa1e11f667.rlib"
  serde_json 1.0.10 (registry+https://github.com/rust-lang/crates.io-index)
    `cargo` enabled features ["default"] at "libserde_json-f3e37e0dddd3eac2.rlib"
    `rls` enabled features ["default"] at "libserde_json-14e1634b89a3f35d.rlib"

thread 'main' panicked at 'tools should not compile multiple copies of the same crate', bootstrap/tool.rs:168:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /home/alex/code/rust4/build/bootstrap/debug/bootstrap build src/tools/rls src/tools/cargo
Build completed unsuccessfully in 0:00:03

@@ -253,6 +253,10 @@ pub struct Build {
ci_env: CiEnv,
delayed_failures: RefCell<Vec<String>>,
prerelease_version: Cell<Option<u32>>,
tool_artifacts: RefCell<HashMap<
Interned<String>,
HashMap<String, (&'static str, PathBuf, Vec<String>)>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please either document these fields here or make this a struct instead of a tuple.

@Mark-Simulacrum
Copy link
Member

So it's not 100% clear to me why this is only done in ToolBuild. I'd imagine we should avoid recompiling crates across rust because of features, though perhaps we'll have to limit that to out-of-tree crates. Does that expansion seem reasonable?

@alexcrichton
Copy link
Member Author

A good point! I think, though, that we're covered in the other locations for now. The other three locations, std/test/rustc, all are self-protecting effectively against this sort of error. For both std/test we don't build more than one project in those directories so we wouldn't be able to detect duplicates (and not sure what that would mean in that context).

For rustc, though, we build both rustc and the trans backends. The good news here though is that everything crashes and burns in stage1 if the trans backends accidentally recompile rustc. This has already caused a PR to two to bounce on CI. In that sense we don't actually have to verify because by construction the compiler will fail if they don't share dependencies.

So in that sense I think that all we need to check here is the tools themselves, everything else should already be taken care of.

@Mark-Simulacrum
Copy link
Member

Sounds good. Pending a cargo update r=me I think...

@matklad
Copy link
Member

matklad commented Mar 16, 2018

PR with Cargo update: #48986 (comment)

@alexcrichton alexcrichton force-pushed the fail-if-build-cargo-twice branch from f1a677a to 7acaf46 Compare March 16, 2018 14:21
@alexcrichton
Copy link
Member Author

I think this PR is now green but it breaks the RLS (see rust-lang/rls#767). I'll also hold off on merging at least until @matklad's regression is fixed

@bors
Copy link
Contributor

bors commented Mar 16, 2018

☔ The latest upstream changes (presumably #48097) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton force-pushed the fail-if-build-cargo-twice branch from 7acaf46 to 90f1568 Compare March 17, 2018 17:58
@alexcrichton
Copy link
Member Author

@bors: r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Mar 17, 2018

📌 Commit 90f1568 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Mar 17, 2018
@kennytm
Copy link
Member

kennytm commented Mar 17, 2018

@bors r-

Needs to fix error-index-generator vs rustbook. The error comes from compiling error-index-generator at stage0 and rustbook at stage2.

[00:39:50]    Compiling error_index_generator v0.0.0 (file:///checkout/src/tools/error_index_generator)
[00:39:52]     Finished release [optimized] target(s) in 2.80 secs
[00:39:53] duplicate artfacts found when compiling a tool, this typically means that something was recompiled because a transitive dependency has different features activated than in a previous build:
[00:39:53] 
[00:39:53]   bitflags 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features ["default", "example_generated"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libbitflags-3f1a1fadb9b1a8ec.rlib"
[00:39:53]     `rustbook` enabled features ["default", "example_generated"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libbitflags-8b5679a6ec267b49.rlib"
[00:39:53]   winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libwinapi-6efc14177cb3de4a.rlib"
[00:39:53]     `rustbook` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libwinapi-2332d26f2d360689.rlib"
[00:39:53]   libc 0.2.39 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features ["default", "use_std"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/liblibc-fff2bd13c38c95a8.rlib"
[00:39:53]     `rustbook` enabled features ["default", "use_std"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/liblibc-170c926795128861.rlib"
[00:39:53]   rand 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features ["default", "libc", "std"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/librand-4291d425b15909cc.rlib"
[00:39:53]     `rustbook` enabled features ["default", "libc", "std"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/librand-705da6ff1eaaba3c.rlib"
[00:39:53]   pulldown-cmark 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libpulldown_cmark-b06a147dfe86b49e.rlib"
[00:39:53]     `rustbook` enabled features ["default", "getopts"] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libpulldown_cmark-7bdb88b3c356cbc9.rlib"
[00:39:53]   kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libkernel32-098f6019eeaaaf39.rlib"
[00:39:53]     `rustbook` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libkernel32-4a53592702dd8140.rlib"
[00:39:53]   remove_dir_all 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libremove_dir_all-4f18feda118fd3e9.rlib"
[00:39:53] thread 'main' panicked at 'tools should not compile multiple copies of the same crate', bootstrap/tool.rs:188:13
[00:39:53]     `rustbook` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libremove_dir_all-7077b3afe6766cb9.rlib"
[00:39:53]   tempdir 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)
[00:39:53]     `error_index_generator` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps/libtempdir-89fd2625a654b5c7.rlib"
[00:39:53]     `rustbook` enabled features [] at "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libtempdir-68fa061d83ba4e87.rlib"
[00:39:53] 
[00:39:53] note: Run with `RUST_BACKTRACE=1` for a backtrace.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 17, 2018
@bors
Copy link
Contributor

bors commented Mar 18, 2018

☔ The latest upstream changes (presumably #48986) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton force-pushed the fail-if-build-cargo-twice branch from 90f1568 to 18e7686 Compare March 18, 2018 15:47
bors added a commit to rust-lang/cargo that referenced this pull request Mar 26, 2018
More effort to only compile Cargo once

Hopefully one final change necessary for rust-lang/rust#49053
@alexcrichton alexcrichton force-pushed the fail-if-build-cargo-twice branch from 661a2bc to e9a02ee Compare March 26, 2018 07:29
@alexcrichton
Copy link
Member Author

@bors: r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Mar 26, 2018

📌 Commit e9a02ee has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 26, 2018
@bors
Copy link
Contributor

bors commented Mar 26, 2018

⌛ Testing commit e9a02ee6fbb59a4039644cb91c350195607b8459 with merge 5d5f9f67aa036160dfa6957ae16ee5ac1ebb23ca...

@bors
Copy link
Contributor

bors commented Mar 26, 2018

💔 Test failed - status-travis

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 26, 2018
@kennytm kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 26, 2018
@kennytm
Copy link
Member

kennytm commented Mar 26, 2018

Updating cargo broke RLS again? 🤔

[01:21:17]    Compiling json v0.11.12
[01:21:17]    Compiling rls v0.126.0 (file:///checkout/src/tools/rls)
[01:21:23] error[E0277]: the `?` operator can only be applied to values that implement `std::ops::Try`
[01:21:23]    --> tools/rls/src/build/plan.rs:135:21
[01:21:23]     |
[01:21:23] 135 |         let units = cx.dep_targets(unit)?;
[01:21:23]     |                     ^^^^^^^^^^^^^^^^^^^^^ the `?` operator cannot be applied to type `std::vec::Vec<cargo::ops::Unit<'_>>`
[01:21:23]     |
[01:21:23]     = help: the trait `std::ops::Try` is not implemented for `std::vec::Vec<cargo::ops::Unit<'_>>`
[01:21:23]     = note: required by `std::ops::Try::into_result`
[01:21:23]
[01:21:24] [RUSTC-TIMING] json test:false 6.339
[01:21:24] error: aborting due to previous error
[01:21:24]
[01:21:24] For more information about this error, try `rustc --explain E0277`.
[01:21:24] [RUSTC-TIMING] rls test:false 5.565
[01:21:24] error: Could not compile `rls`.
[01:21:24] warning: build failed, waiting for other jobs to finish...
[01:21:29] error[E0277]: the `?` operator can only be applied to values that implement `std::ops::Try`
[01:21:29]    --> tools/rls/src/build/plan.rs:135:21
[01:21:29]     |
[01:21:29] 135 |         let units = cx.dep_targets(unit)?;
[01:21:29]     |                     ^^^^^^^^^^^^^^^^^^^^^ the `?` operator cannot be applied to type `std::vec::Vec<cargo::ops::Unit<'_>>`
[01:21:29]     |
[01:21:29]     = help: the trait `std::ops::Try` is not implemented for `std::vec::Vec<cargo::ops::Unit<'_>>`
[01:21:29]     = note: required by `std::ops::Try::into_result`
[01:21:29]
[01:21:30] [RUSTC-TIMING] tests test:true 6.132
[01:21:30] error: aborting due to previous error

@bors
Copy link
Contributor

bors commented Mar 26, 2018

☔ The latest upstream changes (presumably #49255) made this pull request unmergeable. Please resolve the merge conflicts.

This commit updates the `ToolBuild` step to stream Cargo's JSON messages, parse
them, and record all libraries built. If we build anything twice (aka Cargo)
it'll most likely happen due to dependencies being recompiled which is caught by
this check.
@alexcrichton alexcrichton force-pushed the fail-if-build-cargo-twice branch from e9a02ee to faebcc1 Compare March 26, 2018 20:07
@alexcrichton
Copy link
Member Author

@bors: r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Mar 26, 2018

📌 Commit faebcc1 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 26, 2018
@bors
Copy link
Contributor

bors commented Mar 26, 2018

⌛ Testing commit faebcc1 with merge 989b257...

bors added a commit that referenced this pull request Mar 26, 2018
…-Simulacrum

rustbuild: Fail the build if we build Cargo twice

This commit updates the `ToolBuild` step to stream Cargo's JSON messages, parse
them, and record all libraries built. If we build anything twice (aka Cargo)
it'll most likely happen due to dependencies being recompiled which is caught by
this check.
@bors
Copy link
Contributor

bors commented Mar 27, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: Mark-Simulacrum
Pushing 989b257 to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants