Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short circuit Send and Sync impls for TokenTree #60967

Merged
merged 1 commit into from
May 27, 2019

Conversation

Zoxc
Copy link
Contributor

@Zoxc Zoxc commented May 19, 2019

Workaround to make the parallel compiler build after #60444.

r? @nikomatsakis

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 19, 2019
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:2a7cc96e:start=1558293500505906466,finish=1558293501296013601,duration=790107135
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:07:29]    Compiling arena v0.0.0 (/checkout/src/libarena)
[00:07:41]    Compiling rustc_macros v0.1.0 (/checkout/src/librustc_macros)
[00:07:52]    Compiling syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[00:07:56]    Compiling rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:08:06] error[E0277]: `syntax_pos::span_encoding::Span` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `syntax_pos::span_encoding::Span` cannot be shared between threads safely
[00:08:06]    = help: the trait `std::marker::Sync` is not implemented for `syntax_pos::span_encoding::Span`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `syntax_pos::span_encoding::Span` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `syntax_pos::span_encoding::Span` cannot be sent between threads safely
[00:08:06]    = help: the trait `std::marker::Send` is not implemented for `syntax_pos::span_encoding::Span`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<parse::token::Nonterminal>` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<parse::token::Nonterminal>` cannot be shared between threads safely
[00:08:06]    |
[00:08:06]    = help: within `parse::token::Token`, the trait `std::marker::Sync` is not implemented for `std::rc::Rc<parse::token::Nonterminal>`
[00:08:06]    = note: required because it appears within the type `parse::token::Token`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<parse::token::Nonterminal>` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<parse::token::Nonterminal>` cannot be sent between threads safely
[00:08:06]    |
[00:08:06]    = help: within `parse::token::Token`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<parse::token::Nonterminal>`
[00:08:06]    = note: required because it appears within the type `parse::token::Token`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be shared between threads safely
[00:08:06]    |
[00:08:06]    = help: within `tokenstream::TokenStream`, the trait `std::marker::Sync` is not implemented for `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>`
[00:08:06]    = note: required because it appears within the type `std::option::Option<std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>>`
[00:08:06]    = note: required because it appears within the type `tokenstream::TokenStream`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be sent between threads safely
[00:08:06]    |
[00:08:06]    = help: within `tokenstream::TokenStream`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>`
[00:08:06]    = note: required because it appears within the type `std::option::Option<std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>>`
[00:08:06]    = note: required because it appears within the type `tokenstream::TokenStream`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] error: aborting due to 6 previous errors
[00:08:06] 
[00:08:06] For more information about this error, try `rustc --explain E0277`.
[00:08:06] error: Could not compile `syntax`.
---
205996 ./obj/build/cache/2019-04-11
157460 ./obj/build/bootstrap/debug/incremental
156496 ./src/llvm-project/clang
142472 ./obj/build/bootstrap/debug/incremental/bootstrap-gm2kk8y15os9
142468 ./obj/build/bootstrap/debug/incremental/bootstrap-gm2kk8y15os9/s-fcda8kt4rb-umvqm1-34hadj2q5a3fu
108532 ./src/llvm-project/lldb
107952 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc
101812 ./.git
97584 ./src/llvm-project/clang/test

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive
Copy link
Collaborator

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:012bed40:start=1558294773169039585,finish=1558294773964235677,duration=795196092
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:05:23]    Compiling synstructure v0.10.1
[00:05:37]    Compiling rustc_macros v0.1.0 (/checkout/src/librustc_macros)
[00:05:44]     Checking syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[00:05:45]     Checking rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:05:59] error: function is never used: `dummy`
[00:05:59]   --> src/libsyntax/tokenstream.rs:55:1
[00:05:59] 55 | / fn dummy()
[00:05:59] 56 | | where
[00:05:59] 57 | |     Span: Send + Sync,
[00:05:59] 57 | |     Span: Send + Sync,
[00:05:59] 58 | |     token::Token: Send + Sync,
[00:05:59] ...  |
[00:05:59] 61 | |     TokenStream: Send + Sync,
[00:05:59] 62 | | {}
[00:05:59]    |
[00:05:59]    = note: `-D dead-code` implied by `-D warnings`
[00:05:59] 
[00:06:00] error: aborting due to previous error
---
travis_time:end:0366d711:start=1558295146392795974,finish=1558295146398991039,duration=6195065
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:065171c8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1829528a
travis_time:start:1829528a
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:2dc5c9e0
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister
Copy link
Member

Do we know the actual cause of the problem? Is the compiler running into some kind of livelock?

@michaelwoerister
Copy link
Member

This is what perf tells me after running ./x.py build for an hour or so:

Samples: 22M of event 'cycles:uppp', Event count (approx.): 22349476896140
Overhead  Command          Shared Object                                 Symbol
  21.14%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_stack
   6.54%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   4.53%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_predicate_recursively
   3.03%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::context::TyCtxt::_intern_substs
   2.51%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::start_snapshot
   2.48%  rustc            libc-2.28.so                                  [.] __memmove_avx_unaligned_erms
   2.31%  rustc            libc-2.28.so                                  [.] _int_malloc
   1.63%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_predicates_recursively
   1.59%  rustc            libc-2.28.so                                  [.] _int_free
   1.34%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::in_snapshot
   1.25%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   1.22%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::confirm_candidate
   1.10%  rustc            libc-2.28.so                                  [.] malloc
   1.00%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <smallvec::SmallVec<A> as core::iter::traits::collect::FromIterator<<A as smallvec::Array>::Item>>::from_iter
   0.97%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::commit_from
   0.92%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::candidate_from_obligation
   0.88%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next
   0.86%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::context::CtxtInterners::intern_ty
   0.82%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next
   0.81%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   0.70%  rustc            libc-2.28.so                                  [.] realloc
   0.67%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::fold::<impl rustc::ty::context::TyCtxt>::replace_bound_vars
   0.65%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::project::normalize_with_depth
   0.62%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <smallvec::SmallVec<A> as core::iter::traits::collect::FromIterator<<A as smallvec::Array>::Item>>::from_iter
   0.62%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <rustc::traits::ObligationCause as core::clone::Clone>::clone
   0.60%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] llvm::GlobalsAAResult::DeletionCallbackHandle::deleted
   0.58%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <T as rustc::ty::context::InternIteratorElement<T,R>>::intern_with
   0.58%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::probe
   0.57%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::impl_or_trait_obligations
   0.49%  rustc            rustc                                         [.] free
   0.48%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::in_snapshot
   0.45%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::collect_predicates_for_types
   0.45%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <rustc::traits::ObligationCause as core::clone::Clone>::clone
   0.40%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] combineInstructionsOverFunction
   0.38%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] llvm::ValueHandleBase::AddToUseList
   0.37%  rustc            rustc                                         [.] malloc

@Zoxc Zoxc mentioned this pull request May 23, 2019
@pietroalbini pietroalbini added beta-nominated Nominated for backporting to the compiler in the beta channel. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 24, 2019
@pietroalbini
Copy link
Member

Beta nominating this. It's blocking the bootstrap update and the version number bump for nightly 1.37.0.

@Zoxc
Copy link
Contributor Author

Zoxc commented May 24, 2019

We just need this on master to bump the bootstrap compiler.

@michaelwoerister
Copy link
Member

Let's land this for now. @Zoxc, r=me if you add an explanation why the explicit impls for TokenTree are needed (with a link to the GH issue).

@ghost
Copy link

ghost commented May 25, 2019

worked for me with commit 315ab95 (HEAD -> master, origin/master, origin/HEAD)

but without this patch I get infinite loop(?) for 4 hours #61162

Could the root cause of the infinite loop be fixed insteadalso?
Because this seems like a workaround. ie. it should maybe detect and fail without this patch.

@pietroalbini
Copy link
Member

This is blocking making nightly 1.37.0 so I'd prefer for the workaround to land first.

@Zoxc
Copy link
Contributor Author

Zoxc commented May 27, 2019

@bors r=michaelwoerister

@bors
Copy link
Contributor

bors commented May 27, 2019

📌 Commit 3ed0561 has been approved by michaelwoerister

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 27, 2019
@pietroalbini
Copy link
Member

@bors p=1

Blocking the nightly version bump.

@bors
Copy link
Contributor

bors commented May 27, 2019

⌛ Testing commit 3ed0561 with merge 4dbc7f9...

bors added a commit that referenced this pull request May 27, 2019
Short circuit Send and Sync impls for TokenTree

Workaround to make the parallel compiler build after #60444.

r? @nikomatsakis
@bors
Copy link
Contributor

bors commented May 27, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: michaelwoerister
Pushing 4dbc7f9 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label May 27, 2019
@bors bors merged commit 3ed0561 into rust-lang:master May 27, 2019
@rust-highfive
Copy link
Collaborator

📣 Toolstate changed by #60967!

Tested on commit 4dbc7f9.
Direct link to PR: #60967

🎉 rls on windows: test-fail → test-pass (cc @Xanewok, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request May 27, 2019
Tested on commit rust-lang/rust@4dbc7f9.
Direct link to PR: <rust-lang/rust#60967>

🎉 rls on windows: test-fail → test-pass (cc @Xanewok, @rust-lang/infra).
@pietroalbini pietroalbini removed T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. beta-nominated Nominated for backporting to the compiler in the beta channel. labels May 27, 2019
@pietroalbini
Copy link
Member

Removing the beta nomination.

@Zoxc Zoxc deleted the fix-syntax-sync branch May 27, 2019 19:38
@Keruspe
Copy link
Contributor

Keruspe commented May 28, 2019

@pietroalbini why removing the beta nomination? I guess this could be the cause of #61262 ?

@pnkfelix
Copy link
Member

pnkfelix commented Jun 4, 2019

I think @pietroalbini removed the beta nomination because we typically reserve beta backports for things that will help resolve "customer" issues, and entities bootstrapping the compiler (be theyrustc developers or package distribution systems) are, for better or for worse, not always considered first-class customers?

(I am not trying to disparage anyone in my comment above; I made the above statement as someone who both is a rustc developer bootstrapping the compiler all the time, and also as someone who triages bugs on a weekly basis and struggles with the question of which bugs are worth the risk of backporting to beta.)

Anyway, what would be the expected utility of a beta-backport here? As far as I can tell by reading the discussion on #61262, the problem only arises when one uses parallel-compiler=true for a bootstrap, right? Are you bootstrapping using a checkout of the source code of the beta-branch itself?

@Keruspe
Copy link
Contributor

Keruspe commented Jun 4, 2019

@pnkfelix I am the rust packager for Exherbo and the package builds the beta branch directly from a checkout.
I closed the bug though as the issue can indeed be worked around

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants