Make GlobalCtxt implement Sync #45912

Zoxc · 2017-11-10T13:19:09Z

This PR makes the GlobalCtxt implement Sync, which is the first step towards making rustc thread safe so we can execute queries in multiple threads.

The changes here are mostly mechanical where Cell is replaced by LockCell, Rc is replaced by Lrc and RefCell is replaced by either Lock or RwLock. Some thread local variables are now declared with rustc_global!, which indicates that they are intended to be global to a rustc session, however they aren't changed to global variables since rustdoc has multiple threads with rustc sessions in them.

Here is a list of the larger changes:

src/librustc_data_structures/sync.rs has the new types other items related to thread safety.
src/libarena/lib.rs is changed so that there's a lock around each arena if thread safety is enabled.
There's a new [rust] experimental_parallel_queries entry for config.toml which enables thread safety. It is disabled by default.
-Z query_threads allows you to set the number of thread used to compute queries, although it is current ignored.
It uses my fork of owning_ref, so we can erase a OwningRef into a thread-safe type.
GlobalArenas and DroplessArena are combined into AllArenas which is the type drivers shoud use. This refactoring makes it easier to create thread-local arenas in the future.
A SameThread type is introduces in src/librustc_trans/metadata.rs which asserts that operations on the inner type all happen on the same thread. This is used to make ArchiveRO and ObjectFile Send + Sync so we can make the [u8] metadata reference into them Send + Sync.
ThreadCtxt is introduced. TyCtxt now points to it instead of GlobalCtxt. ThreadCtxt contains a references to a GlobalCtxt and contains the parts of the `GlobalCtxt which has to be thread local.
mk_attr_id in src/libsyntax/attr.rs is changed to use an AtomicUsize.
The err_count field of Handler in src/librustc_errors/lib.rs is now a AtomicUsize instead of Cell<usize>.

One thing to note is that Lock and RwLock implements Clone, since there's code which relies the Clone impl for RefCell. There's some risk of deadlocks due to that. Replacing RefCell with Lock also risks deadlocks, since RefCell is re-entrant for read-locks, while Lock is not.

Zoxc · 2017-11-10T13:20:51Z

@bors try

bors · 2017-11-10T13:20:53Z

@Zoxc: 🔑 Insufficient privileges: and not in try users

shepmaster · 2017-11-10T14:54:15Z

@bors try

bors · 2017-11-10T14:54:26Z

⌛ Trying commit ad7b865 with merge d2cb23b...

WIP: Playing with adding thread safety to GlobalCtxt

shepmaster · 2017-11-10T14:55:25Z

Assigning to...

r? @pnkfelix

bors · 2017-11-10T16:19:18Z

💔 Test failed - status-travis

Zoxc · 2017-11-10T21:53:11Z

What is everyone opinion on introducing locks and Arcs in rustc in an effort to make GlobalCtxt thread safe? We could redirect these types to RefCell, Rc, etc. in production builds to avoid paying the overhead for these thread safe types.

cc @rust-lang/compiler

eddyb · 2017-11-10T22:05:47Z

I would rather not introduce #[cfg]s into rustc like that, FWIW.

Zoxc · 2017-11-10T22:17:07Z

I'd like the ability to turn on and off thread safety so we can easily measure the overhead of it. That is going to require some #[cfg]s at least.

michaelwoerister · 2017-11-13T10:36:12Z

If the #[cfg]s are very localized and they are clearly experimental, I'm not strictly against adding some. However, in such a case the effort of keeping such a patch around locally would not be too big either.

bors · 2017-11-14T10:28:31Z

☔ The latest upstream changes (presumably #45916) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · 2017-11-14T14:02:29Z

@bors try

bors · 2017-11-14T14:02:39Z

⌛ Trying commit 5ca1224 with merge 6849630...

WIP: Playing with adding thread safety to GlobalCtxt

Zoxc · 2017-11-14T15:14:06Z

@Mark-Simulacrum Can I get a perf run when bors completes?

bors · 2017-11-14T15:33:28Z

💔 Test failed - status-travis

kennytm · 2017-11-14T16:02:09Z

@Zoxc please break every tool in src/tools/toolstate.toml and try again.

bjorn3 · 2017-11-14T16:02:47Z

src/libarena/Cargo.toml

@@ -7,3 +7,6 @@ version = "0.0.0"
 name = "arena"
 path = "lib.rs"
 crate-type = ["dylib"]
+
+[dependencies]
+rustc_data_structures = { path = "../librustc_data_structures" }


Missing trailing newline

bjorn3 · 2017-11-14T16:11:13Z

src/librustc_data_structures/Cargo.toml

+
+[dependencies.parking_lot]
+version = "0.5"
+features = ["nightly", "owning_ref"]


Missing newline

bjorn3 · 2017-11-14T16:12:59Z

src/librustc_data_structures/lock.rs

+    fn clone(&self) -> Self {
+        Lock::new(self.borrow().clone())
+    }
+}


Missing newline

bors · 2017-11-16T23:30:02Z

☔ The latest upstream changes (presumably #45825) made this pull request unmergeable. Please resolve the merge conflicts.

…n some cases.

…fg!(parallel_queries)

WIP: Parallelize passes using rayon This builds on #46193 and #45912 and actually makes code run in parallel. This is not quite ready yet since `rustc` is not yet completely thread safe. It also uses a rough fork of rayon which uses fibers/stackful coroutines.

bors · 2017-12-12T03:33:54Z

☔ The latest upstream changes (presumably #46657) made this pull request unmergeable. Please resolve the merge conflicts.

@arielb1

Add sync module to rustc_data_structures This PR is split out from #45912, since github apparently can't handle such large PRs. r? @arielb1

Zoxc · 2017-12-17T16:56:13Z

This is being split into smaller PRs.

Work towards thread safety in rustc This PR is split out from #45912. It contains changes which do not require the `sync` module.

WIP: Parallelize passes using rayon This builds on #46193 and #45912 and actually makes code run in parallel. This is not quite ready yet since `rustc` is not yet completely thread safe. It also uses a rough fork of rayon which uses fibers/stackful coroutines.

shepmaster added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 10, 2017

bors added a commit that referenced this pull request Nov 10, 2017

Auto merge of #45912 - Zoxc:par-query, r=<try>

d2cb23b

WIP: Playing with adding thread safety to GlobalCtxt

rust-highfive assigned pnkfelix Nov 10, 2017

Zoxc force-pushed the par-query branch from ad7b865 to 178997f Compare November 10, 2017 22:37

Zoxc force-pushed the par-query branch 2 times, most recently from 7cb9fc5 to 7c78120 Compare November 14, 2017 07:13

Zoxc force-pushed the par-query branch from 7c78120 to 5ca1224 Compare November 14, 2017 13:59

bors added a commit that referenced this pull request Nov 14, 2017

Auto merge of #45912 - Zoxc:par-query, r=<try>

6849630

WIP: Playing with adding thread safety to GlobalCtxt

bjorn3 requested changes Nov 14, 2017

View reviewed changes

Zoxc force-pushed the par-query branch 3 times, most recently from 7a7b7e8 to edcc51a Compare November 16, 2017 19:29

Zoxc added 12 commits December 8, 2017 22:18

Make TransitiveRelation thread safe. Avoid locking by using get_mut i…

b66aab1

…n some cases.

Remove useless Rc

e12074f

Use rustc_erase_owner! which produces a Send + Sync erased owner if c…

5549ada

…fg!(parallel_queries)

Add Encodable and Decodable impls for Arc<[T]>

1155689

Make USED_ATTRS and KNOWN_ATTRS into globals

35fbe11

Make REGISTERED_DIAGNOSTICS a global

5c4dacd

FIXME note + thread safety

9251af3

Make HYGIENE_DATA a global

868dda0

Make Span and Symbol implement Send and Sync

2d3112a

Make InternedString Send

6e573cb

Make syntax_pos interners globals

3bb008d

Mechanical translation which results in GlobalCtxt being Sync

e3a63ed

Zoxc force-pushed the par-query branch from b7c6360 to e3a63ed Compare December 9, 2017 03:01

Zoxc mentioned this pull request Dec 9, 2017

Make rustc thread safe when queries can be used #46596

Closed

16 tasks

Zoxc mentioned this pull request Dec 13, 2017

Add sync module to rustc_data_structures #46709

Merged

Zoxc mentioned this pull request Dec 17, 2017

Work towards thread safety in rustc #46779

Merged

bors added a commit that referenced this pull request Dec 17, 2017

Auto merge of #46709 - Zoxc:par-merge, r=arielb1

af57ace

Add sync module to rustc_data_structures This PR is split out from #45912, since github apparently can't handle such large PRs. r? @arielb1

Zoxc closed this Dec 17, 2017

bors added a commit that referenced this pull request Dec 22, 2017

Auto merge of #46779 - Zoxc:par-merge-without-sync, r=arielb1

2c037d5

Work towards thread safety in rustc This PR is split out from #45912. It contains changes which do not require the `sync` module.

Zoxc mentioned this pull request Dec 23, 2017

Mechanical translation towards making GlobalCtxt implement Sync #46958

Closed

nikomatsakis mentioned this pull request Apr 6, 2018

Query Parallelization Tracking Issue #48685

Open

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make GlobalCtxt implement Sync #45912

Make GlobalCtxt implement Sync #45912

Zoxc commented Nov 10, 2017 •

edited

Loading

Zoxc commented Nov 10, 2017

bors commented Nov 10, 2017

shepmaster commented Nov 10, 2017

bors commented Nov 10, 2017

shepmaster commented Nov 10, 2017

bors commented Nov 10, 2017

Zoxc commented Nov 10, 2017

eddyb commented Nov 10, 2017

Zoxc commented Nov 10, 2017

michaelwoerister commented Nov 13, 2017

bors commented Nov 14, 2017

Zoxc commented Nov 14, 2017

bors commented Nov 14, 2017

Zoxc commented Nov 14, 2017

bors commented Nov 14, 2017

kennytm commented Nov 14, 2017

bjorn3 Nov 14, 2017 •

edited

Loading

bjorn3 Nov 14, 2017

bjorn3 Nov 14, 2017

bors commented Nov 16, 2017

bors commented Dec 12, 2017

Zoxc commented Dec 17, 2017

Make GlobalCtxt implement Sync #45912

Make GlobalCtxt implement Sync #45912

Conversation

Zoxc commented Nov 10, 2017 • edited Loading

Zoxc commented Nov 10, 2017

bors commented Nov 10, 2017

shepmaster commented Nov 10, 2017

bors commented Nov 10, 2017

shepmaster commented Nov 10, 2017

bors commented Nov 10, 2017

Zoxc commented Nov 10, 2017

eddyb commented Nov 10, 2017

Zoxc commented Nov 10, 2017

michaelwoerister commented Nov 13, 2017

bors commented Nov 14, 2017

Zoxc commented Nov 14, 2017

bors commented Nov 14, 2017

Zoxc commented Nov 14, 2017

bors commented Nov 14, 2017

kennytm commented Nov 14, 2017

bjorn3 Nov 14, 2017 • edited Loading

Choose a reason for hiding this comment

bjorn3 Nov 14, 2017

Choose a reason for hiding this comment

bjorn3 Nov 14, 2017

Choose a reason for hiding this comment

bors commented Nov 16, 2017

bors commented Dec 12, 2017

Zoxc commented Dec 17, 2017

Zoxc commented Nov 10, 2017 •

edited

Loading

bjorn3 Nov 14, 2017 •

edited

Loading