Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement edition hygiene for keywords #50307

Merged
merged 9 commits into from
May 18, 2018
Merged

Conversation

petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Apr 29, 2018

Determine "keywordness" of an identifier in its hygienic context.
cc #49611

I've resurrected proc as an Edition-2015-only keyword for testing purposes, but it should probably be buried again. EDIT: proc is removed again.

@rust-highfive
Copy link
Collaborator

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 29, 2018
@petrochenkov
Copy link
Contributor Author

r? @nikomatsakis
cc @Manishearth

@Manishearth
Copy link
Member

I've resurrected proc as an Edition-2015-only keyword for testing purposes, but it should probably be buried again.

We don't need keywords that are keywords on old editions but not new, we need it the other way around 😄

@Manishearth
Copy link
Member

Haven't really gone through this completely, but could you include some form of the changes from 0eabb1b so that it's less fragile? (and liberally sprinkle comments that link things together)

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 30, 2018
r#async = consumes_async!(async); // OK
// r#async = consumes_async!(r#async); // ERROR, not a match
// r#async = consumes_async_raw!(async); // ERROR, not a match
r#async = consumes_async_raw!(r#async); // OK
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I continue to be nervous about the idea that macros can observe the different here -- what happens if I have a macro from a 2018 Edition crate that accepts r#async? Can a 2015 crate pass in async and have it work?

Put another way:

If I have a macro that accepts a async in its input, can I transition my crate to 2018 Edition without a breaking change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With scheme implemented in this PR keyword hygiene and raw identifiers are entirely orthogonal.
(So this PR doesn't make changes to any interactions between raw identifiers and macros.)

If macro accepts async literally (not as $i: ident or something) in its left hand side, then it will not appear in the macro output, so we are safe even without keyword hygiene (as consumes_async!(async) shows, yeah).

let mut async = 1; // OK
let mut r#async = 1; // OK

r#async = consumes_async!(async); // OK
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the answers to my question are here, maybe?

@nikomatsakis
Copy link
Contributor

This looks pretty good so far! (modulo my questions about the desired behavior of raw idents in macros)

@petrochenkov
Copy link
Contributor Author

Update: I'll return to this once I'm done with proc macro 1.2 API review, likely this weekend.

@@ -359,16 +359,20 @@ declare_keywords! {
(53, Virtual, "virtual")
(54, Yield, "yield")

// Edition-specific keywords reserved for future use.
(55, Async, "async") // >= 2018 Edition Only
(56, Proc, "proc") // <= 2015 Edition Only
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we keeping proc reserved on 2015? We'll surely never use it for anything on the 2015 epoch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVM saw the comment explaining this

@bors
Copy link
Contributor

bors commented May 7, 2018

☔ The latest upstream changes (presumably #50454) made this pull request unmergeable. Please resolve the merge conflicts.

@Manishearth
Copy link
Member

any updates

@petrochenkov
Copy link
Contributor Author

petrochenkov commented May 10, 2018

@Manishearth
I still need to do the plumbing mentioned in TODO (thread edition info from crate metadata into libsyntax/ext/expand.rs where it's needed).
I'll work on this tomorrow.

@Manishearth
Copy link
Member

Manishearth commented May 10, 2018 via email

@petrochenkov
Copy link
Contributor Author

PR updated, all tests are now fixed.
It's too late today, I'll cleanup stuff / remove proc tomorrow and it should be ready.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[01:00:15] ............test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[01:00:45] ........................................................................................
[01:01:06] .....................................................................................ii.............
[01:02:04] .................................................i..................................................
[01:02:09] ..i.ii.........test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[01:03:02] ..........iiiiiii...................................................................................
[01:03:24] ....................................................................................................
[01:03:44] ....................................................................................................
[01:04:05] ...................................................................................
---
travis_time:start:test_run-pass-fulldeps
Check compiletest suite=run-pass-fulldeps mode=run-pass (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:08:48] 
[01:08:48] running 88 tests
[01:10:38] ..........................................FFF...........................test [run-pass] run-pass-fulldeps/myriad-closures.rs has been running for over 60 seconds
[01:14:11] failures:
[01:14:11] 
[01:14:11] ---- [run-pass] run-pass-fulldeps/plugin-args-1.rs stdout ----
[01:14:11]  
[01:14:11]  
[01:14:11] error: auxiliary build of "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" failed to compile: 
[01:14:11] status: exit code: 101
[01:14:11] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-1.stage2-x86_64-unknown-linux-gnu.aux" "-Crpath" "-O" "-Zunstable-options" "--crate-type" "dylib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-1.stage2-x86_64-unknown-linux-gnu.aux"
[01:14:11] ------------------------------------------
[01:14:11] 
[01:14:11] ------------------------------------------
[01:14:11] stderr:
---
[01:14:11]    |     ^^^^^^^^^^^^^^
[01:14:11]    |
[01:14:11]    = note: #[warn(unused_imports)] on by default
[01:14:11] 
[01:14:11] error[E0063]: missing field `edition` in initializer of `syntax::ext::base::SyntaxExtension`
[01:14:11]   --> /checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs:51:9
[01:14:11] 51 |         NormalTT {
[01:14:11] 51 |         NormalTT {
[01:14:11]    |         ^^^^^^^^ missing `edition`
[01:14:11] error: aborting due to previous error
[01:14:11] 
[01:14:11] For more information about this error, try `rustc --explain E0063`.
[01:14:11] 
[01:14:11] 
[01:14:11] ------------------------------------------
[01:14:11] 
[01:14:11] thread '[run-pass] run-pass-fulldeps/plugin-args-1.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3033:9
[01:14:11] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:14:11] 
[01:14:11] ---- [run-pass] run-pass-fulldeps/plugin-args-2.rs stdout ----
[01:14:11]  
[01:14:11] error: auxiliary build of "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" failed to compile: 
[01:14:11] status: exit code: 101
[01:14:11] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-2.stage2-x86_64-unknown-linux-gnu.aux" "-Crpath" "-O" "-Zunstable-options" "--crate-type" "dylib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-2.stage2-x86_64-unknown-linux-gnu.aux"
[01:14:11] ------------------------------------------
[01:14:11] 
[01:14:11] ------------------------------------------
[01:14:11] stderr:
---
[01:14:11]    |     ^^^^^^^^^^^^^^
[01:14:11]    |
[01:14:11]    = note: #[warn(unused_imports)] on by default
[01:14:11] 
[01:14:11] error[E0063]: missing field `edition` in initializer of `syntax::ext::base::SyntaxExtension`
[01:14:11]   --> /checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs:51:9
[01:14:11] 51 |         NormalTT {
[01:14:11] 51 |         NormalTT {
[01:14:11]    |         ^^^^^^^^ missing `edition`
[01:14:11] error: aborting due to previous error
[01:14:11] 
[01:14:11] For more information about this error, try `rustc --explain E0063`.
[01:14:11] 
[01:14:11] 
[01:14:11] ------------------------------------------
[01:14:11] 
[01:14:11] thread '[run-pass] run-pass-fulldeps/plugin-args-2.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3033:9
[01:14:11] 
[01:14:11] ---- [run-pass] run-pass-fulldeps/plugin-args-3.rs stdout ----
[01:14:11]  
[01:14:11] error: auxiliary build of "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" failed to compile: 
[01:14:11] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:492:22
[01:14:11] status: exit code: 101
[01:14:11] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-3.stage2-x86_64-unknown-linux-gnu.aux" "-Crpath" "-O" "-Zunstable-options" "--crate-type" "dylib" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/plugin-args-3.stage2-x86_64-unknown-linux-gnu.aux"
[01:14:11] ------------------------------------------
[01:14:11] 
[01:14:11] ------------------------------------------
[01:14:11] stderr:
---
[01:14:11]    |     ^^^^^^^^^^^^^^
[01:14:11]    |
[01:14:11]    = note: #[warn(unused_imports)] on by default
[01:14:11] 
[01:14:11] error[E0063]: missing field `edition` in initializer of `syntax::ext::base::SyntaxExtension`
[01:14:11]   --> /checkout/src/test/run-pass-fulldeps/auxiliary/plugin_args.rs:51:9
[01:14:11] 51 |         NormalTT {
[01:14:11] 51 |         NormalTT {
[01:14:11]    |         ^^^^^^^^ missing `edition`
[01:14:11] error: aborting due to previous error
[01:14:11] 
[01:14:11] For more information about this error, try `rustc --explain E0063`.
[01:14:11] 
---
[01:14:11] test result: FAILED. 85 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out
[01:14:11] 
[01:14:11] 
[01:14:11] 
[01:14:11] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-pass-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-pass" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-3.9/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "3.9.1\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:14:11linux-gnu/release
41260 ./src/llvm/test/CodeGen/X86
40772 ./src/libcompiler_builtins
40268 ./src/libcompiler_builtins/compiler-rt

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented May 13, 2018

☔ The latest upstream changes (presumably #50235) made this pull request unmergeable. Please resolve the merge conflicts.

@petrochenkov petrochenkov changed the title [WIP] Implement edition hygiene for keywords Implement edition hygiene for keywords May 13, 2018
@petrochenkov petrochenkov added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 13, 2018
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-debug of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:03:48]    Compiling syntax_pos v0.0.0 (file:///checkout/src/libsyntax_pos)
[00:03:49]    Compiling rustc_errors v0.0.0 (file:///checkout/src/librustc_errors)
[00:03:51]    Compiling backtrace v0.3.6
[00:04:03]    Compiling proc_macro v0.0.0 (file:///checkout/src/libproc_macro)
[00:04:03] error[E0425]: cannot find function `is_path_segment_keyword` in module `token`
[00:04:03]    --> libproc_macro/lib.rs:821:19
[00:04:03]     |
[00:04:03] 821 |            token::is_path_segment_keyword(ast::Ident::with_empty_ctxt(ident.sym)) {
[00:04:03]     |                   ^^^^^^^^^^^^^^^^^^^^^^^ not found in `token`
[00:04:04] error: aborting due to previous error
[00:04:04] 
[00:04:04] For more information about this error, try `rustc --explain E0425`.
[00:04:04] error: Could not compile `proc_macro`.
[00:04:04] error: Could not compile `proc_macro`.
[00:04:04] 
[00:04:04] Caused by:
[00:04:04]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name proc_macro libproc_macro/lib.rs --color always --error-format json --crate-type dylib --emit=dep-info,metadata -C prefer-dynamic -C debug-assertions=off -C overflow-checks=on -C metadata=f65e917d1a6aeb8a -C extra-filename=-f65e917d1a6aeb8a --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps --extern rustc_errors=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-24d16f00de0b3e02.rmeta --extern rustc_data_structures=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-750c848698d19171.rmeta --extern syntax_pos=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-54e4cced589011ca.rmeta --extern syntax=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-b878b60a3b79382b.rmeta` (exit code: 101)
[00:04:04] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "--message-format" "json"
[00:04:04] expected success, got: exit code: 101
[00:04:04] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1091:9
[00:04:04] travis_fold:end:stage0-rustc

[00:04:04] travis_time:end:stage0-rustc:start=1526541407644825260,finish=1526541444689757796,duration=37044932536

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@kennytm kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2018
@petrochenkov
Copy link
Contributor Author

Needs rebase.

@petrochenkov
Copy link
Contributor Author

@bors r=nikomatsakis

@bors
Copy link
Contributor

bors commented May 17, 2018

📌 Commit d8bbc1e has been approved by nikomatsakis

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 17, 2018
@withoutboats withoutboats mentioned this pull request May 17, 2018
9 tasks
@bors
Copy link
Contributor

bors commented May 18, 2018

⌛ Testing commit d8bbc1e with merge df40e61...

bors added a commit that referenced this pull request May 18, 2018
Implement edition hygiene for keywords

Determine "keywordness" of an identifier in its hygienic context.
cc #49611

I've resurrected `proc` as an Edition-2015-only keyword for testing purposes, but it should probably be buried again. EDIT: `proc` is removed again.
@bors
Copy link
Contributor

bors commented May 18, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: nikomatsakis
Pushing df40e61 to master...

@cramertj
Copy link
Member

cramertj commented Jun 6, 2018

@petrochenkov Looking over this PR I'm unable to find a way to get the current edition while parsing (not necessarily in a macro). Should the edition be passed down into the parser, or is there something else I should be using to determine if something is a keyword in the current edition while parsing?

@petrochenkov
Copy link
Contributor Author

petrochenkov commented Jun 9, 2018

@cramertj

a way to get the current edition

hygiene::default_edition()
(aka edition of the "root" hygienic context (no expansion) aka DUMMY_SP.edition(), edition needs to be kept in global data like other hygiene-related things)

@petrochenkov
Copy link
Contributor Author

@cramertj

is there something else I should be using to determine if something is a keyword

For a given identifier is_reserved_ident should automatically return the correct answer adjusted for editions, without additional user intervention.

carols10cents added a commit to rust-lang/book that referenced this pull request Oct 23, 2018
`async` was added in rust-lang/rust#50307 and is
not yet implemented

`try` was added in rust-lang/rust#52602 and is
not yet stable: rust-lang/rust#31436
@petrochenkov petrochenkov deleted the keyhyg2 branch June 5, 2019 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants