Raw lifetimes #1603

compiler-errors · 2024-09-08T03:21:30Z

src/tokens.md

mattheww · 2024-09-09T22:06:15Z

The

(not immediately followed by ')

bit is there in LIFETIME_OR_LABEL as a way to say that 'xxx'yyy is rejected (rather than being interpreted as two lifetimes).

But (after rust-lang/rust#126452) 'r#xxx'yyy is interpreted as two lifetimes. So I think that bit should be left out of the RAW_LIFETIME rule.

(There's no need to say anything special to indicate that 'r#kw' is rejected: that's true because ' on its own isn't a token, and character literals can't have more than one character between the quotes.)

Alternatively, it might be worth considering changing the implementation to reject 'r#xxx'yyy for consistency with the non-raw case.

Note that allowing that form would close the door to delaying the rejection of overlong character literals to post-expansion, which was being considered late last year at rust-lang/rust#118699 .

ehuss

Can you update https://doc.rust-lang.org/nightly/reference/tokens.html#reserved-prefixes to include a new rule for the reserved prefix lifetime? I would assume it is something like

`'` (IDENTIFIER_OR_KEYWORD | _) `#`

And update the examples in the "Edition differences" in that section to include reserved lifetime prefixes.

Can you add a "Raw lifetime or label" section similar to https://doc.rust-lang.org/nightly/reference/identifiers.html#raw-identifiers that explains what the raw lifetime means?

Can you add an "Edition differences" block in the "Lifetimes and loop labels" section that explains that raw lifetimes are not supported before 2021? There are some examples within the tokens.md chapter of "Edition differences" blocks for the kind of wording to use.

src/tokens.md

ehuss · 2024-10-22T21:40:28Z

@compiler-errors Just checking if you'll be able to look at the requested changes?

compiler-errors · 2024-10-30T00:49:46Z

I believe I addressed the comments, but please look closely because markdown is not my thing.

compiler-errors · 2024-10-30T00:50:08Z

Well, except for:

Alternatively, it might be worth considering changing the implementation to reject 'r#xxx'yyy for consistency with the non-raw case.

Which I think I will open a rustc PR to implement.

compiler-errors · 2024-10-30T03:07:25Z

@rustbot ready

ehuss · 2024-10-30T14:11:10Z

src/tokens.md

+> &nbsp;&nbsp; | RAW_LIFETIME
+>
+> RAW_LIFETIME :\
+> &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]


I believe this should also include _, right? That is, 'r#_ is valid, correct?

I also just want to double-check, it is intentional that 'r#_ is allowed as a loop label? (I only ask because '_ is explicitly not allowed).

Suggested change

>    `'r#` [IDENTIFIER_OR_KEYWORD][identifier]

>       `'r#` [IDENTIFIER_OR_KEYWORD][identifier]

> _(not immediately followed by `'`)_\

>    | `'r#_`

Actually this is likely something else we want to deny here. I feel like 'r#_ is invalid, or at least not something we need to support initially.

rust-lang/rust#132363

Hopefully resolved by 912a6d5

mattheww · 2024-10-30T15:18:13Z

Do we need changes outside the Lexical structure chapter?

As far as I can see nothing is saying that r#a and 'a are treated as equivalent when used as a lifetime or label.

That is, I don't think the Reference is saying that these are now accepted:

fn foo<'r#a>(s: &'a str) {}

    'r#a: { break 'a; }

mattheww · 2024-10-30T15:21:43Z

src/tokens.md

@@ -851,7 +866,8 @@ r[lex.token.reserved-prefix.syntax]
 > **<sup>Lexer 2021+</sup>**\
 > RESERVED_TOKEN_DOUBLE_QUOTE : ( IDENTIFIER_OR_KEYWORD <sub>_Except `b` or `c` or `r` or `br` or `cr`_</sub> | `_` ) `"`\
 > RESERVED_TOKEN_SINGLE_QUOTE : ( IDENTIFIER_OR_KEYWORD <sub>_Except `b`_</sub> | `_` ) `'`\
-> RESERVED_TOKEN_POUND : ( IDENTIFIER_OR_KEYWORD <sub>_Except `r` or `br` or `cr`_</sub> | `_` ) `#`
+> RESERVED_TOKEN_POUND : ( IDENTIFIER_OR_KEYWORD <sub>_Except `r` or `br` or `cr`_</sub> | `_` ) `#`\
+> RESERVED_TOKEN_LIFETIME : `'` (IDENTIFIER_OR_KEYWORD <sub>_Except `r`_</sub> | _) `#`


I think this is isn't reserving 'r# on its own, which rustc rejects but could lex as 'r followed by #.

But I'm not sure it's worth trying to get all the RESERVED_ rules exactly right at this stage.

ehuss · 2024-10-30T17:21:01Z

Posted rust-lang/edition-guide#330 as the companion for the edition.

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

ehuss · 2024-11-05T22:59:54Z

@rustbot author

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

Rollup merge of rust-lang#132363 - compiler-errors:raw-lt-id-valid, r=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

Rollup merge of rust-lang#132341 - compiler-errors:raw-lt-prefix-id, r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…r=chenyukang Reject raw lifetime followed by `'`, like regular lifetimes do See comment. We want to reject cases like `'r#long'id`, which currently gets interpreted as a raw lifetime (`'r#long`) followed by a lifetime (`'id`). This could have alternative lexes, such as an overlong char literal (`'r#long'`) followed by an identifier (`id`). To avoid committing to this in any case, let's reject the whole thing. `@mattheww,` is this what you were looking for in rust-lang/reference#1603 (comment)? I'd say ignore the details about the specific error message (the fact that this gets reinterpreted as a char literal is 🤷), just that because this causes a lexer error we're effectively saving syntactical space like you wanted.

…=wesleywiser Enforce that raw lifetimes must be valid raw identifiers Make sure that the identifier part of a raw lifetime is a valid raw identifier. This precludes `'r#_` and all module segment paths for now. I don't believe this is compelling to support. This was raised by `@ehuss` in rust-lang/reference#1603 (comment) (well, specifically the `'r#_` case), but I don't see why we shouldn't just make it consistent with raw identifiers.

ehuss · 2024-11-14T17:16:11Z

@compiler-errors I pushed a change for the validation, and also rebased since we made a slight change in the definition of the lifetime token in #1668.

Please let me know if you think the current version looks good.

Update for rust-lang/rust#132363

These are rejected by the lexer.

ehuss · 2024-11-15T04:01:17Z

I pushed a commit specifying that 'r#_ will generate an error (and similarly for r#_).

This is a bit of an awkward issue around whether or not _ is an identifier. Currently the reference does not define it that way (intentionally), even though internally rustc allows that (but quickly rejects it). IIRC, there are some issues around that and proc-macros, but I did not look into that.

I did not want to define an identifier token that allowed _, because then I would have to define separate identifier tokens that exclude it, and this _ identifier would only be used by these raw tokens.

compiler-errors · 2024-11-20T04:28:45Z

LGTM!

ehuss

Thanks!

Update cargo and books 10 commits in 66221abdeca2002d318fde6efff516aab091df0e..4c39aaff66862cc0da52fe529aa1990bb8bb9a22 2024-11-19 21:30:02 +0000 to 2024-11-25 16:36:17 +0000 - feat: Stabilize Edition 2024 (rust-lang/cargo#14828) - Improve error handling when PathSource is relative (rust-lang/cargo#14854) - test: address test output nondeterminism (rust-lang/cargo#14855) - chore: move supports-unicode to workspace deps (rust-lang/cargo#14853) - Check build target supports std when building with -Zbuild-std=std (rust-lang/cargo#14183) - fix(publish): Allow dry-run of a non-bumped workspace (rust-lang/cargo#14847) - test: Switch from 'exec_with_output' to 'run' (rust-lang/cargo#14848) - test(rustflags): Verify -Cmetadata directly, not through -Cextra-filename (rust-lang/cargo#14846) - chore: remove bors mentions (rust-lang/cargo#14845) - Clarify how `cargo::metadata` env var is selected (rust-lang/cargo#14842) ## nomicon 1 commits in eac89a3cbe6c4714e5029ae8b5a1c556fd4e8c42..0674321898cd454764ab69702819d39a919afd68 2024-11-16 14:05:28 +0000 to 2024-11-19 12:35:48 +0000 - races: Clarify a “mostly” that might be misread (rust-lang/nomicon#468) ## reference 12 commits in 41ccb0e6478305401dad92e8fd3d04a4304edb4c..5c86c739ec71b8bc839310ff47fa94e94635bba9 2024-11-15 21:45:16 +0000 to 2024-11-25 17:23:35 +0000 - Document `gen` keyword as reserved in Rust 2024 (rust-lang/reference#1501) - 2024: Update `expr` macro fragment specifier (rust-lang/reference#1639) - Add rust_2024 prelude (rust-lang/reference#1552) - 2024: Add reserved syntax (rust-lang/reference#1652) - Add Lifetime Capture Rules 2024 (rust-lang/reference#1601) - Add a section dedicated to Edition 2024 changes to temporary scopes (rust-lang/reference#1592) - 2024: Add unsafe attribute differences (rust-lang/reference#1579) - 2024: Add updates for unsafe extern blocks (rust-lang/reference#1565) - Fix rule for lazy boolean temporary drop scope (rust-lang/reference#1681) - Raw lifetimes (rust-lang/reference#1603) - Fix some missing emdashes (rust-lang/reference#1676) - Added an additional example of lifetime elision (rust-lang/reference#1678) ## rustc-dev-guide 6 commits in b679e71..787b416 2024-11-18 16:18:15 +0800 to 2024-11-22 11:17:57 +0000 - Remove constants section as it is outdated - Flatten generic parameter defs section - Add instructions to test error code docs (rust-lang/rustc-dev-guide#2145) - Reorganize the "Source Code Representation" chapters (rust-lang/rustc-dev-guide#2142) - Make `Diag` a clickable link in Suggestion section (rust-lang/rustc-dev-guide#2140) - update for rustc_intrinsic_const_stable_indirect (rust-lang/rustc-dev-guide#2138) ## edition-guide 6 commits in 915f9b319c2823f310430ecdecd86264a7870d7e..f48b0e842a3911c63240e955d042089e9e0894c7 2024-11-06 07:23:07 +0000 to 2024-11-25 16:20:27 +0000 - Update for 2024 stabilization (rust-lang/edition-guide#338) - Enable triagebot merge-conflicts and shortcuts (rust-lang/edition-guide#336) - Organize the 2024 chapters into sub-chapters by category (rust-lang/edition-guide#334) - Fix broken Cargo Book link in cargo-resolver.md (rust-lang/edition-guide#335) - Edition 2024 guide for temporary lifetime changes (rust-lang/edition-guide#318) - 2024: rustfmt sorting (rust-lang/edition-guide#332)

compiler-errors commented Sep 8, 2024

View reviewed changes

src/tokens.md Outdated Show resolved Hide resolved

ehuss reviewed Sep 10, 2024

View reviewed changes

src/tokens.md Show resolved Hide resolved

src/tokens.md Show resolved Hide resolved

src/tokens.md Outdated Show resolved Hide resolved

ehuss added the S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. label Sep 10, 2024

compiler-errors force-pushed the patch-1 branch from e718896 to 0b6ce17 Compare October 30, 2024 00:49

compiler-errors mentioned this pull request Oct 30, 2024

Reject raw lifetime followed by ', like regular lifetimes do rust-lang/rust#132341

Merged

rustbot added S-waiting-on-review Status: The marked PR is awaiting review from a maintainer and removed S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. labels Oct 30, 2024

ehuss reviewed Oct 30, 2024

View reviewed changes

compiler-errors mentioned this pull request Oct 30, 2024

Enforce that raw lifetimes must be valid raw identifiers rust-lang/rust#132363

Merged

mattheww reviewed Oct 30, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. and removed S-waiting-on-review Status: The marked PR is awaiting review from a maintainer labels Nov 5, 2024

compiler-errors added 3 commits November 14, 2024 08:13

Raw lifetimes

ef87ed6

Don't allow 'r#kw'

df53c5e

update

5c0c306

ehuss force-pushed the patch-1 branch from 0b6ce17 to 912a6d5 Compare November 14, 2024 17:13

ehuss added 2 commits November 14, 2024 19:16

Clarify raw lifetime cannot be certain keywords

f04cd63

Update for rust-lang/rust#132363

Add rule name for raw lifetime edition differences

f490b7c

ehuss force-pushed the patch-1 branch from 912a6d5 to f490b7c Compare November 15, 2024 03:17

Document r#_ and 'r#_ as reserved

baa3d73

These are rejected by the lexer.

ehuss approved these changes Nov 20, 2024

View reviewed changes

ehuss added this pull request to the merge queue Nov 20, 2024

Merged via the queue into rust-lang:master with commit e0a0414 Nov 20, 2024
5 checks passed

ehuss mentioned this pull request Nov 25, 2024

Update cargo and books rust-lang/rust#133465

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw lifetimes #1603

Raw lifetimes #1603

compiler-errors commented Sep 8, 2024

mattheww commented Sep 9, 2024

ehuss left a comment

ehuss commented Oct 22, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

ehuss Oct 30, 2024

compiler-errors Oct 30, 2024

ehuss Nov 14, 2024

mattheww commented Oct 30, 2024

mattheww Oct 30, 2024

ehuss commented Oct 30, 2024

ehuss commented Nov 5, 2024

ehuss commented Nov 14, 2024

ehuss commented Nov 15, 2024

compiler-errors commented Nov 20, 2024

ehuss left a comment

-> &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]
+> &nbsp;&nbsp; &nbsp;&nbsp; `'r#` [IDENTIFIER_OR_KEYWORD][identifier]
+>   _(not immediately followed by `'`)_\
+> &nbsp;&nbsp; | `'r#_`

Raw lifetimes #1603

Raw lifetimes #1603

Conversation

compiler-errors commented Sep 8, 2024

mattheww commented Sep 9, 2024

ehuss left a comment

Choose a reason for hiding this comment

ehuss commented Oct 22, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

compiler-errors commented Oct 30, 2024

ehuss Oct 30, 2024

Choose a reason for hiding this comment

compiler-errors Oct 30, 2024

Choose a reason for hiding this comment

ehuss Nov 14, 2024

Choose a reason for hiding this comment

mattheww commented Oct 30, 2024

mattheww Oct 30, 2024

Choose a reason for hiding this comment

ehuss commented Oct 30, 2024

ehuss commented Nov 5, 2024

ehuss commented Nov 14, 2024

ehuss commented Nov 15, 2024

compiler-errors commented Nov 20, 2024

ehuss left a comment

Choose a reason for hiding this comment