Document syntax reserved in Rust 2021

[mattheww]

Closes #1069.

I believe this is a correct description of what was implemented in rust-lang/rust#84599 .

I haven't tried to make use of the Reference-level explanation section from RFC3101, which I think doesn't capture what was implemented (for example, it doesn't indicate that continue'label is now rejected).

This PR leaves reserved prefixes undocumented in the Lexer rules blocks, but I hope that's acceptable for now (suffixes for string literals are already missing there).

[ehuss]

Thanks, this looks good!

I'm not sure I understand your comment about the Lexer rule block. What is in the RFC looks like it could be copied here. Can you clarify why that wouldn't be included?

I think string literal missing suffixes is just a bug.

Also, BTW, when opening a PR to resolve an issue, can you use one of the GitHub keywords to link the issue? Something like Closes #1069 in the description will link the issue, and GitHub will automatically close it when the PR is merged.

[mattheww]

The main thing I don't like about those lexer rules is that they say that (for example) x#foo is a single token, but that's never true:

• before Rust 2021 it's two tokens
• from Rust 2021 on it's an error

So if we adopt those rules I think we'd have to say that they exist only from Rust 2021 on.

Then we'd be relying on separate text along the lines of "any instance of the above produces a tokenization error" to complete the explanation, and I think it's not clear that we're gaining anything by having the rules.

As written the rules are also missing the case where ' appears as a lifetime or label rather than as part of a literal. I'm not sure whether they're missing other cases.

(And I now see that _ isn't treated as a keyword in the reference, so we'd need to tweak the places where they say IDENTIFIER_OR_KEYWORD.)

Also, I'm not comfortable with the parts that are like « Except b, r, br ». I think this is attempting to make the lexer rules unambiguous without needing a priority rule, but I believe to make that work we'd need to also modify the rule for IDENTIFIER_OR_KEYWORD, using some kind of lookahead notation. This doesn't seem to be the way the reference is currently written (it isn't trying to disambiguate the prefixes for string literals).

In the long run I think the lexer description is going to be better off using some kind of explicit priority rules instead.

[ehuss]

So if we adopt those rules I think we'd have to say that they exist only from Rust 2021 on.

For edition-differences in the lexer, the reference uses the form > **<sup>Lexer 2018+</sup>** to denote that those lexing rules are only for the given edition. I also think that the PR text clarifies the edition differences pretty well.

Then we'd be relying on separate text along the lines of "any instance of the above produces a tokenization error" to complete the explanation, and I think it's not clear that we're gaining anything by having the rules.

I think it helps with clarifying exactly what the rules are, similar to the same reasoning why other productions are listed. There are several other situations where the rules are listed, and then the text explains that it isn't allowed, like the reserved keywords.

As written the rules are also missing the case where ' appears as a lifetime or label rather than as part of a literal. I'm not sure whether they're missing other cases.

I think we can work on writing the correct rules, since it appears the implementation diverged from the RFC.

I understand it can be difficult to write or interpret the grammar since it is not very formal, particularly in the face of precedence. Unfortunately that is unlikely to improve in the foreseeable future. I think for now there is some value in trying to do the best with what we have.

I think something like the following should work:

RESERVED_TOKEN_DOUBLE_QUOTE : IDENTIFIER_OR_KEYWORD _{Except r or br or b} "
RESERVED_TOKEN_SINGLE_QUOTE : IDENTIFIER_OR_KEYWORD _{Except b} '
RESERVED_TOKEN_POUND : IDENTIFIER_OR_KEYWORD _{Except r or br} #

(And I now see that _ isn't treated as a keyword in the reference, so we'd need to tweak the places where they say IDENTIFIER_OR_KEYWORD.)

I'm not sure I follow. IDENTIFIER_OR_KEYWORD includes _. XID_Start is defined as including _.

[mattheww]

I've added those rules.

On lone _:

XID_Start doesn't include _, and the second part of IDENTITIER_OR_KEYWORD has ⁺ rather than ^*.

The profile in identifiers.md defines "Start" to include _, but doesn't attempt to redefine XID_Start.

(Also, I think it's clear that the reference isn't trying to make _ be an identifier, because if it was that would make the grammar say you could use it as the name of a function or struct.)

[ehuss]

Oh, yea, I got confused mixing up the Start profile. I'm trying to remember why I added that profile stuff, as the UAX #31 definition of <Identifier> := <Start> <Continue>* (<Medial> <Continue>+)* would allow a bare _. Oh well.

Thanks for updating it! I pushed a small fix to have the 2021 edition.