Accept tuple.0.0 as tuple indexing (take 2)

[petrochenkov]

If we expect something identifier-like when parsing a field name after ., but encounter a float token, we break that float token into parts, similarly to how we break && into & &, or << into < <, etc.

An alternative to #70420.

[rust-highfive]

r? @cramertj

(rust_highfive has picked a reviewer for you, use r? to override)

[nikomatsakis]

Nominating for @rust-lang/lang to discuss in upcoming meeting. (This was created at our request, see discussion in #70420.)

[CAD97]

_{(I don't know where else to put it but it felt relevant to note somewhere. You mention the possibility of only lexing decomposed floating point literals (i.e. 1.2 would be lit_int dot lit_int and not lit_float) and gluing floating point literals together in the parser. I'm taking this approach in an experimental formal lexical grammar, and it works, but is not without its fun edge cases. The most relevant is 0.0b·, where · is an XID_Continue character. Without floating point in the lexer, this naively lexes as lit_int dot lit_bin_int ERROR.)}

_{And as a side side note: don't decompose away the e exponent of float literals or the 0b/0o/0x of non-base-10 integer literals; that way lies madness and very ugly identifier introspection.}

_{As such,} As a member of wg-grammar (but not speaking for the team), I'm somewhat wary of the impact this will have on an eventual syntactical specification. I don't think our current draft can split proc_macro tokens in the way required to implement this. However, I still think that the intent here is a net positive, and this token splitting seems like the least bad way to get this short term (i.e. before full decomposed tokens throughout).

[petrochenkov]

@CAD97

And as a side side note: don't decompose away the e exponent of float literals or the 0b/0o/0x of non-base-10 integer literals; that way lies madness and very ugly identifier introspection.

:)
I didn't thought about splitting 0b/0o/0x, but lexically e seems to be just a integer token suffix like e.g. u8 (or any other currently invalid suffix reserved for future use).
To understand what part of 0xabcdefu8 is a suffix we need to lex until we encounter something outside of the [a-f] range. I guess that's the ugly introspection? (Hexadecimal integers will never be able to have the e suffix in this case.)

[CAD97]

@petrochenkov It might be easier if the entire stack is fully decomposed to treat 1e10f32 as a regular suffix. But the most annoying part for my experimental specification is that e.g. 0o0e is rejected by the lexer as an octal float, but if it were just a literal suffix, that would be accepted. Plus exponentiation is [eE][+-]?, so it can't just be handled by an integer suffix to a numeric base stem.

Treating 0[box] nondecimal integers as an integer suffix is annoying because then you have to introspect the integer token to see if it's a zero, in addition to checking the integer suffix for what looks like the rest of the literal.

I guess most of the pain comes down to the fact that 0o0e0 is rejected at the lexer level, otherwise float exponents probably could just be part of the literal suffix as far as the lexer is concerned.

In general, though, if you're doing decomposed tokens, it's really not great if you have to re-introspect tokens more granularly, except for the special case of reserved words (keywords). Eating the entire non-decimal integers and e-based floats in the raw lexer layer seems to be a lot more convenient so far. To not derail this PR further, if you want to talk about this further, you can open a thread over at the repo where I'm working on the experimental spec or message me on zulip. I'm open to being corrected 😄

[joshtriplett]

Dropping nomination. Based on discussions in today's lang team meeting, we're fine with this approach, and we're happy to delegate further iteration and implementation to the compiler team.

[crlf0710]

r? @estebank

[nikomatsakis]

I would note a few things that we said in the lang team meeting but which weren't in Josh's summary, but I think they're worth noting:

• I think this approach is clever, but it's going to push a bit of a burden onto all Rust parser implementations (e.g., syn will have to reproduce this sort of logic as well).
• Modifying the lexer to use 'compound tokens' for floats (i.e., 1.0 produces a integer and then a dot and then another integer, with the "no whitespace" flag , or whatever scheme we wound up with for joint tokens) seems like it would've been the right approach, but it's backwards incompatible and would break existing code.
• The approach @dtolnay used initially, I think, was to modify the lexer definition so that 0.0 lexes differently when preceded by a ., right? Or something like that? (I didn't find a very precise definition in the PR)

I guess that upon further reflection I'm not sure that putting this burden on the parser is the right call. It's going to effect our ability to make a formal grammar, and it's going to have to be replicated by all other parsers?

We could plausibly handle the backwards compatibility option by having the lexer have a flag, or by adopting the "smarter lexer" approach that I think @dtolnay originally proposed (since all proc-macros and the like would then benefit from using our lexer automatically).

I think in the end I'm comfortable with whatever approach we take, but I do think that we should

(a) have a write-up that expands on the different approaches considered
(b) summarize the pros/cons of each approach and the impacts on procedural macro authors and parser authors
(c) lays out the reason to pick one particular approach -- that's a bit what I'm missing.

As always, I don't think this has to be a treatise -- this comment is maybe a good part of the way there? But I think it'd be useful to write out the logic.

[nikomatsakis]

Also, not to be a procedural stickler, but I am thinking that we really ought to be doing a FCP here, right? This is a visible change to the language and I think it merits a FCP to give folks a "heads up".

@petrochenkov do you think you could try a write-up like I described so I can fcp merge?

[petrochenkov]

do you think you could try a write-up like I described so I can fcp merge?

Yes, probably next week.

[CAD97]

@nikomatsakis

[Using decomposed float tokens is] backwards incompatible

I don't think this is necessarily true.

Using decomposed tokens for floats in proc_macro::TokenStream or macro_rules!'s $:tt would be, but using a decomposed model for rustc_lexer and rustc_parse is not, so long as they're glued back together for the above stable APIs.

(That said: proc_macro based parsers (e.g. syn) would still have to do token splitting anyway, unless the gluing was done with the "smarter lexer" (informal) context. Effectively, it's requiring the macro_rules!-style operator gluing for float literals. Final side note: I'm exploring decomposing the literal suffix off of the literal literal in my micro lexer, so a design following that would have to have this gluing-for-proc_macro anyway.)

It depends on how close you want librustc_parse to be to proc_macro, I suppose. Or if you're willing to have librustc_parse diverge slightly from proc_macro and introduce a new meta::tokens::TokenStream that is more fully decomposed in the future. (proc_macro::TokenStream -> meta::tokens::TokenStream would be lossless (given a span splitting API) if it continues to not include whitespace explicitly.)

Looong term, I think properly (always) decomposing floats in the lexer is the better choice, but would require a breaking proc_macro -> meta transition to expose fully to proc_macro consumers. However, if (and only if) always representing floats as decomposed in the lexer token stream is infeasable, then I think adding another "smarter lexer" rule for this is probably less technical debt long term, because it's fully contained within the lexer (like the ident dot identifier rule; I think that most people would agree that having that split is way better).

(I can just say that my micro lexer experiment would not like it if it has to handle the "smarter lexer" option in its current design; the existing "smarter lexer" behavior for integer dot ident is the most complicated part of the definition as currently written, and dot ident $(dot ident)+, as currently specified, would take an infinite amount of rules. (It's fixable but would take a complete restructuring of the current formalism.))

TL;DR we can stability without stagnation our lexer if we just always shim to/from the stable proc_macro API. (Note: said shim will still be required if/when the "IDE syntax tree unification" happens (cc @matklad).)

[petrochenkov]

TL;DR we can stability without stagnation our lexer if we just always shim to/from the stable proc_macro API.

And the shim already exists due to translation of doc comments and maybe some other stuff, it just works only in the opposite right now - "one rustc token -> multiple proc_macro token", rather than "multiple rustc tokens <-> one proc_macro token".

EDIT: The shim also currently has "multiple proc_macro tokens -> one rustc token" for lifetimes and "one proc_macro token -> multiple rustc tokens" for negative literals. (And multi-character operators of course, but they will go away if rust migrates to more fine-grained tokens.)

[nikomatsakis]

@CAD97 Indeed. I believe the approach you're suggesting is what I was hinting at when I wrote this:

We could plausibly handle the backwards compatibility option by having the lexer have a flag, or by adopting the "smarter lexer" approach that I think @dtolnay originally proposed (since all proc-macros and the like would then benefit from using our lexer automatically).

In particular, we can clearly preserve the lexing behavior for procedural macros by inserting some shims to convert from the new tokens to the old ones -- and I think that's fine. But it's also true that procedural macros would probably prefer the tokens that rustc is seeing, since it would enable e.g. syn to parse Rust more easily (i.e., the same reason that our parser wants different tokens).

I guess this is what you meant when you talked about the "meta" shift -- i.e., we might use that shift as a kind of "edition" change for procedural macros?

[petrochenkov]

Stabilization report

This PR implements parsing tuple.0.0 as two field accesses instead of erroneous tuple, . and floating point 0.0.
The feature is implemented as insta-stable.

To be more specific, if we expect something identifier-like when parsing a field name after ., but encounter a float token, we break that float token into parts, similarly to how we break && into & &, or << into < <, etc.

Larger picture

There are two ways to achieve tuple.0.0 being parsed as two tuple field accesses.

• One way is to keep the current "compound" float tokens and introduce a special case into the lexer so it treats 0.0 differently depending on its context (as one float token, or as three integer and punctuation tokens).
• Another way is to break the float tokens into more basic components (possibly suffixed integer tokens and punctuation) and defer the task of discerning between floats and tuple field accesses to the parser.

Pros and cons of both approaches:

• Compound float tokens in general, broken in the lexer in few special cases
  • (Negative) From the general design point of view, if we have basic components which we want to break into parts sometimes (like 0.0 tokens), perhaps they are not basic enough and we need to rethink our set of basic components. Especially given that breaking floats doesn't require introducing any new basic components, since they can be broken into already existing integers and punctuation.
  • (Positive) Freezing the tokens produced by rustc lexer (which provides no stability guarantees) allows to avoid any compatibility issues with proc macro APIs and macro_rules matchers which do provide stability guarantees.
  • (Positive) There's some precedent for treating float tokens specially in the lexer, 0. is lexed as two tokens rather than as a float if it's followed by . or an identifier (to support ranges 0..1 and integer methods 0.foo()).
• Always breaking float tokens into integers and punctuation in the lexer
  • (Positive) The general design point of view, see above. Compound operators (like <<) are currently represented as single tokens in rustc lexer and there are already plans to break them up with the same motivation.
  • (Positive) Can remove the existing floating-point-related special cases in rustc lexer (0..1 and 0.foo()).
  • (Negative) Proc macro API and tt matcher have stability guarantees, we will need shims at proc and decl macro boundaries to represent multiple internal tokens as one (0.0 or <<) when matching with tt and when producing proc macro tokens. (Bumping the major version of proc macro API and the ways to make migration from pmAPI1 to pmAPI2 painless are out of scope right now.)
  • (Negative?) Lexer does less work, so parsers have to do more work. There are multiple parsers, so they all will need to provide tools for combining multi-character operators (< < -> <<) and floating point literals (0 . 0 -> 0.0, assuming proc macro API migrates to the new model) from basic tokens, or breaking float tokens into parts (0.0 -> 0 . 0, assuming proc macro API is frozen). proc_macro crate itself doesn't provide such facilities, so they need to be implemented by crates like syn that provide conveniences on top of the barebones proc_macro.

This PR chooses the second strategy (breaking float tokens) by not introducing special float-breaking cases to the lexer, but emulates it in the parser, similarly to how we emulate future single-character punctuation by breaking << into < and < in the parser.
Later when we have some necessary infrastructural work done we migrate both << and 0.0 to more basic tokens properly in the lexer.

[petrochenkov]

@rust-lang/lang
Could someone start an FCP?

[nikomatsakis]

@petrochenkov

Before I do, a point of clarification. Under the heading "Compound float tokens", you wrote

(Positive) There's some precedent for treating float tokens specially in the lexer, 0. is lexed as two tokens rather than as a float if it's followed by . or an identifier (to support ranges 0..1 and integer methods 0.foo()).

But this doesn't sound like using a compound float token, unless I'm misunderstanding. In particular, you wrote that 0. is lexed in different ways, whereas to be precedent, I would expect 0. to always be lexed as a floating point number and then to have the parser break it up into subtokens.

So I guess the question is, in what way is this a precedent? Or perhaps there's kind of a third route, somewhat similar to what @dtolnay originally proposed, where the lexer understands .0.0 and tries to get enough context to decide what tokens to produce?

[jplatte]

Should this have the relnotes tag and a milestone?