Less `NoDelim` #96421

nnethercote · 2022-04-26T06:51:18Z

Currently there are several places where NoDelim (which really means "implicit delimiter" or "invisible delimiter") is used to mean "no delimiter". The name NoDelim is a bit misleading, and may be a cause.

This PR changes these places, e.g. by changing a DelimToken to Option<DelimToken> and then using None to mean "no delimiter". As a result, the only place where NoDelim values are now produced is within:

Delimiter::to_internal(), when converting from Delimiter::None.
FlattenNonterminals::process_token(), when converting TokenKind::Interpolated.

r? @petrochenkov

rust-highfive · 2022-04-26T06:51:20Z

Some changes occurred in src/tools/rustfmt.

cc @rust-lang/rustfmt

nnethercote · 2022-04-26T06:52:15Z

@petrochenkov: see what you think of this. I was trying to understand NoDelim better, and clearing this up improved my understanding, by making it easier to see where the "real" NoDelim uses are.

compiler/rustc_parse/src/parser/mod.rs

petrochenkov · 2022-04-26T13:15:03Z

compiler/rustc_parse/src/parser/attr_wrapper.rs

    let mut token_and_spacing = iter.next();
    while let Some((token, spacing)) = token_and_spacing {
        match token {
            FlatToken::Token(Token { kind: TokenKind::OpenDelim(delim), span }) => {
-                stack.push(FrameData { open: span, open_delim: delim, inner: vec![] });
+                stack.push(FrameData { open_delim_sp: Some((delim, span)), inner: vec![] });
            }
            FlatToken::Token(Token { kind: TokenKind::CloseDelim(delim), span }) => {
                // HACK: If we encounter a mismatched `None` delimiter at the top
                // level, just ignore it.


Is this still necessary?
There's no longer a DelimToken::NoDelim at the top level.

There's also a similar // HACK comment lower in the same function.

The "mismatched None delimiter at the top level" here refers to token, which comes from the iterator, not the frame. Indeed, the condition looks for a non-NoDelim in the frame. So this change preserves the existing behaviour, as I understand it.

More generally: there are three HACK comments in this function, and a comment at the top of the function referring to #67062. The HACK comments are on three if statements. I tried changing the bodies of those three if statements to panic!() and the test suite passed. I then tried removing all three if statements and the test suite again passed.

These if statements were added in #82608, with an explanation here. I don't know if (a) they never did anything useful, (b) they do something useful but we have no test coverage for them, or (c) something has changed in the meantime that means they are no longer necessary.

I suggest we leave this change as is, and consider removing these hacks in a separate PR. That way if there are any regressions they'll be easier to deal with.

I have filed #96543 to remove the hacks.

compiler/rustc_parse/src/parser/attr_wrapper.rs

compiler/rustc_expand/src/mbe/transcribe.rs

The code currently ignores the actual delimiter on the RHS and fakes up a `NoDelim`/`DelimSpan::dummy()` one. This commit changes it to use the actual delimiter. The commit also reorders the fields for the `Delimited` variant to match the `Sequence` variant.

nnethercote · 2022-04-27T00:38:53Z

I have addresses the comments, except for those explained above.

petrochenkov · 2022-04-27T10:27:35Z

@bors r+

bors · 2022-04-27T10:27:36Z

📌 Commit 86f0117 has been approved by petrochenkov

…enkov Less `NoDelim` Currently there are several places where `NoDelim` (which really means "implicit delimiter" or "invisible delimiter") is used to mean "no delimiter". The name `NoDelim` is a bit misleading, and may be a cause. This PR changes these places, e.g. by changing a `DelimToken` to `Option<DelimToken>` and then using `None` to mean "no delimiter". As a result, the *only* place where `NoDelim` values are now produced is within: - `Delimiter::to_internal()`, when converting from `Delimiter::None`. - `FlattenNonterminals::process_token()`, when converting `TokenKind::Interpolated`. r? `@petrochenkov`

…enkov Less `NoDelim` Currently there are several places where `NoDelim` (which really means "implicit delimiter" or "invisible delimiter") is used to mean "no delimiter". The name `NoDelim` is a bit misleading, and may be a cause. This PR changes these places, e.g. by changing a `DelimToken` to `Option<DelimToken>` and then using `None` to mean "no delimiter". As a result, the *only* place where `NoDelim` values are now produced is within: - `Delimiter::to_internal()`, when converting from `Delimiter::None`. - `FlattenNonterminals::process_token()`, when converting `TokenKind::Interpolated`. r? ``@petrochenkov``

…enkov Less `NoDelim` Currently there are several places where `NoDelim` (which really means "implicit delimiter" or "invisible delimiter") is used to mean "no delimiter". The name `NoDelim` is a bit misleading, and may be a cause. This PR changes these places, e.g. by changing a `DelimToken` to `Option<DelimToken>` and then using `None` to mean "no delimiter". As a result, the *only* place where `NoDelim` values are now produced is within: - `Delimiter::to_internal()`, when converting from `Delimiter::None`. - `FlattenNonterminals::process_token()`, when converting `TokenKind::Interpolated`. r? ```@petrochenkov```

Rollup of 7 pull requests Successful merges: - rust-lang#96377 (make `fn() -> _ { .. }` suggestion MachineApplicable) - rust-lang#96397 (Make EncodeWide implement FusedIterator) - rust-lang#96421 (Less `NoDelim`) - rust-lang#96432 (not need `Option` for `dbg_scope`) - rust-lang#96466 (Better error messages when collecting into `[T; n]`) - rust-lang#96471 (replace let else with `?`) - rust-lang#96483 (Add missing `target_feature` to the list of well known cfg names) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

rustc_ast: Harmonize delimiter naming with `proc_macro::Delimiter` Compiler cannot reuse `proc_macro::Delimiter` directly due to extra impls, but can at least use the same naming. After this PR the only difference between these two enums is that `proc_macro::Delimiter::None` is turned into `token::Delimiter::Invisible`. It's my mistake that the invisible delimiter is called `None` on stable, during the stabilization I audited the naming and wrote the docs, but missed the fact that the `None` naming gives a wrong and confusing impression about what this thing is. cc rust-lang#96421 r? `@nnethercote`

rustc_ast: Harmonize delimiter naming with `proc_macro::Delimiter` Compiler cannot reuse `proc_macro::Delimiter` directly due to extra impls, but can at least use the same naming. After this PR the only difference between these two enums is that `proc_macro::Delimiter::None` is turned into `token::Delimiter::Invisible`. It's my mistake that the invisible delimiter is called `None` on stable, during the stabilization I audited the naming and wrote the docs, but missed the fact that the `None` naming gives a wrong and confusing impression about what this thing is. cc rust-lang#96421 r? ``@nnethercote``

rustc_ast: Harmonize delimiter naming with `proc_macro::Delimiter` Compiler cannot reuse `proc_macro::Delimiter` directly due to extra impls, but can at least use the same naming. After this PR the only difference between these two enums is that `proc_macro::Delimiter::None` is turned into `token::Delimiter::Invisible`. It's my mistake that the invisible delimiter is called `None` on stable, during the stabilization I audited the naming and wrote the docs, but missed the fact that the `None` naming gives a wrong and confusing impression about what this thing is. cc rust-lang/rust#96421 r? ``@nnethercote``

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Apr 26, 2022

rust-highfive assigned petrochenkov Apr 26, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 26, 2022

This comment has been minimized.

Sign in to view

nnethercote force-pushed the less-NoDelim branch from 73f9af9 to 7c32842 Compare April 26, 2022 08:27

petrochenkov reviewed Apr 26, 2022

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 26, 2022

petrochenkov mentioned this pull request Apr 26, 2022

rustc_ast: Harmonize delimiter naming with proc_macro::Delimiter #96433

Merged

nnethercote added 5 commits April 27, 2022 08:15

Avoid producing NoDelim values in TokenCursorFrame.

f0bbc78

Avoid producing NoDelim values in MacArgs::delim().

6b367a0

Avoid producing NoDelim values in FrameData.

9665da3

Make explicit an unreachable NoDelim case in rustfmt.

86f0117

nnethercote force-pushed the less-NoDelim branch from 7c32842 to 86f0117 Compare April 27, 2022 00:38

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 27, 2022

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 27, 2022

Dylan-DPC mentioned this pull request Apr 27, 2022

Rollup of 6 pull requests #96476

Closed

Dylan-DPC mentioned this pull request Apr 27, 2022

Rollup of 5 pull requests #96484

Closed

Dylan-DPC mentioned this pull request Apr 28, 2022

Rollup of 4 pull requests #96494

Closed

Dylan-DPC mentioned this pull request Apr 28, 2022

Rollup of 7 pull requests #96495

Merged

bors merged commit 80045d6 into rust-lang:master Apr 28, 2022

rustbot added this to the 1.62.0 milestone Apr 28, 2022

nnethercote deleted the less-NoDelim branch April 28, 2022 05:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Less `NoDelim` #96421

Less `NoDelim` #96421

nnethercote commented Apr 26, 2022

rust-highfive commented Apr 26, 2022

nnethercote commented Apr 26, 2022

This comment has been minimized.

petrochenkov Apr 26, 2022

petrochenkov Apr 26, 2022

nnethercote Apr 27, 2022

nnethercote Apr 29, 2022

nnethercote commented Apr 27, 2022

petrochenkov commented Apr 27, 2022

bors commented Apr 27, 2022

Less NoDelim #96421

Less NoDelim #96421

Conversation

nnethercote commented Apr 26, 2022

rust-highfive commented Apr 26, 2022

nnethercote commented Apr 26, 2022

This comment has been minimized.

petrochenkov Apr 26, 2022

Choose a reason for hiding this comment

petrochenkov Apr 26, 2022

Choose a reason for hiding this comment

nnethercote Apr 27, 2022

Choose a reason for hiding this comment

nnethercote Apr 29, 2022

Choose a reason for hiding this comment

nnethercote commented Apr 27, 2022

petrochenkov commented Apr 27, 2022

bors commented Apr 27, 2022

Less `NoDelim` #96421

Less `NoDelim` #96421