Code

More lexer improvements#102302

Code

Merged

bors merged 15 commits intorust-lang:masterrust-lang/rust:masterfrom nnethercote:more-lexer-improvementsnnethercote/rust:more-lexer-improvementsCopy head branch name to clipboardSep 28, 2022

+429-443

Commits

Commits on Sep 25, 2022

• Rearrange TokenTreesReader::parse_token_tree.

  Show description for 66e9b11
  nnethercote

  committed
  66e9b11

• Remove unnecessary spacing assignment.

  Show description for 14281e6
  nnethercote

  committed
  14281e6

Commits on Sep 26, 2022

• Move #! checking.

  Show description for 9640d1c
  nnethercote

  committed
  9640d1c

• Clarify spacing computation.

  Show description for d7928a9
  nnethercote

  committed
  d7928a9

• Remove TokenTreesReader::bump.

  Show description for 5b2075e
  nnethercote

  committed
  5b2075e

• Remove ast::Token::take.

  Show description for 33ba277
  nnethercote

  committed
  33ba277

• [ui] Rearrange StringReader/TokenTreesReader creation.

  Show description for 33516ac
  nnethercote

  committed
  33516ac

• Make rustc_lexer::cursor::Cursor public.

  Show description for aa6bfaf
  nnethercote

  committed
  aa6bfaf

• Use less DRY in cook_lexer_token.

  Show description for ceb25d1
  nnethercote

  committed
  ceb25d1

• Rename some things.

  Show description for cc0022a
  nnethercote

  committed
  cc0022a

• Add rustc_lexer::TokenKind::Eof.

  Show description for da84f0f
  nnethercote

  committed
  da84f0f

• Inline and remove cook_lexer_token.

  Show description for fb4dba0
  nnethercote

  committed
  fb4dba0

• Minor improvements.

  Show description for 880ebb6
  nnethercote

  committed
  880ebb6

Commits on Sep 27, 2022

• Rename some variables.

  Show description for 7f7e216
  nnethercote

  committed
  7f7e216

Commits on Sep 28, 2022

• Address review comments.

  nnethercote

  committed
  d0a26ac