Keep original literal tokens in AST #60679
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bors try (for perf) |
bors
added a commit
that referenced
this pull request
May 9, 2019
Keep original literal tokens in AST The original literal tokens (`token::Lit`) are kept in AST until lowering to HIR. The tokens are kept together with their lowered "semantic" representation (`ast::LitKind`), so the size of `ast::Lit` is increased (this also increases the size of meta-item structs used for processing built-in attributes). However, the size of `ast::Expr` stays the same. The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the [lexer refactoring](#59706). Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc) Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore) This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357). cc @matklad
petrochenkov
added
the
S-waiting-on-perf
This was referenced May 9, 2019
matklad
reviewed
May 9, 2019
matklad
reviewed
May 9, 2019
|
☀️ Try build successful - checks-travis |
|
@rust-timer build 11c353b |
|
Success: Queued 11c353b with parent ef01f29, comparison URL. |
|
Finished benchmarking try commit 11c353b |
|
Perf looks acceptable. |
petrochenkov
added
S-waiting-on-author
|
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
|
Updated with two commits (one with refactoring and another moving code around). @bors delegate=matklad |
|
✌️ @matklad can now approve this pull request |
petrochenkov
added
S-waiting-on-review
matklad
reviewed
May 11, 2019
Remove some dead code
bors
added
the
S-waiting-on-review
|
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
|
Failure seems spurious? |
|
Yes, looks spurious. |
bors
added
S-waiting-on-bors
Merged
bors
added a commit
that referenced
this pull request
May 12, 2019
Keep original literal tokens in AST The original literal tokens (`token::Lit`) are kept in AST until lowering to HIR. The tokens are kept together with their lowered "semantic" representation (`ast::LitKind`), so the size of `ast::Lit` is increased (this also increases the size of meta-item structs used for processing built-in attributes). However, the size of `ast::Expr` stays the same. The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the [lexer refactoring](#59706). Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc) Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore) This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357). cc @matklad
|
☀️ Test successful - checks-travis, status-appveyor |
|
📣 Toolstate changed by #60679! Tested on commit 1764b29. 💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). |
rust-highfive
added a commit
to rust-lang-nursery/rust-toolstate
that referenced
this pull request
May 12, 2019
Tested on commit rust-lang/rust@1764b29. Direct link to PR: <rust-lang/rust#60679> 💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → build-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). 💔 rls on windows: test-pass → build-fail (cc @Xanewok, @rust-lang/infra). 💔 rls on linux: test-pass → build-fail (cc @Xanewok, @rust-lang/infra).
bors
added a commit
that referenced
this pull request
May 23, 2019
syntax: Continue refactoring literals A follow up to #60679. a2fd002: Similarly to `EscapeError`, literal parsing now produces a `LitError`. This way we can get rid of `diag: Option<(Span, &Handler)>` in interfaces while leaving attr/mod alone. d9516d1: Gathers all components of a literal token in a single struct.
Centril
added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril
added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril
added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril
added a commit
to Centril/rust
that referenced
this pull request
Jun 8, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Merged
Centril
added a commit
to Centril/rust
that referenced
this pull request
Nov 17, 2019
syntax: Keep string literals in ABIs and `asm!` more precisely As a result we don't lose spans when `extern` functions or blocks are passed to proc macros, and also escape all string literals consistently. Continuation of rust-lang#60679, which did a similar thing with all literals besides those in ABIs and `asm!`. TODO: Add tests. Fixes rust-lang#60493 Fixes rust-lang#64561 r? @Centril
JohnTitor
added a commit
to JohnTitor/rust
that referenced
this pull request
Nov 17, 2019
syntax: Keep string literals in ABIs and `asm!` more precisely As a result we don't lose spans when `extern` functions or blocks are passed to proc macros, and also escape all string literals consistently. Continuation of rust-lang#60679, which did a similar thing with all literals besides those in ABIs and `asm!`. TODO: Add tests. Fixes rust-lang#60493 Fixes rust-lang#64561 r? @Centril
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The original literal tokens (
token::Lit) are kept in AST until lowering to HIR.The tokens are kept together with their lowered "semantic" representation (
ast::LitKind), so the size ofast::Litis increased (this also increases the size of meta-item structs used for processing built-in attributes).However, the size of
ast::Exprstays the same.The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the lexer refactoring.
Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc)
Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore)
This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357).
cc @matklad