Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

quote_expr! and friends should emit token-trees directly #16472

Closed
pnkfelix opened this issue Aug 13, 2014 · 1 comment · Fixed by #23085
Closed

quote_expr! and friends should emit token-trees directly #16472

pnkfelix opened this issue Aug 13, 2014 · 1 comment · Fixed by #23085
Labels
A-syntaxext Area: Syntax extensions

Comments

@pnkfelix
Copy link
Member

The lines of code here: quote.rs#L226 are sub-optimal and also require hacks like the one described on #15962 to actually work correctly.

Effectively, the current code is implementing to_tokens by going through the transformation sequence:

AST -> string -> token-trees

The methods should instead be emitting token-trees directly from the AST, avoiding the serialization to and de-serialization from the intermediate string.
@pnkfelix pnkfelix mentioned this issue Aug 20, 2014
Closed
@kmcallister
Copy link
Contributor

A more ambitious solution would also convert the parser to operate on token trees, so that we only do string ↔ tokens ↔ token-trees ↔ AST.

@pczarn pczarn mentioned this issue Feb 26, 2015
Closed
goffrie added a commit to goffrie/rust that referenced this issue Mar 5, 2015
@goffrie
Interpolate AST nodes in quasiquote.
239775c 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
@goffrie goffrie mentioned this issue Mar 5, 2015
Merged
goffrie added a commit to goffrie/rust that referenced this issue Mar 10, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ce0a295 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Mar 13, 2015
@goffrie
Interpolate AST nodes in quasiquote.
5b8312d 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Mar 16, 2015
@goffrie
Interpolate AST nodes in quasiquote.
2a7c548 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 9, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ec6db72 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 19, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ec1b84c 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 23, 2015
@goffrie
Interpolate AST nodes in quasiquote.
d653477 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 23, 2015
@goffrie
Interpolate AST nodes in quasiquote.
7691bce 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 24, 2015
@goffrie
Interpolate AST nodes in quasiquote.
dac0f82 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 26, 2015
@goffrie
Interpolate AST nodes in quasiquote.
2d9831d 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
bors added a commit that referenced this issue Apr 26, 2015
@bors
Auto merge of #23085 - goffrie:interpolating-quote, r=huonw
6365080 
This changes the `ToTokens` implementations for expressions, statements, etc. with almost-trivial ones that produce `Interpolated(*Nt(...))` pseudo-tokens. In this way, quasiquote now works the same way as macros do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves pretty-printing at all, which removes the need for the `encode_with_hygiene` hack. All associated machinery is removed.

New `Nonterminal`s are added: NtArm, NtImplItem, and NtTraitItem. These are just for quasiquote, not macros.

`ToTokens` is no longer implemented for `Arg` (although this could be added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of `ToTokens` to turn AST fragments back into inspectable token trees. For this reason, this closes #16987.

As such, this is a [breaking-change].

Fixes #16472.
Fixes #15962.
Fixes #17397.
Fixes #16617.
bors added a commit to rust-lang-ci/rust that referenced this issue Mar 3, 2024
@bors
Auto merge of rust-lang#16630 - ShoyuVanilla:fix-closure-kind-inferen…
a3236be 
…ce, r=Veykril

fix: Wrong closure kind deduction for closures with predicates

Completes rust-lang#16472, fixes rust-lang#16421

The changed closure kind deduction is mostly simlar to `rustc_hir_typeck/src/closure.rs`.
Porting closure sig deduction from it seems possible too and I'm considering doing it with another PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-syntaxext Area: Syntax extensions
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Interpolate AST nodes in quasiquote. goffrie/rust
3 participants
@pnkfelix @kmcallister @huonw