From b6fa5e290cccc384dc79289ce5f73ddd271a0633 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 22 Aug 2018 18:09:12 -0700 Subject: [PATCH 01/24] Expand documentation on procedural macros In preparation for the 1.30 stabilization I figured I'd get started and help write some documentation! --- src/procedural-macros.md | 553 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 539 insertions(+), 14 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 993bd0e44..d0d1fa91e 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -1,24 +1,549 @@ ## Procedural Macros *Procedural macros* allow creating syntax extensions as execution of a function. -Procedural macros can be used to implement custom [derive] on your own -types. See [the book][procedural macros] for a tutorial. +Procedural macros currently come in one of three flavors: -Procedural macros involve a few different parts of the language and its -standard libraries. First is the `proc_macro` crate, included with Rust, -that defines an interface for building a procedural macro. The -`#[proc_macro_derive(Foo)]` attribute is used to mark the deriving -function. This function must have the type signature: +* Bang macros - `my_macro!(...)` +* Derive macros - `#[derive(MyTrait)]` +* Attribute macros - `#[my_attribute]` + +Procedural macros allow you to run code at compile time that operates over Rust +syntax, both consuming and producing Rust syntax. You can sort of think of +procedural macros as Rust functions from an AST to another AST. + +### Crates and procedural macros + +All procedural macros in Rust are compiled as a crate. Procedural macro crates +are defined with Cargo via the `proc-macro` key in your manfiest: + +```toml +[lib] +proc-macro = true +``` + +Procedural macros are always compiled with the same target as the compiler +itself. For example if you execute `cargo build --target +arm-unknown-linux-gnueabi` then procedural macros will not be compiled for ARM, +but rather your build computer (for example `x86_64-unknown-linux-gnu`). + +Procedural macro crates are not currently allowed to export any items except +procedural macros (we'll see how to define these in a bit). For example this +crate is not allowed: + +```rust +pub fn foo() {} +``` + +because the `foo` function is not a procedural macro. Procedural macros are +loaded dynamically by the compiler when they are needed during compilation. +Cargo will naturally make procedural macro crates available to crates which +depend on them, or you can use the `--extern` argument. + +### The `proc_macro` crate + +Procedural macro crates almost always will link to the in-tree `proc_macro` +crate. The `proc_macro` crate is a compiler-provided crate which provides +facilities to working with the types of each procedural macro function. You can +learn more about this crate by exploring [the documentation][pm-dox]. + +[pm-dox]: https://doc.rust-lang.org/stable/proc_macro/ + +Linking to the `proc_macro` crate can currently be done with: ```rust,ignore -use proc_macro::TokenStream; +extern crate proc_macro; +``` + +In the 2018 edition, however, this statement will not be necessary. + +### The `TokenStream` Type + +One aspect you may notice about the `proc_macro` crate is that it doesn't +contain any AST items! Instead, it primarily contains a `TokenStream` type. +Procedural macros in Rust operate over *token streams* instead of AST nodes, +which is a far more stable interface over time for both the compiler and for +procedural macros to target. + +A *token stream* is roughly equivalent to `Vec` where a `TokenTree` +can roughly be thought of as lexical token. For example `foo` is an `Ident` +token, `.` is a `Punct` token, and `1.2` is a `Literal` token. The `TokenStream` +type, unlike `Vec`, is cheap to clone (like `Rc`). + +To learn more about token streams, let's first dive into writing our first +procedural macro. + +### Bang Macros -#[proc_macro_derive(Hello)] -pub fn hello_world(input: TokenStream) -> TokenStream +The first kind of procedural macro is the "procedural bang macro" macro. This +flavor of procedural macro is like writing `macro_rules!` only you get to +execute arbitrary code over the input tokens instead of being limited to +`macro_rules!` syntax. + +Procedural bang macros are defined with the `#[proc_macro]` attribute and have +the following signature: + +```rust,ignore +#[proc_macro] +pub fn foo(input: TokenStream) -> TokenStream { + // ... +} +``` + +This item is defining a procedural bang macro (`#[proc_macro]`) which is called +`foo`. The first argument is the input to the macro which explore in a second, +and the return value is the tokens that it should expand to. Let's fill in all +the pieces here with a noop macro. + +First up, let's generate a skeleton project: + +```sh +$ cargo new foo +$ cd foo +$ cargo new my-macro --lib +$ echo 'my-macro = { path = "my-macro" }' >> Cargo.toml +$ echo '[lib]' >> my-macro/Cargo.toml +$ echo 'proc-macro = true' >> my-macro/Cargo.toml +``` + +This'll set up a main binary project called `foo` along with a subcrate called +`my-macro` which is declared as a procedural macro. Next up we'll fill in: + +```rust,ignore +// my-macro/src/lib.rs +extern crate proc_macro; + +use proc_macro::*; + +#[proc_macro] +pub fn foo(input: TokenStream) -> TokenStream { + input +} +``` + +And now let's invoke it: + +```rust,ignore +// src/main.rs +extern crate my_macro; + +my_macro::foo!(fn answer() -> u32 { 3 }); + +fn main() { + println!("the answer was: {}", answer()); +} ``` -Finally, procedural macros must be in their own crate, with the `proc-macro` -crate type. +and finally, build it! + +```sh +$ cargo run + Compiling my-macro v0.1.0 (file://.../foo/my-macro) + Compiling foo v0.1.0 (file://.../foo) + Finished dev [unoptimized + debuginfo] target(s) in 0.37s + Running `target/debug/foo` +the answer was: 3 +``` + +Alright! This end-to-end example shows how we can create a macro that doesn't +do anything, so let's do something a bit more useful. + +First up, let's see what the input to our macro looks like by modifying our +macro: + +```rust,ignore +// my-macro/src/lib.rs +#[proc_macro] +pub fn foo(input: TokenStream) -> TokenStream { + println!("{:#?}", input); + input +} +``` + +and reexecute (output edited slightly here): + +```sh +$ cargo run + Compiling my-macro v0.1.0 (file://.../foo/my-macro) + Compiling foo v0.1.0 (file://.../foo) +TokenStream [ + Ident { ident: "fn", span: #0 bytes(39..41) }, + Ident { ident: "answer", span: #0 bytes(42..48) }, + Group { delimiter: Parenthesis, stream: TokenStream [], span: #0 bytes(48..50) }, + Punct { ch: '-', spacing: Joint, span: #0 bytes(51..53) }, + Punct { ch: '>', spacing: Alone, span: #0 bytes(51..53) }, + Ident { ident: "u32", span: #0 bytes(54..57) }, + Group { + delimiter: Brace, + stream: TokenStream [ + Literal { lit: Integer(3), suffix: None, span: #0 bytes(60..61) } + ], + span: #0 bytes(58..63) + } +] + Finished dev [unoptimized + debuginfo] target(s) in 0.37s + Running `target/debug/foo` +the answer was: 3 +``` + +Here we can see how a procedural bang macro's input is a token stream (list) of +all the tokens provided as input to the macro itself, excluding the delimiters +used to invoke the macro. Notice how the braces and parentheses are using the +`Group` token tree which is used to enforce that macros always have balanced +delimiters. + +As you may have guessed by now the macro invocation is effectively replaced by +the return value of the macro, creating the function that we provided as input. +We can see another example of this where we simply ignore the input: + +```rust,ignore +// my-macro/src/lib.rs +#[proc_macro] +pub fn foo(_input: TokenStream) -> TokenStream { + "fn answer() -> u32 { 4 }".parse().unwrap() +} +``` + +and recompiling shows: + +```sh +$ cargo run + Compiling my-macro v0.1.0 (file://.../foo/my-macro) + Compiling foo v0.1.0 (file://.../foo) + Finished dev [unoptimized + debuginfo] target(s) in 0.37s + Running `target/debug/foo` +the answer was: 4 +``` + +showing us how the input was ignored and the macro's output was used instead. + +### Derive macros + +[The book][procedural macros] has a tutorial on creating derive macros and here +we'll go into some of the nitty-gritty of how this works. The derive macro +feature allows you to define a new `#[derive(Foo)]` mode which often makes it +much easier to systematically implement traits, removing quite a lot of +boilerplate. + +Custom derives are defined like so: + +```rust,ignore +#[proc_macro_derive(MyTrait)] +pub fn foo(item: TokenStream) -> TokenStream { + // ... +} +``` + +Here the argument to the `proc_macro_derive` attribute, `MyTrait`, is the name +of the identifier to pass to `#[derive]`. The name of the function here, `foo`, +is not currently used (but it may one day be used). + +Like procedural bang macros the input to the macro here is the item that the +attribute was applied to. Unlike bang macros, however, the output is *appended* +to the program rather than replacing the item its attached to. We can see this +behavior by defining a macro like: + +```rust,ignore +extern crate proc_macro; +use proc_macro::*; + +#[proc_macro_derive(MyTrait)] +pub fn foo(item: TokenStream) -> TokenStream { + println!("{:#?}", item); + "fn answer() -> u32 { 2 }".parse().unwrap() +} +``` + +using it liek: + +```rust,ignore +extern crate my_macro; + +use my_macro::MyTrait; + +#[derive(MyTrait)] +struct Foo; + +fn main() { + drop(Foo); + println!("the answer was: {}", answer()); +} +``` + +and compiling it: + +```sh +$ cargo run + Compiling my-macro v0.1.0 (file://.../foo/my-macro) + Compiling foo v0.1.0 (file://.../foo) +TokenStream [ + Ident { ident: "struct", span: #0 bytes(67..73) }, + Ident { ident: "Foo", span: #0 bytes(74..77) }, + Punct { ch: ';', spacing: Alone, span: #0 bytes(77..78) } +] + Finished dev [unoptimized + debuginfo] target(s) in 0.34s + Running `target/debug/foo` +the answer was: 2 +``` + +Here we can see how the input to the macro was a `TokenStream` representing +the three input tokens `struct Foo;`. While our output only contained the +`answer` function, we were still able to use `Foo` in the main program because +derive macros *append* items, they don't replace them. + +Now this is a pretty wonky macro derive, and would likely be confusing to +users! Derive macros are primarily geared towards implementing traits, like +`Serialize` and `Deserialize`. The `syn` crate also has a [number of +examples][synex] of defining derive macros. + +[procedural macros]: ../book/first-edition/procedural-macros.html +[synex]: https://github.com/dtolnay/syn/tree/master/examples + +#### Derive helper attributes + +An additional feature of derive macros is that they can whitelist names +of attributes which are considered "helper attributes" and don't participate in +normal attribute macro expansion. Taking our example from earlier we can +define: + +```rust,ignore +#[proc_macro_derive(MyTrait, attributes(my_attribute))] +pub fn foo(item: TokenStream) -> TokenStream { + // ... +} +``` + +The extra `attributes` key in the `proc_macro_derive` attribute contains a +comma-separated list of identifiers. Each identifier is a whitelist of +an attribute name that can be attached to items which also have +`#[derive(MyTrait)]`. These derive helper attributes will not do anything but +will be passed through to the `foo` procedural macro defined above as part of +the input. + +If we change our invocation to look like: + +```rust,ignore +#[derive(MyTrait)] +#[my_attribute(hello)] +struct Foo; +``` + +you'll see that the `#[my_attribute(hello)]` attribute is fed through to the +macro for processing. + +Attributes are often used to customize the behavior of derive macros, such as +the `#[serde]` attribute for the `serde` crate. + +### Attribute macros + +The third and final form of procedural macros is the attribute macro. Attribute +macros allow you to define a new `#[attr]`-style attribute which can be +attached to items and generate wrappers and such. These macros are defined like: + +```rust,ignore +#[proc_macro_attribute] +pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream { + // ... +} +``` + +The `#[proc_macro_attribute]` indicates that this macro is an attribute macro +and can only be invoked like `#[foo]`. The name of the function here will be the +name of the attribute as well. + +The first input, `attr`, is the arguments to the attribute provided, which +we'll see in a moment. The second argument, `item`, is the item that the +attribute is attached to. + +Like with bang macros at the beginning (and unlike derive macros), the return +value here *replaces* the input `item`. + +Let's see this attribute in action: + +```rust,ignore +// my-macro/src/lib.rs +extern crate proc_macro; +use proc_macro::*; + +#[proc_macro_attribute] +pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream { + println!("attr: {}", attr.to_string()); + println!("input: {}", input.to_string()); + item +} +``` + +and invoke it as: + +```rust,ignore +// src/main.rs +extern crate my_macro; + +use my_macro::foo; + +#[foo] +fn invoke1() {} + +#[foo(bar)] +fn invoke2() {} + +#[foo(crazy custom syntax)] +fn invoke3() {} + +#[foo { delimiters }] +fn invoke4() {} + +fn main() { + // ... +} +``` + +compiled as: + +```sh +$ cargo run + Compiling foo v0.1.0 (file://.../foo) +attr: +input: fn invoke1() { } +attr: bar +input: fn invoke2() { } +attr: crazy custom syntax +input: fn invoke3() { } +attr: delimiters +input: fn invoke4() { } + Finished dev [unoptimized + debuginfo] target(s) in 0.12s + Running `target/debug/foo` +``` + +Here we can see how the arguments to the attribute show up in the `attr` +argument. Notably these arguments do not include the delimiters used to enclose +the arguments (like procedural bang macros. Furthermore we can see the item +continue to operate on it, either replacing it or augmenting it. + +### Spans + +All tokens have an associated `Span`. A `Span` is currently an opaque value you +cannot modify (but you can manufacture). `Span`s represent an extent of source +code within a program and are primarily used for error reporting currently. You +can modify the `Span` of any token, and by doing so if the compiler would like +to print an error for the token in question it'll use the `Span` that you +manufactured. + +For example let's create a wonky procedural macro which swaps the spans of the +first two tokens: + +```rust,ignore +// my-macro/src/lib.rs +extern crate proc_macro; + +use proc_macro::*; + +#[proc_macro] +pub fn swap_spans(input: TokenStream) -> TokenStream { + let mut iter = input.into_iter(); + let mut a = iter.next().unwrap(); + let mut b = iter.next().unwrap(); + let a_span = a.span(); + a.set_span(b.span()); + b.set_span(a_span); + return vec![a, b].into_iter().chain(iter).collect() +} +``` + +We can see what's going on here by feeding invalid syntax into the macro and +seeing what the compiler reports. Let's start off by seeing what the compiler +does normally: + +```rust,ignore +// src/main.rs +fn _() {} +``` + +is compiled as: + +```sh +$ cargo run + Compiling foo v0.1.0 (file://.../foo) +error: expected identifier, found reserved identifier `_` + --> src/main.rs:1:4 + | +1 | fn _() {} + | ^ expected identifier, found reserved identifier + +error: aborting due to previous error +``` + +but when we feed it through our macro: + +```rust,ignore +extern crate my_macro; + +my_macro::swap_spans! { + fn _() {} +} + +fn main() {} +``` + +and compile it: + +```sh +$ cargo run + Compiling foo v0.1.0 (file://.../foo) +error: expected identifier, found reserved identifier `_` + --> src/main.rs:4:5 + | +4 | fn _() {} + | ^^ expected identifier, found reserved identifier + +error: aborting due to previous error +``` + +notice how the error message is pointing to the wrong span! This is because we +swapped the spans of the first two tokens, giving the compiler false information +about where the tokens came from. + +Controlling spans is quite a powerful feature and needs to be used with care, +misuse can lead to some excessively confusing error messages! + +### Procedural macros and hygiene + +Currently all procedural macros are "unhygienic". This means that all procedural +macros behave as if the output token stream was simply written inline to the +code it's next to. This means that it's affected by external items and also +affected by external imports and such. + +Macro authors need to be careful to ensure their macros work in as many contexts +as possible given this limitation. This often includes using absolute paths to +items in libraries (`::std::option::Option` instead of `Option`) or +by ensuring that generated functions have names that are unlikely to clash with +other functions (like `__internal_foo` instead of `foo`). + +Eventually more support for hygiene will be added to make it easier to write a +robust macro. + +### Limitations of procedural macros + +Procedural macros are not currently quite as powerful as `macro_rules!`-defined +macros in some respects. These limitations include: + +* Procedural bang macros can only be invoked in *item* contexts. For example you + can't define your own `format!` yet because that's only ever invoked in an + expression context. Put another way, procedural bang macros can only expand to + items (things like functions, impls, etc), not expressions. + + This is primarily due to the lack of hygiene with procedural macros. Once a + better hygiene story is stabilized this restriction will likely be lifted. + +* Procedural macros cannot expand to definitions of `macro_rules!` macros (none + of them). This restriction may be lifted over time but for now this is a + conservative limitation while compiler bugs are worked through and a design is + agreed upon. + +* Procedural attributes can only be attached to items, not expressions. For + example `#[my_attr] fn foo() {}` is ok but `#[my_attr] return 3` is not. This + is again due to the lack of hygiene today but this restriction may eventually + be lifted. -[derive]: attributes.html#derive -[procedural macros]: ../book/first-edition/procedural-macros.html \ No newline at end of file +* Error reporting is currently quite primitive. While an unstable diagnostic API + exists on stable your only option is to `panic!` or to in some cases expand to + an invocation of the `compile_error!` macro with a custom message. From b3758ad20bd8b6c3b9f9b134cee866a9895bf2fb Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Fri, 24 Aug 2018 22:33:37 -0700 Subject: [PATCH 02/24] Remove 'in Rust' in proc macros --- src/procedural-macros.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index d0d1fa91e..e6ab8c3e5 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -9,11 +9,11 @@ Procedural macros currently come in one of three flavors: Procedural macros allow you to run code at compile time that operates over Rust syntax, both consuming and producing Rust syntax. You can sort of think of -procedural macros as Rust functions from an AST to another AST. +procedural macros as functions from an AST to another AST. ### Crates and procedural macros -All procedural macros in Rust are compiled as a crate. Procedural macro crates +All procedural macros are compiled as a crate. Procedural macro crates are defined with Cargo via the `proc-macro` key in your manfiest: ```toml @@ -60,7 +60,7 @@ In the 2018 edition, however, this statement will not be necessary. One aspect you may notice about the `proc_macro` crate is that it doesn't contain any AST items! Instead, it primarily contains a `TokenStream` type. -Procedural macros in Rust operate over *token streams* instead of AST nodes, +Procedural macros operate over *token streams* instead of AST nodes, which is a far more stable interface over time for both the compiler and for procedural macros to target. From 50d8cf478c0445a51d2b985328be250ebb45bca1 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Fri, 24 Aug 2018 22:36:58 -0700 Subject: [PATCH 03/24] Notify cargo proc-macro crate info --- src/procedural-macros.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index e6ab8c3e5..343616af3 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -13,13 +13,15 @@ procedural macros as functions from an AST to another AST. ### Crates and procedural macros -All procedural macros are compiled as a crate. Procedural macro crates -are defined with Cargo via the `proc-macro` key in your manfiest: - -```toml -[lib] -proc-macro = true -``` +All procedural macros are compiled as a crate. + +> **Note**: When using Cargo, Procedural macro crates are defined with the +> `proc-macro` key in your manfiest: +> +> ```toml +> [lib] +> proc-macro = true +> ``` Procedural macros are always compiled with the same target as the compiler itself. For example if you execute `cargo build --target From 36d6be11f434b97adfc58786b79a40f0251563ee Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Fri, 24 Aug 2018 23:09:40 -0700 Subject: [PATCH 04/24] Clarify proc macros as crate type, not crate --- src/procedural-macros.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 343616af3..a96dd3d2c 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -13,7 +13,8 @@ procedural macros as functions from an AST to another AST. ### Crates and procedural macros -All procedural macros are compiled as a crate. +Procedural macros must be defined in a crate with the [crate type] of +`proc-macro`. > **Note**: When using Cargo, Procedural macro crates are defined with the > `proc-macro` key in your manfiest: @@ -549,3 +550,5 @@ macros in some respects. These limitations include: * Error reporting is currently quite primitive. While an unstable diagnostic API exists on stable your only option is to `panic!` or to in some cases expand to an invocation of the `compile_error!` macro with a custom message. + +[crate type]: linkage.html \ No newline at end of file From ef6ca2b5212e7022190afff6198f7b88689f8c6d Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 25 Aug 2018 02:35:30 -0700 Subject: [PATCH 05/24] First pass removal of guide-level material --- src/procedural-macros.md | 180 +++++---------------------------------- 1 file changed, 22 insertions(+), 158 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index a96dd3d2c..2b13d5584 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -39,30 +39,17 @@ pub fn foo() {} because the `foo` function is not a procedural macro. Procedural macros are loaded dynamically by the compiler when they are needed during compilation. -Cargo will naturally make procedural macro crates available to crates which -depend on them, or you can use the `--extern` argument. ### The `proc_macro` crate -Procedural macro crates almost always will link to the in-tree `proc_macro` -crate. The `proc_macro` crate is a compiler-provided crate which provides +Procedural macro crates almost always will link to the compiler-provided +`proc_macro` crate. The `proc_macro` crate is a crate which provides facilities to working with the types of each procedural macro function. You can -learn more about this crate by exploring [the documentation][pm-dox]. - -[pm-dox]: https://doc.rust-lang.org/stable/proc_macro/ - -Linking to the `proc_macro` crate can currently be done with: - -```rust,ignore -extern crate proc_macro; -``` - -In the 2018 edition, however, this statement will not be necessary. +learn more about this crate by exploring [its documentation][proc macro crate]. ### The `TokenStream` Type -One aspect you may notice about the `proc_macro` crate is that it doesn't -contain any AST items! Instead, it primarily contains a `TokenStream` type. +It primarily contains a `TokenStream` type. Procedural macros operate over *token streams* instead of AST nodes, which is a far more stable interface over time for both the compiler and for procedural macros to target. @@ -72,13 +59,9 @@ can roughly be thought of as lexical token. For example `foo` is an `Ident` token, `.` is a `Punct` token, and `1.2` is a `Literal` token. The `TokenStream` type, unlike `Vec`, is cheap to clone (like `Rc`). -To learn more about token streams, let's first dive into writing our first -procedural macro. - ### Bang Macros -The first kind of procedural macro is the "procedural bang macro" macro. This -flavor of procedural macro is like writing `macro_rules!` only you get to +This flavor of procedural macro is like writing `macro_rules!` only you get to execute arbitrary code over the input tokens instead of being limited to `macro_rules!` syntax. @@ -94,22 +77,7 @@ pub fn foo(input: TokenStream) -> TokenStream { This item is defining a procedural bang macro (`#[proc_macro]`) which is called `foo`. The first argument is the input to the macro which explore in a second, -and the return value is the tokens that it should expand to. Let's fill in all -the pieces here with a noop macro. - -First up, let's generate a skeleton project: - -```sh -$ cargo new foo -$ cd foo -$ cargo new my-macro --lib -$ echo 'my-macro = { path = "my-macro" }' >> Cargo.toml -$ echo '[lib]' >> my-macro/Cargo.toml -$ echo 'proc-macro = true' >> my-macro/Cargo.toml -``` - -This'll set up a main binary project called `foo` along with a subcrate called -`my-macro` which is declared as a procedural macro. Next up we'll fill in: +and the return value is the tokens that it should expand to. ```rust,ignore // my-macro/src/lib.rs @@ -136,20 +104,6 @@ fn main() { } ``` -and finally, build it! - -```sh -$ cargo run - Compiling my-macro v0.1.0 (file://.../foo/my-macro) - Compiling foo v0.1.0 (file://.../foo) - Finished dev [unoptimized + debuginfo] target(s) in 0.37s - Running `target/debug/foo` -the answer was: 3 -``` - -Alright! This end-to-end example shows how we can create a macro that doesn't -do anything, so let's do something a bit more useful. - First up, let's see what the input to our macro looks like by modifying our macro: @@ -162,39 +116,7 @@ pub fn foo(input: TokenStream) -> TokenStream { } ``` -and reexecute (output edited slightly here): - -```sh -$ cargo run - Compiling my-macro v0.1.0 (file://.../foo/my-macro) - Compiling foo v0.1.0 (file://.../foo) -TokenStream [ - Ident { ident: "fn", span: #0 bytes(39..41) }, - Ident { ident: "answer", span: #0 bytes(42..48) }, - Group { delimiter: Parenthesis, stream: TokenStream [], span: #0 bytes(48..50) }, - Punct { ch: '-', spacing: Joint, span: #0 bytes(51..53) }, - Punct { ch: '>', spacing: Alone, span: #0 bytes(51..53) }, - Ident { ident: "u32", span: #0 bytes(54..57) }, - Group { - delimiter: Brace, - stream: TokenStream [ - Literal { lit: Integer(3), suffix: None, span: #0 bytes(60..61) } - ], - span: #0 bytes(58..63) - } -] - Finished dev [unoptimized + debuginfo] target(s) in 0.37s - Running `target/debug/foo` -the answer was: 3 -``` - -Here we can see how a procedural bang macro's input is a token stream (list) of -all the tokens provided as input to the macro itself, excluding the delimiters -used to invoke the macro. Notice how the braces and parentheses are using the -`Group` token tree which is used to enforce that macros always have balanced -delimiters. - -As you may have guessed by now the macro invocation is effectively replaced by +The macro invocation is effectively replaced by the return value of the macro, creating the function that we provided as input. We can see another example of this where we simply ignore the input: @@ -206,26 +128,15 @@ pub fn foo(_input: TokenStream) -> TokenStream { } ``` -and recompiling shows: - -```sh -$ cargo run - Compiling my-macro v0.1.0 (file://.../foo/my-macro) - Compiling foo v0.1.0 (file://.../foo) - Finished dev [unoptimized + debuginfo] target(s) in 0.37s - Running `target/debug/foo` +``` the answer was: 4 ``` -showing us how the input was ignored and the macro's output was used instead. - ### Derive macros -[The book][procedural macros] has a tutorial on creating derive macros and here -we'll go into some of the nitty-gritty of how this works. The derive macro -feature allows you to define a new `#[derive(Foo)]` mode which often makes it -much easier to systematically implement traits, removing quite a lot of -boilerplate. +The derive macro feature allows you to define a new `#[derive(Foo)]` mode which +often makes it much easier to systematically implement traits, removing quite a +lot of boilerplate. Custom derives are defined like so: @@ -274,17 +185,7 @@ fn main() { and compiling it: -```sh -$ cargo run - Compiling my-macro v0.1.0 (file://.../foo/my-macro) - Compiling foo v0.1.0 (file://.../foo) -TokenStream [ - Ident { ident: "struct", span: #0 bytes(67..73) }, - Ident { ident: "Foo", span: #0 bytes(74..77) }, - Punct { ch: ';', spacing: Alone, span: #0 bytes(77..78) } -] - Finished dev [unoptimized + debuginfo] target(s) in 0.34s - Running `target/debug/foo` +``` the answer was: 2 ``` @@ -295,11 +196,7 @@ derive macros *append* items, they don't replace them. Now this is a pretty wonky macro derive, and would likely be confusing to users! Derive macros are primarily geared towards implementing traits, like -`Serialize` and `Deserialize`. The `syn` crate also has a [number of -examples][synex] of defining derive macros. - -[procedural macros]: ../book/first-edition/procedural-macros.html -[synex]: https://github.com/dtolnay/syn/tree/master/examples +`Serialize` and `Deserialize`. #### Derive helper attributes @@ -333,14 +230,13 @@ struct Foo; you'll see that the `#[my_attribute(hello)]` attribute is fed through to the macro for processing. -Attributes are often used to customize the behavior of derive macros, such as -the `#[serde]` attribute for the `serde` crate. +Attributes are often used to customize the behavior of derive macros. ### Attribute macros -The third and final form of procedural macros is the attribute macro. Attribute -macros allow you to define a new `#[attr]`-style attribute which can be -attached to items and generate wrappers and such. These macros are defined like: +Attribute macros allow you to define a new `#[attr]`-style attribute which can +be attached to items and generate wrappers and such. These macros are defined +like: ```rust,ignore #[proc_macro_attribute] @@ -353,9 +249,8 @@ The `#[proc_macro_attribute]` indicates that this macro is an attribute macro and can only be invoked like `#[foo]`. The name of the function here will be the name of the attribute as well. -The first input, `attr`, is the arguments to the attribute provided, which -we'll see in a moment. The second argument, `item`, is the item that the -attribute is attached to. +The first input, `attr`, is the arguments to the attribute provided. The second +argument, `item`, is the item that the attribute is attached to. Like with bang macros at the beginning (and unlike derive macros), the return value here *replaces* the input `item`. @@ -402,9 +297,7 @@ fn main() { compiled as: -```sh -$ cargo run - Compiling foo v0.1.0 (file://.../foo) +``` attr: input: fn invoke1() { } attr: bar @@ -413,8 +306,6 @@ attr: crazy custom syntax input: fn invoke3() { } attr: delimiters input: fn invoke4() { } - Finished dev [unoptimized + debuginfo] target(s) in 0.12s - Running `target/debug/foo` ``` Here we can see how the arguments to the attribute show up in the `attr` @@ -452,31 +343,6 @@ pub fn swap_spans(input: TokenStream) -> TokenStream { } ``` -We can see what's going on here by feeding invalid syntax into the macro and -seeing what the compiler reports. Let's start off by seeing what the compiler -does normally: - -```rust,ignore -// src/main.rs -fn _() {} -``` - -is compiled as: - -```sh -$ cargo run - Compiling foo v0.1.0 (file://.../foo) -error: expected identifier, found reserved identifier `_` - --> src/main.rs:1:4 - | -1 | fn _() {} - | ^ expected identifier, found reserved identifier - -error: aborting due to previous error -``` - -but when we feed it through our macro: - ```rust,ignore extern crate my_macro; @@ -505,9 +371,6 @@ notice how the error message is pointing to the wrong span! This is because we swapped the spans of the first two tokens, giving the compiler false information about where the tokens came from. -Controlling spans is quite a powerful feature and needs to be used with care, -misuse can lead to some excessively confusing error messages! - ### Procedural macros and hygiene Currently all procedural macros are "unhygienic". This means that all procedural @@ -551,4 +414,5 @@ macros in some respects. These limitations include: exists on stable your only option is to `panic!` or to in some cases expand to an invocation of the `compile_error!` macro with a custom message. -[crate type]: linkage.html \ No newline at end of file +[crate type]: linkage.html +[proc_macro crate]: ../proc_macro/index.html \ No newline at end of file From 0dd95e8d69b4daf953918fbdd8e5fcfd49cd6412 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 25 Aug 2018 02:37:04 -0700 Subject: [PATCH 06/24] Move proc-macro type sections to bottom --- src/procedural-macros.md | 202 +++++++++++++++++++-------------------- 1 file changed, 101 insertions(+), 101 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 2b13d5584..bbf26b155 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -59,6 +59,107 @@ can roughly be thought of as lexical token. For example `foo` is an `Ident` token, `.` is a `Punct` token, and `1.2` is a `Literal` token. The `TokenStream` type, unlike `Vec`, is cheap to clone (like `Rc`). +### Spans + +All tokens have an associated `Span`. A `Span` is currently an opaque value you +cannot modify (but you can manufacture). `Span`s represent an extent of source +code within a program and are primarily used for error reporting currently. You +can modify the `Span` of any token, and by doing so if the compiler would like +to print an error for the token in question it'll use the `Span` that you +manufactured. + +For example let's create a wonky procedural macro which swaps the spans of the +first two tokens: + +```rust,ignore +// my-macro/src/lib.rs +extern crate proc_macro; + +use proc_macro::*; + +#[proc_macro] +pub fn swap_spans(input: TokenStream) -> TokenStream { + let mut iter = input.into_iter(); + let mut a = iter.next().unwrap(); + let mut b = iter.next().unwrap(); + let a_span = a.span(); + a.set_span(b.span()); + b.set_span(a_span); + return vec![a, b].into_iter().chain(iter).collect() +} +``` + +```rust,ignore +extern crate my_macro; + +my_macro::swap_spans! { + fn _() {} +} + +fn main() {} +``` + +and compile it: + +```sh +$ cargo run + Compiling foo v0.1.0 (file://.../foo) +error: expected identifier, found reserved identifier `_` + --> src/main.rs:4:5 + | +4 | fn _() {} + | ^^ expected identifier, found reserved identifier + +error: aborting due to previous error +``` + +notice how the error message is pointing to the wrong span! This is because we +swapped the spans of the first two tokens, giving the compiler false information +about where the tokens came from. + +### Procedural macros and hygiene + +Currently all procedural macros are "unhygienic". This means that all procedural +macros behave as if the output token stream was simply written inline to the +code it's next to. This means that it's affected by external items and also +affected by external imports and such. + +Macro authors need to be careful to ensure their macros work in as many contexts +as possible given this limitation. This often includes using absolute paths to +items in libraries (`::std::option::Option` instead of `Option`) or +by ensuring that generated functions have names that are unlikely to clash with +other functions (like `__internal_foo` instead of `foo`). + +Eventually more support for hygiene will be added to make it easier to write a +robust macro. + +### Limitations of procedural macros + +Procedural macros are not currently quite as powerful as `macro_rules!`-defined +macros in some respects. These limitations include: + +* Procedural bang macros can only be invoked in *item* contexts. For example you + can't define your own `format!` yet because that's only ever invoked in an + expression context. Put another way, procedural bang macros can only expand to + items (things like functions, impls, etc), not expressions. + + This is primarily due to the lack of hygiene with procedural macros. Once a + better hygiene story is stabilized this restriction will likely be lifted. + +* Procedural macros cannot expand to definitions of `macro_rules!` macros (none + of them). This restriction may be lifted over time but for now this is a + conservative limitation while compiler bugs are worked through and a design is + agreed upon. + +* Procedural attributes can only be attached to items, not expressions. For + example `#[my_attr] fn foo() {}` is ok but `#[my_attr] return 3` is not. This + is again due to the lack of hygiene today but this restriction may eventually + be lifted. + +* Error reporting is currently quite primitive. While an unstable diagnostic API + exists on stable your only option is to `panic!` or to in some cases expand to + an invocation of the `compile_error!` macro with a custom message. + ### Bang Macros This flavor of procedural macro is like writing `macro_rules!` only you get to @@ -313,106 +414,5 @@ argument. Notably these arguments do not include the delimiters used to enclose the arguments (like procedural bang macros. Furthermore we can see the item continue to operate on it, either replacing it or augmenting it. -### Spans - -All tokens have an associated `Span`. A `Span` is currently an opaque value you -cannot modify (but you can manufacture). `Span`s represent an extent of source -code within a program and are primarily used for error reporting currently. You -can modify the `Span` of any token, and by doing so if the compiler would like -to print an error for the token in question it'll use the `Span` that you -manufactured. - -For example let's create a wonky procedural macro which swaps the spans of the -first two tokens: - -```rust,ignore -// my-macro/src/lib.rs -extern crate proc_macro; - -use proc_macro::*; - -#[proc_macro] -pub fn swap_spans(input: TokenStream) -> TokenStream { - let mut iter = input.into_iter(); - let mut a = iter.next().unwrap(); - let mut b = iter.next().unwrap(); - let a_span = a.span(); - a.set_span(b.span()); - b.set_span(a_span); - return vec![a, b].into_iter().chain(iter).collect() -} -``` - -```rust,ignore -extern crate my_macro; - -my_macro::swap_spans! { - fn _() {} -} - -fn main() {} -``` - -and compile it: - -```sh -$ cargo run - Compiling foo v0.1.0 (file://.../foo) -error: expected identifier, found reserved identifier `_` - --> src/main.rs:4:5 - | -4 | fn _() {} - | ^^ expected identifier, found reserved identifier - -error: aborting due to previous error -``` - -notice how the error message is pointing to the wrong span! This is because we -swapped the spans of the first two tokens, giving the compiler false information -about where the tokens came from. - -### Procedural macros and hygiene - -Currently all procedural macros are "unhygienic". This means that all procedural -macros behave as if the output token stream was simply written inline to the -code it's next to. This means that it's affected by external items and also -affected by external imports and such. - -Macro authors need to be careful to ensure their macros work in as many contexts -as possible given this limitation. This often includes using absolute paths to -items in libraries (`::std::option::Option` instead of `Option`) or -by ensuring that generated functions have names that are unlikely to clash with -other functions (like `__internal_foo` instead of `foo`). - -Eventually more support for hygiene will be added to make it easier to write a -robust macro. - -### Limitations of procedural macros - -Procedural macros are not currently quite as powerful as `macro_rules!`-defined -macros in some respects. These limitations include: - -* Procedural bang macros can only be invoked in *item* contexts. For example you - can't define your own `format!` yet because that's only ever invoked in an - expression context. Put another way, procedural bang macros can only expand to - items (things like functions, impls, etc), not expressions. - - This is primarily due to the lack of hygiene with procedural macros. Once a - better hygiene story is stabilized this restriction will likely be lifted. - -* Procedural macros cannot expand to definitions of `macro_rules!` macros (none - of them). This restriction may be lifted over time but for now this is a - conservative limitation while compiler bugs are worked through and a design is - agreed upon. - -* Procedural attributes can only be attached to items, not expressions. For - example `#[my_attr] fn foo() {}` is ok but `#[my_attr] return 3` is not. This - is again due to the lack of hygiene today but this restriction may eventually - be lifted. - -* Error reporting is currently quite primitive. While an unstable diagnostic API - exists on stable your only option is to `panic!` or to in some cases expand to - an invocation of the `compile_error!` macro with a custom message. - [crate type]: linkage.html [proc_macro crate]: ../proc_macro/index.html \ No newline at end of file From 6f694fc900f7ccf125cb071e1e3ad3f85c3f9680 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 25 Aug 2018 03:45:25 -0700 Subject: [PATCH 07/24] Improve proc_macro crate type docs --- src/linkage.md | 13 +++++++------ src/procedural-macros.md | 16 ---------------- 2 files changed, 7 insertions(+), 22 deletions(-) diff --git a/src/linkage.md b/src/linkage.md index 3e004315a..1d941dd22 100644 --- a/src/linkage.md +++ b/src/linkage.md @@ -60,12 +60,13 @@ be ignored in favor of only building the artifacts specified by command line. * `--crate-type=proc-macro`, `#[crate_type = "proc-macro"]` - The output produced is not specified, but if a `-L` path is provided to it then the compiler will recognize the output artifacts as a macro and it can be loaded - for a program. If a crate is compiled with the `proc-macro` crate type it - will forbid exporting any items in the crate other than those functions - tagged `#[proc_macro_derive]` and those functions must also be placed at the - crate root. Finally, the compiler will automatically set the - `cfg(proc_macro)` annotation whenever any crate type of a compilation is the - `proc-macro` crate type. + for a program. Crates compiled with this crate type must only export + [procedural macros]. The compiler will automatically set the `proc_macro` + [configuration option]. The crates are always compiled with the same target + that the compiler itself was built with. For example, if you are executing + the compiler from Linux with an `x86_64` CPU, the target will be + `x86_64-unknown-linux-gnu` even if the crate is a dependency of another crate + being built for a different target. Note that these outputs are stackable in the sense that if multiple are specified, then the compiler will produce each form of output at once without diff --git a/src/procedural-macros.md b/src/procedural-macros.md index bbf26b155..b0c3c4b86 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -24,22 +24,6 @@ Procedural macros must be defined in a crate with the [crate type] of > proc-macro = true > ``` -Procedural macros are always compiled with the same target as the compiler -itself. For example if you execute `cargo build --target -arm-unknown-linux-gnueabi` then procedural macros will not be compiled for ARM, -but rather your build computer (for example `x86_64-unknown-linux-gnu`). - -Procedural macro crates are not currently allowed to export any items except -procedural macros (we'll see how to define these in a bit). For example this -crate is not allowed: - -```rust -pub fn foo() {} -``` - -because the `foo` function is not a procedural macro. Procedural macros are -loaded dynamically by the compiler when they are needed during compilation. - ### The `proc_macro` crate Procedural macro crates almost always will link to the compiler-provided From 7f80970ae41dc012fafe3179a7a039c9a489d2be Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 25 Aug 2018 03:49:37 -0700 Subject: [PATCH 08/24] Clean up language in linkage. Note about compiler specifiness --- src/linkage.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/src/linkage.md b/src/linkage.md index 1d941dd22..49286cc6b 100644 --- a/src/linkage.md +++ b/src/linkage.md @@ -1,8 +1,11 @@ # Linkage -The Rust compiler supports various methods to link crates together both +> Note: This section is described more in terms of the compiler than of +> the language. + +The compiler supports various methods to link crates together both statically and dynamically. This section will explore the various methods to -link Rust crates together, and more information about native libraries can be +link crates together, and more information about native libraries can be found in the [FFI section of the book][ffi]. [ffi]: ../book/ffi.html @@ -35,7 +38,7 @@ be ignored in favor of only building the artifacts specified by command line. * `--crate-type=staticlib`, `#[crate_type = "staticlib"]` - A static system library will be produced. This is different from other library outputs in that - the Rust compiler will never attempt to link to `staticlib` outputs. The + the compiler will never attempt to link to `staticlib` outputs. The purpose of this output type is to create a static library containing all of the local crate's code along with all upstream dependencies. The static library is actually a `*.a` archive on linux and osx and a `*.lib` file on @@ -44,7 +47,7 @@ be ignored in favor of only building the artifacts specified by command line. dynamic dependencies on other Rust code. * `--crate-type=cdylib`, `#[crate_type = "cdylib"]` - A dynamic system - library will be produced. This is used when compiling Rust code as + library will be produced. This is used when compiling a dynamic library to be loaded from another language. This output type will create `*.so` files on Linux, `*.dylib` files on macOS, and `*.dll` files on Windows. @@ -52,7 +55,7 @@ be ignored in favor of only building the artifacts specified by command line. * `--crate-type=rlib`, `#[crate_type = "rlib"]` - A "Rust library" file will be produced. This is used as an intermediate artifact and can be thought of as a "static Rust library". These `rlib` files, unlike `staticlib` files, are - interpreted by the Rust compiler in future linkage. This essentially means + interpreted by the compiler in future linkage. This essentially means that `rustc` will look for metadata in `rlib` files like it looks for metadata in dynamic libraries. This form of output is used to produce statically linked executables as well as `staticlib` outputs. @@ -125,7 +128,7 @@ dependencies will be used: In general, `--crate-type=bin` or `--crate-type=lib` should be sufficient for all compilation needs, and the other options are just available if more -fine-grained control is desired over the output format of a Rust crate. +fine-grained control is desired over the output format of a crate. ## Static and dynamic C runtimes @@ -206,3 +209,7 @@ a statically linked binary on MSVC you would execute: ```ignore,notrust RUSTFLAGS='-C target-feature=+crt-static' cargo build --target x86_64-pc-windows-msvc ``` + + +[configuration option]: conditional-compilation.html +[procedural macros]: procedural-macros.html \ No newline at end of file From 6dbdc8cc5fa5d3954d9d280336c8120e0e33b52f Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 25 Aug 2018 03:50:41 -0700 Subject: [PATCH 09/24] Add 'proc_macro' to configuation options --- src/conditional-compilation.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/conditional-compilation.md b/src/conditional-compilation.md index 56835773a..7df8566e7 100644 --- a/src/conditional-compilation.md +++ b/src/conditional-compilation.md @@ -84,6 +84,8 @@ The following configurations must be defined by the implementation: This can be used to enable extra debugging code in development but not in production. For example, it controls the behavior of the standard library's `debug_assert!` macro. +* `proc_macro` - Set when the crate being compiled is being compiled with the + `proc_macro` [crate type]. You can also set another [attribute] based on a `cfg` variable with `cfg_attr`: @@ -96,4 +98,5 @@ This is the same as `#[b]` if `a` is set by `cfg`, and nothing otherwise. Lastly, configuration options can be used in expressions by invoking the `cfg!` macro: `cfg!(a)` evaluates to `true` if `a` is set, and `false` otherwise. -[attribute]: attributes.html \ No newline at end of file +[attribute]: attributes.html +[crate type]: linkage.html \ No newline at end of file From 9c0045fb5dd4ffa08c75a5d2f529533640cef585 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sun, 26 Aug 2018 02:23:03 -0700 Subject: [PATCH 10/24] Add proc_macro attributes to attributes page --- src/attributes.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/src/attributes.md b/src/attributes.md index ccb7bca64..1b3ef1efa 100644 --- a/src/attributes.md +++ b/src/attributes.md @@ -161,14 +161,20 @@ which can be used to control type layout. - `macro_reexport` on an `extern crate` — re-export the named macros. -- `macro_export` - export a macro for cross-crate usage. +- `macro_export` - export a `macro_rules` macro for cross-crate usage. - `no_link` on an `extern crate` — even if we load this crate for macros, don't link it into the output. See the [macros section of the first edition of the book](../book/first-edition/macros.html#scoping-and-macro-importexport) for more -information on macro scope. +information on `macro_rules` macro scope. + +- `proc_macro` - Defines a [bang macro]. + +- `proc_macro_derive` - Defines a [derive macro]. + +- `proc_macro_attribute` - Defines an [attribute macro]. ## Miscellaneous attributes @@ -525,9 +531,11 @@ You can implement `derive` for your own traits through [procedural macros]. [match expressions]: expressions/match-expr.html [external blocks]: items/external-blocks.html [items]: items.html +[attribute macro]: procedural-macros.html#attribute-macros +[bang macro]: procedural-macros.html#bang-macros [conditional compilation]: conditional-compilation.html -[trait]: items/traits.html -[main]: crates-and-source-files.html +[derive macro]: procedural-macros.html#derive-macros +[trait]: items/traits.html[main]: crates-and-source-files.html [`Termination`]: ../std/process/trait.Termination.html [where clause]: items/where-clauses.html [trait or lifetime bounds]: trait-bounds.html From 6851feaed719289e89e6a9491148498b56950c5c Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Wed, 29 Aug 2018 10:23:18 -0700 Subject: [PATCH 11/24] Attribute resolution, types of attrs, inert/dynamic attrs --- src/attributes.md | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/src/attributes.md b/src/attributes.md index 1b3ef1efa..2c2867fa6 100644 --- a/src/attributes.md +++ b/src/attributes.md @@ -35,8 +35,7 @@ Attributes may appear as any of: _Inner attributes_, written with a bang ("!") after the hash ("#"), apply to the item that the attribute is declared within. _Outer attributes_, written without -the bang after the hash, apply to the item or generic parameter that follow the -attribute. +the bang after the hash, apply to the thing that follows the attribute. Attributes may be applied to many things in the language: @@ -82,6 +81,47 @@ fn some_unused_variables() { } ``` +There are three kinds of attributes: + +* Built-in attributes +* Macro attributes +* Derive mode helper attributes + +## Dynamic and inert attributes + +An attribute is either dynamic or inert. During attribute processing, *dynamic +attributes* remove themselves from the thing they are on while *inert attriutes* +stay on. + +The `cfg` and `cfg_attr` attributes are dynamic. The `test` attribute is inert +when compiling for tests and dynamic otherwise. Attribute macros are dynamic. +All other attributes are inert. + +## Attribute resolution + +On all things except for items, attribute resolution is straightforward. The +`cfg` and `cfg_attr` attributes are applied and only inert attributes are +allowed. Each of those inert attributes then resolve to either a built-in +attribute or a derive mode helper attribute. If the attribute cannot resolve to +either, it is an error. + +For items, attribute resolution is a bit more involved. First off, if there are +any `cfg`, `cfg_attr`, or `test` attributes when not compiling tests, they are +resolved first, and removed from the item. Then, each attribute is checked in +the order they are written. If the attribute resolves to an inert attribute, +check the next attribute. If the attribute resolves to a dynamic attribute, then +perform its dynamic behavior. This will effectively replace the item with a new +set of items that must go through attribute resolution from the beginning. If +the attribute resolves to a macro that is not an attribute macro, it is an +error. Otherwise, the attriute is not current resolveable. In this case, wait +until another item brings the attribute into scope, and then recheck it. If all +other items have their attributes resolved or are also waiting for attribute or +derive mode resolution, it is an error. If all of the attributes are inert, then +the item is finalized. If the item defines a new path for a macro, it is now +available for other items. + +--- + The rest of this page describes or links to descriptions of which attribute names have meaning. From 690bdee0c5b12f8fa321f876abfd5088973b5138 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Wed, 29 Aug 2018 15:05:20 -0700 Subject: [PATCH 12/24] Referencify proc_macro crate section --- src/procedural-macros.md | 85 +++++++--------------------------------- 1 file changed, 14 insertions(+), 71 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index b0c3c4b86..e0053b498 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -27,79 +27,21 @@ Procedural macros must be defined in a crate with the [crate type] of ### The `proc_macro` crate Procedural macro crates almost always will link to the compiler-provided -`proc_macro` crate. The `proc_macro` crate is a crate which provides -facilities to working with the types of each procedural macro function. You can -learn more about this crate by exploring [its documentation][proc macro crate]. +[`proc_macro` crate]. The `proc_macro` crate provides types required for +writing procedural macros and facilities to make it easier. -### The `TokenStream` Type - -It primarily contains a `TokenStream` type. -Procedural macros operate over *token streams* instead of AST nodes, -which is a far more stable interface over time for both the compiler and for -procedural macros to target. - -A *token stream* is roughly equivalent to `Vec` where a `TokenTree` +This crate primarily contains a [`TokenStream`] type. Procedural macros operate +over *token streams* instead of AST nodes, which is a far more stable interface +over time for both the compiler and for procedural macros to target. A +*token stream* is roughly equivalent to `Vec` where a `TokenTree` can roughly be thought of as lexical token. For example `foo` is an `Ident` token, `.` is a `Punct` token, and `1.2` is a `Literal` token. The `TokenStream` -type, unlike `Vec`, is cheap to clone (like `Rc`). - -### Spans - -All tokens have an associated `Span`. A `Span` is currently an opaque value you -cannot modify (but you can manufacture). `Span`s represent an extent of source -code within a program and are primarily used for error reporting currently. You -can modify the `Span` of any token, and by doing so if the compiler would like -to print an error for the token in question it'll use the `Span` that you -manufactured. - -For example let's create a wonky procedural macro which swaps the spans of the -first two tokens: - -```rust,ignore -// my-macro/src/lib.rs -extern crate proc_macro; - -use proc_macro::*; - -#[proc_macro] -pub fn swap_spans(input: TokenStream) -> TokenStream { - let mut iter = input.into_iter(); - let mut a = iter.next().unwrap(); - let mut b = iter.next().unwrap(); - let a_span = a.span(); - a.set_span(b.span()); - b.set_span(a_span); - return vec![a, b].into_iter().chain(iter).collect() -} -``` - -```rust,ignore -extern crate my_macro; - -my_macro::swap_spans! { - fn _() {} -} - -fn main() {} -``` - -and compile it: - -```sh -$ cargo run - Compiling foo v0.1.0 (file://.../foo) -error: expected identifier, found reserved identifier `_` - --> src/main.rs:4:5 - | -4 | fn _() {} - | ^^ expected identifier, found reserved identifier - -error: aborting due to previous error -``` +type, unlike `Vec`, is cheap to clone. -notice how the error message is pointing to the wrong span! This is because we -swapped the spans of the first two tokens, giving the compiler false information -about where the tokens came from. +All tokens have an associated `Span`. A `Span` is an opaque value that cannot +be modified but can be manufactured. `Span`s represent an extent of source +code within a program and are primarily used for error reporting. You can modify +the `Span` of any token. ### Procedural macros and hygiene @@ -398,5 +340,6 @@ argument. Notably these arguments do not include the delimiters used to enclose the arguments (like procedural bang macros. Furthermore we can see the item continue to operate on it, either replacing it or augmenting it. -[crate type]: linkage.html -[proc_macro crate]: ../proc_macro/index.html \ No newline at end of file +[`TokenStream`]: ../proc_macro/struct.TokenStream.html +[`proc_macro` crate]: ../proc_macro/index.html +[crate type]: linkage.html \ No newline at end of file From 8bb6ca34750e1124f462a7a8f4f1f77520966ffd Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Thu, 30 Aug 2018 01:58:21 -0700 Subject: [PATCH 13/24] More referenceifying proc macros --- src/procedural-macros.md | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index e0053b498..b742da3c2 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -1,7 +1,7 @@ ## Procedural Macros *Procedural macros* allow creating syntax extensions as execution of a function. -Procedural macros currently come in one of three flavors: +Procedural macros come in one of three flavors: * Bang macros - `my_macro!(...)` * Derive macros - `#[derive(MyTrait)]` @@ -45,37 +45,28 @@ the `Span` of any token. ### Procedural macros and hygiene -Currently all procedural macros are "unhygienic". This means that all procedural -macros behave as if the output token stream was simply written inline to the -code it's next to. This means that it's affected by external items and also -affected by external imports and such. +Procedural macros are *unhygienic*. This means they behave as if the output +token stream was simply written inline to the code it's next to. This means that +it's affected by external items and also affects external imports. Macro authors need to be careful to ensure their macros work in as many contexts as possible given this limitation. This often includes using absolute paths to -items in libraries (`::std::option::Option` instead of `Option`) or +items in libraries (for example, `::std::option::Option` instead of `Option`) or by ensuring that generated functions have names that are unlikely to clash with other functions (like `__internal_foo` instead of `foo`). -Eventually more support for hygiene will be added to make it easier to write a -robust macro. - ### Limitations of procedural macros -Procedural macros are not currently quite as powerful as `macro_rules!`-defined -macros in some respects. These limitations include: - -* Procedural bang macros can only be invoked in *item* contexts. For example you - can't define your own `format!` yet because that's only ever invoked in an - expression context. Put another way, procedural bang macros can only expand to - items (things like functions, impls, etc), not expressions. +Procedural macros are not quite as powerful as `macro_rules!`-defined macros +in certain respects. These limitations include: - This is primarily due to the lack of hygiene with procedural macros. Once a - better hygiene story is stabilized this restriction will likely be lifted. +* Bang macros can only be invoked in *item* contexts. For example, + `format!` cannot yet be created in user libraries because it is only ever + invoked in an expression context. Put another way, these macros can only + expand to [items], not expressions. -* Procedural macros cannot expand to definitions of `macro_rules!` macros (none - of them). This restriction may be lifted over time but for now this is a - conservative limitation while compiler bugs are worked through and a design is - agreed upon. +* Procedural macros cannot expand to definitions of `macro_rules!` macros, with + exception to derive mode macros. * Procedural attributes can only be attached to items, not expressions. For example `#[my_attr] fn foo() {}` is ok but `#[my_attr] return 3` is not. This From 58c49c79af9ef289c8743ab1f4107786a09adc99 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Fri, 31 Aug 2018 14:00:49 -0700 Subject: [PATCH 14/24] Derive mode macros --- src/procedural-macros.md | 122 ++++++++++++++++----------------------- 1 file changed, 50 insertions(+), 72 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index b742da3c2..b8be0681c 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -3,9 +3,9 @@ *Procedural macros* allow creating syntax extensions as execution of a function. Procedural macros come in one of three flavors: -* Bang macros - `my_macro!(...)` -* Derive macros - `#[derive(MyTrait)]` -* Attribute macros - `#[my_attribute]` +* Bang macros - `custom_bang!(...)` +* Derive mode macros - `#[derive(CustomMode)]` +* Attribute macros - `#[CustomAttribute]` Procedural macros allow you to run code at compile time that operates over Rust syntax, both consuming and producing Rust syntax. You can sort of think of @@ -150,105 +150,83 @@ pub fn foo(_input: TokenStream) -> TokenStream { the answer was: 4 ``` -### Derive macros +### Derive mode macros -The derive macro feature allows you to define a new `#[derive(Foo)]` mode which -often makes it much easier to systematically implement traits, removing quite a -lot of boilerplate. +*Derive mode macros* define new modes for the `derive` attribute. The macros +define new items given the token stream of a [struct], [enum], or [union]. They +also define derive mode helper attributes. -Custom derives are defined like so: +Custom derivers are defined by a [public] [function] with the `proc_maco_derive` +attribute that takes a single input of the type [`TokenStream`] and returns a +[`TokenStream`]. -```rust,ignore -#[proc_macro_derive(MyTrait)] -pub fn foo(item: TokenStream) -> TokenStream { - // ... -} -``` - -Here the argument to the `proc_macro_derive` attribute, `MyTrait`, is the name -of the identifier to pass to `#[derive]`. The name of the function here, `foo`, -is not currently used (but it may one day be used). +The input [`TokenStream`] is the token stream of the item that has the `derive` +attribute on it. The output [`TokenStream`] must be a set of items that are +then appended to the [module] or [block] that the item from the input +[`TokenStream`] is in. -Like procedural bang macros the input to the macro here is the item that the -attribute was applied to. Unlike bang macros, however, the output is *appended* -to the program rather than replacing the item its attached to. We can see this -behavior by defining a macro like: +The following is an example of a derive mode macro. Instead of doing anything +useful with its input, it just appends a function `answer`. ```rust,ignore extern crate proc_macro; -use proc_macro::*; +use proc_macro::TokenStream; -#[proc_macro_derive(MyTrait)] -pub fn foo(item: TokenStream) -> TokenStream { - println!("{:#?}", item); - "fn answer() -> u32 { 2 }".parse().unwrap() +#[proc_macro_derive(AnswerFn)] +pub fn foo(_item: TokenStream) -> TokenStream { + "fn answer() -> u32 { 42 }".parse().unwrap() } ``` -using it liek: +And then using said derive mode: ```rust,ignore -extern crate my_macro; - -use my_macro::MyTrait; +extern crate proc_macro_examples; +use proc_macro_examples::AnswerFn; -#[derive(MyTrait)] -struct Foo; +#[derive(AnswerFn)] +struct Struct; fn main() { - drop(Foo); - println!("the answer was: {}", answer()); + assert_eq!(42, answer()); } ``` -and compiling it: +#### Derive mode helper attributes -``` -the answer was: 2 -``` - -Here we can see how the input to the macro was a `TokenStream` representing -the three input tokens `struct Foo;`. While our output only contained the -`answer` function, we were still able to use `Foo` in the main program because -derive macros *append* items, they don't replace them. +Derive mode macros can add additional [attributes] into the scope of the item +they are on. Said attributes are called *derive mode helper attributes*. These +attributes are inert, and their only purpose is to be fed into the derive +mode macro that defined them. That said, they can be seen by all macros. -Now this is a pretty wonky macro derive, and would likely be confusing to -users! Derive macros are primarily geared towards implementing traits, like -`Serialize` and `Deserialize`. +The way to define helper attributes is to put an `attributes` key in the +`proc_macro_derive` macro with a comma separated list of identifiers that are +the names of the helper attributes. -#### Derive helper attributes +For example, the following derive mode macro defines a helper attribute +`helper`, but ultimately doesn't do anything with it. -An additional feature of derive macros is that they can whitelist names -of attributes which are considered "helper attributes" and don't participate in -normal attribute macro expansion. Taking our example from earlier we can -define: +```rust, ignore +# extern crate proc_macro; +# use proc_macro::TokenStream; -```rust,ignore -#[proc_macro_derive(MyTrait, attributes(my_attribute))] -pub fn foo(item: TokenStream) -> TokenStream { - // ... +#[proc_macro_derive(HelperAttr, attributes(helper))] +pub fn derive_helper_attr(_item: TokenStream) -> TokenStream { + TokenStream::new(); } ``` -The extra `attributes` key in the `proc_macro_derive` attribute contains a -comma-separated list of identifiers. Each identifier is a whitelist of -an attribute name that can be attached to items which also have -`#[derive(MyTrait)]`. These derive helper attributes will not do anything but -will be passed through to the `foo` procedural macro defined above as part of -the input. - -If we change our invocation to look like: +And then usage on the derive mode on a struct: -```rust,ignore -#[derive(MyTrait)] -#[my_attribute(hello)] -struct Foo; ``` +# extern crate proc_macro_examples; +# use proc_macro_examples::HelperAttr; -you'll see that the `#[my_attribute(hello)]` attribute is fed through to the -macro for processing. - -Attributes are often used to customize the behavior of derive macros. +#[derive(HelperAttr)] +struct Struct { + #[helper] field: () +} +``` ### Attribute macros From caa8877ee6ed710311c04048c4f974b589422361 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 1 Sep 2018 01:56:37 -0700 Subject: [PATCH 15/24] Attribute macros --- src/procedural-macros.md | 115 ++++++++++++++++++++------------------- 1 file changed, 60 insertions(+), 55 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index b8be0681c..00d0d5d34 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -157,8 +157,7 @@ define new items given the token stream of a [struct], [enum], or [union]. They also define derive mode helper attributes. Custom derivers are defined by a [public] [function] with the `proc_maco_derive` -attribute that takes a single input of the type [`TokenStream`] and returns a -[`TokenStream`]. +attribute and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is the token stream of the item that has the `derive` attribute on it. The output [`TokenStream`] must be a set of items that are @@ -230,85 +229,91 @@ struct Struct { ### Attribute macros -Attribute macros allow you to define a new `#[attr]`-style attribute which can -be attached to items and generate wrappers and such. These macros are defined -like: +*Attribute macros* define new [attributes] which can be attached to [items]. -```rust,ignore -#[proc_macro_attribute] -pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream { - // ... -} -``` +Attribute macros are defined by a [public] [function] with the +`proc_macro_attribute` attribute that a signature of `(TokenStream, TokenStream) +-> TokenStream`. The first [`TokenStream`] is the attribute's metaitems, not +including the delimiters. If the attribute is written without a metaitem, the +attribute [`TokenStream`] is empty. The second [`TokenStream`] is of the rest of +the item including other attributes on the item. The returned [`TokenStream`] +replaces the item. It may contain an arbitrary number of items. The returned +[`TokenStream`] cannot include any [macro] definitions. + +For example, this attribute macro takes the input stream and returns it as is, +effectively being the no-op of attributes. + +```rust +#![crate_type = "proc_macro"] -The `#[proc_macro_attribute]` indicates that this macro is an attribute macro -and can only be invoked like `#[foo]`. The name of the function here will be the -name of the attribute as well. +extern crate proc_macro; -The first input, `attr`, is the arguments to the attribute provided. The second -argument, `item`, is the item that the attribute is attached to. +use proc_macro::TokenStream; -Like with bang macros at the beginning (and unlike derive macros), the return -value here *replaces* the input `item`. +#[proc_macro_attribute] +pub fn return_as_is(_attr: TokenStream, item: TokenStream) -> TokenStream { + item +} +``` -Let's see this attribute in action: +This following example shows the stringified [`TokenStream`s] that the attribute +macros see. The output will show in the output of the compiler. The output is +shown in the comments after the function prefixed with "out:". ```rust,ignore // my-macro/src/lib.rs -extern crate proc_macro; -use proc_macro::*; +# extern crate proc_macro; +# use proc_macro::TokenStream; #[proc_macro_attribute] -pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream { - println!("attr: {}", attr.to_string()); - println!("input: {}", input.to_string()); +pub fn show_streams(attr: TokenStream, input: TokenStream) -> TokenStream { + println!("attr: \"{}\"", attr.to_string()); + println!("item: \"{}\"', input.to_string()); item } ``` -and invoke it as: - ```rust,ignore -// src/main.rs +// src/lib.rs extern crate my_macro; -use my_macro::foo; +use my_macro::show_streams; -#[foo] +// Example: Basic function +#[show_streams] fn invoke1() {} +// out: attr: "" +// out: item: "fn invoke1() { }" -#[foo(bar)] +// Example: Attribute has a metaitem +#[show_streams(bar)] fn invoke2() {} +// out: attr: "bar" +// out: item: "fn invoke2() {}" -#[foo(crazy custom syntax)] +// Example: Multiple words in metaitem +#[show_streams(multiple words)] fn invoke3() {} +// out: attr: "multiple words" +// out: item: "fn invoke3() {}" -#[foo { delimiters }] +// Example: +#[show_streams { delimiters }] fn invoke4() {} - -fn main() { - // ... -} +// out: "delimiters" +// out: "fn invoke4() {}" ``` -compiled as: - -``` -attr: -input: fn invoke1() { } -attr: bar -input: fn invoke2() { } -attr: crazy custom syntax -input: fn invoke3() { } -attr: delimiters -input: fn invoke4() { } -``` - -Here we can see how the arguments to the attribute show up in the `attr` -argument. Notably these arguments do not include the delimiters used to enclose -the arguments (like procedural bang macros. Furthermore we can see the item -continue to operate on it, either replacing it or augmenting it. - [`TokenStream`]: ../proc_macro/struct.TokenStream.html +[`TokenStream`s]: ../proc_macro/struct.TokenStream.html +[`derive`]: attributes.html#derive [`proc_macro` crate]: ../proc_macro/index.html -[crate type]: linkage.html \ No newline at end of file +[attributes]: attributes.html +[custom attributes]: attributes.html +[crate type]: linkage.html +[item]: items.html +[function]: items/functions.html +[macro]: macros.html +[module]: items/modules.html +[procedural macro tutorial]: ../book/2018-edition/appendix-04-macros.html#procedural-macros-for-custom-derive +[public]: visibility-and-privacy.html \ No newline at end of file From f800c045afb6ebf86475e0c16c18c2c5bf49864e Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sat, 1 Sep 2018 12:29:55 -0700 Subject: [PATCH 16/24] Runtime information of proc macros --- src/procedural-macros.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 00d0d5d34..57c8fb198 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -11,8 +11,6 @@ Procedural macros allow you to run code at compile time that operates over Rust syntax, both consuming and producing Rust syntax. You can sort of think of procedural macros as functions from an AST to another AST. -### Crates and procedural macros - Procedural macros must be defined in a crate with the [crate type] of `proc-macro`. @@ -24,6 +22,20 @@ Procedural macros must be defined in a crate with the [crate type] of > proc-macro = true > ``` +As functions, they must either return syntax, panic, or loop endlessly. Returned +syntax either replaces or adds the syntax depending on the kind of procedural +macro. Panics are caught by the compiler and are turned into a compiler error. +Endless loops are not caught by the compiler which hangs the compiler. + +Procedural macros run during compilation, and thus have the same resources that +the compiler has. For example, standard input, error, and output are the same +that the compiler has access to. Similarly, file access is the same. Because +of this, procedural macros have the same security concerns that [Cargo's +build scripts] have. + +Procedural macros have two ways of reporting errors. The first is to panic. The +second is to emit a [`compile_error`] macro invocation. + ### The `proc_macro` crate Procedural macro crates almost always will link to the compiler-provided @@ -306,8 +318,10 @@ fn invoke4() {} [`TokenStream`]: ../proc_macro/struct.TokenStream.html [`TokenStream`s]: ../proc_macro/struct.TokenStream.html +[`compile_error`]: ../std/macro.compile_error.html [`derive`]: attributes.html#derive [`proc_macro` crate]: ../proc_macro/index.html +[Cargo's build scripts]: ../cargo/reference/build-scripts.html [attributes]: attributes.html [custom attributes]: attributes.html [crate type]: linkage.html From 19a988a166d336099be06001fc81ad60954d1c96 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sun, 2 Sep 2018 03:30:39 -0700 Subject: [PATCH 17/24] Function-like proc macros --- src/procedural-macros.md | 85 +++++++++++++--------------------------- 1 file changed, 28 insertions(+), 57 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 57c8fb198..096749c5e 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -3,7 +3,7 @@ *Procedural macros* allow creating syntax extensions as execution of a function. Procedural macros come in one of three flavors: -* Bang macros - `custom_bang!(...)` +* Function-like macros - `custom!(...)` * Derive mode macros - `#[derive(CustomMode)]` * Attribute macros - `#[CustomAttribute]` @@ -89,78 +89,47 @@ in certain respects. These limitations include: exists on stable your only option is to `panic!` or to in some cases expand to an invocation of the `compile_error!` macro with a custom message. -### Bang Macros +## Function-like procedural macros -This flavor of procedural macro is like writing `macro_rules!` only you get to -execute arbitrary code over the input tokens instead of being limited to -`macro_rules!` syntax. +Function-like procedural macros define new invokable macros. -Procedural bang macros are defined with the `#[proc_macro]` attribute and have -the following signature: +These macros are defined by a [public] [function] with the `proc_maco` +[attribute] and a signature of `(TokenStream) -> TokenStream`. The input +[`TokenStream`] is what is inside the delimiters of the macro invocation and the +output [`TokenStream`] replaces the entire macro invocation. It may contain an +arbitrary number of items. -```rust,ignore -#[proc_macro] -pub fn foo(input: TokenStream) -> TokenStream { - // ... -} -``` - -This item is defining a procedural bang macro (`#[proc_macro]`) which is called -`foo`. The first argument is the input to the macro which explore in a second, -and the return value is the tokens that it should expand to. +For example, the following macro definition ignores its input and outputs a +function `answer` into its scope. ```rust,ignore -// my-macro/src/lib.rs extern crate proc_macro; - -use proc_macro::*; +use proc_macro::TokenStream; #[proc_macro] -pub fn foo(input: TokenStream) -> TokenStream { - input +pub fn make_answer(_item: TokenStream) -> TokenStream { + "fn answer() -> u32 { 42 }".parse().unwrap() } ``` -And now let's invoke it: +And then we use it a binary crate to print "42" to standard output. ```rust,ignore -// src/main.rs -extern crate my_macro; +extern crate proc_macro_examples; +use proc_macro_examples::make_answer; -my_macro::foo!(fn answer() -> u32 { 3 }); +make_answer!(); fn main() { - println!("the answer was: {}", answer()); + println!("{}", answer()); } ``` -First up, let's see what the input to our macro looks like by modifying our -macro: - -```rust,ignore -// my-macro/src/lib.rs -#[proc_macro] -pub fn foo(input: TokenStream) -> TokenStream { - println!("{:#?}", input); - input -} -``` - -The macro invocation is effectively replaced by -the return value of the macro, creating the function that we provided as input. -We can see another example of this where we simply ignore the input: - -```rust,ignore -// my-macro/src/lib.rs -#[proc_macro] -pub fn foo(_input: TokenStream) -> TokenStream { - "fn answer() -> u32 { 4 }".parse().unwrap() -} -``` - -``` -the answer was: 4 -``` +These macros are only invokable in [modules]. They cannot even be invoked to +make [item declaration statements]. Furthermore, they must either be invoked +with curly braces and no semicolon or a different delimiter followed by a +semicolon. For example, `make_answer` from the previous example can be invoked +as `make_answer!{}`, `make_answer!();` or `make_answer![];`. ### Derive mode macros @@ -168,8 +137,8 @@ the answer was: 4 define new items given the token stream of a [struct], [enum], or [union]. They also define derive mode helper attributes. -Custom derivers are defined by a [public] [function] with the `proc_maco_derive` -attribute and a signature of `(TokenStream) -> TokenStream`. +Custom deriver modes are defined by a [public] [function] with the +`proc_maco_derive` attribute and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is the token stream of the item that has the `derive` attribute on it. The output [`TokenStream`] must be a set of items that are @@ -184,7 +153,7 @@ extern crate proc_macro; use proc_macro::TokenStream; #[proc_macro_derive(AnswerFn)] -pub fn foo(_item: TokenStream) -> TokenStream { +pub fn derive_answer_fn(_item: TokenStream) -> TokenStream { "fn answer() -> u32 { 42 }".parse().unwrap() } ``` @@ -326,8 +295,10 @@ fn invoke4() {} [custom attributes]: attributes.html [crate type]: linkage.html [item]: items.html +[item declaration statements]: statements.html#item-declarations [function]: items/functions.html [macro]: macros.html [module]: items/modules.html +[modules]: items/modules.html [procedural macro tutorial]: ../book/2018-edition/appendix-04-macros.html#procedural-macros-for-custom-derive [public]: visibility-and-privacy.html \ No newline at end of file From 0a57ca932e3ef854f509f82a1824920be8567bbe Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sun, 2 Sep 2018 03:37:18 -0700 Subject: [PATCH 18/24] Cleanup proc macros after multi-day editing --- src/procedural-macros.md | 44 +++++++++++++--------------------------- 1 file changed, 14 insertions(+), 30 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 096749c5e..58488dd5a 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -3,9 +3,9 @@ *Procedural macros* allow creating syntax extensions as execution of a function. Procedural macros come in one of three flavors: -* Function-like macros - `custom!(...)` -* Derive mode macros - `#[derive(CustomMode)]` -* Attribute macros - `#[CustomAttribute]` +* [Function-like macros] - `custom!(...)` +* [Derive mode macros] - `#[derive(CustomMode)]` +* [Attribute macros] - `#[CustomAttribute]` Procedural macros allow you to run code at compile time that operates over Rust syntax, both consuming and producing Rust syntax. You can sort of think of @@ -55,7 +55,7 @@ be modified but can be manufactured. `Span`s represent an extent of source code within a program and are primarily used for error reporting. You can modify the `Span` of any token. -### Procedural macros and hygiene +### Procedural macro hygiene Procedural macros are *unhygienic*. This means they behave as if the output token stream was simply written inline to the code it's next to. This means that @@ -67,29 +67,7 @@ items in libraries (for example, `::std::option::Option` instead of `Option`) or by ensuring that generated functions have names that are unlikely to clash with other functions (like `__internal_foo` instead of `foo`). -### Limitations of procedural macros - -Procedural macros are not quite as powerful as `macro_rules!`-defined macros -in certain respects. These limitations include: - -* Bang macros can only be invoked in *item* contexts. For example, - `format!` cannot yet be created in user libraries because it is only ever - invoked in an expression context. Put another way, these macros can only - expand to [items], not expressions. - -* Procedural macros cannot expand to definitions of `macro_rules!` macros, with - exception to derive mode macros. - -* Procedural attributes can only be attached to items, not expressions. For - example `#[my_attr] fn foo() {}` is ok but `#[my_attr] return 3` is not. This - is again due to the lack of hygiene today but this restriction may eventually - be lifted. - -* Error reporting is currently quite primitive. While an unstable diagnostic API - exists on stable your only option is to `panic!` or to in some cases expand to - an invocation of the `compile_error!` macro with a custom message. - -## Function-like procedural macros +### Function-like procedural macros Function-like procedural macros define new invokable macros. @@ -97,7 +75,8 @@ These macros are defined by a [public] [function] with the `proc_maco` [attribute] and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is what is inside the delimiters of the macro invocation and the output [`TokenStream`] replaces the entire macro invocation. It may contain an -arbitrary number of items. +arbitrary number of [items]. The returned [`TokenStream`] cannot include any +[macro] definitions. For example, the following macro definition ignores its input and outputs a function `answer` into its scope. @@ -131,9 +110,11 @@ with curly braces and no semicolon or a different delimiter followed by a semicolon. For example, `make_answer` from the previous example can be invoked as `make_answer!{}`, `make_answer!();` or `make_answer![];`. +These macros cannot expand to syntax that defines new `macro_rule` style macros. + ### Derive mode macros -*Derive mode macros* define new modes for the `derive` attribute. The macros +*Derive mode macros* define new modes for the `derive` attribute. These macros define new items given the token stream of a [struct], [enum], or [union]. They also define derive mode helper attributes. @@ -291,6 +272,9 @@ fn invoke4() {} [`derive`]: attributes.html#derive [`proc_macro` crate]: ../proc_macro/index.html [Cargo's build scripts]: ../cargo/reference/build-scripts.html +[Derive mode macros]: #derive-mode-macros +[Attribute macros]: #attribute-macros +[Function-like macros]: #function-like-procedural-macros [attributes]: attributes.html [custom attributes]: attributes.html [crate type]: linkage.html @@ -301,4 +285,4 @@ fn invoke4() {} [module]: items/modules.html [modules]: items/modules.html [procedural macro tutorial]: ../book/2018-edition/appendix-04-macros.html#procedural-macros-for-custom-derive -[public]: visibility-and-privacy.html \ No newline at end of file +[public]: visibility-and-privacy.html From 5b27fac7abaf35ffab617976c3e24a770fa8f3fe Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Sun, 2 Sep 2018 03:52:43 -0700 Subject: [PATCH 19/24] Add proc macro attrs to attrs on fns list --- src/items/functions.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/items/functions.md b/src/items/functions.md index fab09563b..92af422a4 100644 --- a/src/items/functions.md +++ b/src/items/functions.md @@ -134,7 +134,8 @@ fn test_only() { The attributes that have meaning on a function are [`cfg`], [`deprecated`], [`doc`], `export_name`, `link_section`, `no_mangle`, [the lint check -attributes], [`must_use`], [the testing attributes], and [the optimization hint +attributes], [`must_use`], [the procedural macro attributes], [the testing +attributes], and [the optimization hint attributes]. [external blocks]: items/external-blocks.html @@ -147,6 +148,7 @@ attributes]. [attributes]: attributes.html [`cfg`]: conditional-compilation.html [the lint check attributes]: attributes.html#lint-check-attributes +[the procedural macro attributes]: procedural-macros.html [the testing attributes]: attributes.html#testing [the optimization hint attributes]: attributes.html#optimization-hints [`deprecated`]: attributes.html#deprecation From f52d416060ce5d0109df520ae068f22841ab5513 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Mon, 3 Sep 2018 12:31:29 -0700 Subject: [PATCH 20/24] Remove attribute resolution for now --- src/attributes.md | 23 ----------------------- 1 file changed, 23 deletions(-) diff --git a/src/attributes.md b/src/attributes.md index 2c2867fa6..f9530af78 100644 --- a/src/attributes.md +++ b/src/attributes.md @@ -97,29 +97,6 @@ The `cfg` and `cfg_attr` attributes are dynamic. The `test` attribute is inert when compiling for tests and dynamic otherwise. Attribute macros are dynamic. All other attributes are inert. -## Attribute resolution - -On all things except for items, attribute resolution is straightforward. The -`cfg` and `cfg_attr` attributes are applied and only inert attributes are -allowed. Each of those inert attributes then resolve to either a built-in -attribute or a derive mode helper attribute. If the attribute cannot resolve to -either, it is an error. - -For items, attribute resolution is a bit more involved. First off, if there are -any `cfg`, `cfg_attr`, or `test` attributes when not compiling tests, they are -resolved first, and removed from the item. Then, each attribute is checked in -the order they are written. If the attribute resolves to an inert attribute, -check the next attribute. If the attribute resolves to a dynamic attribute, then -perform its dynamic behavior. This will effectively replace the item with a new -set of items that must go through attribute resolution from the beginning. If -the attribute resolves to a macro that is not an attribute macro, it is an -error. Otherwise, the attriute is not current resolveable. In this case, wait -until another item brings the attribute into scope, and then recheck it. If all -other items have their attributes resolved or are also waiting for attribute or -derive mode resolution, it is an error. If all of the attributes are inert, then -the item is finalized. If the item defines a new path for a macro, it is now -available for other items. - --- The rest of this page describes or links to descriptions of which attribute From 4dc54f27056a2057ce05eeb1c01ee7b469ab764b Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Mon, 3 Sep 2018 12:33:24 -0700 Subject: [PATCH 21/24] Active attributes, not dynamic --- src/attributes.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/attributes.md b/src/attributes.md index f9530af78..d5fca6c28 100644 --- a/src/attributes.md +++ b/src/attributes.md @@ -87,15 +87,15 @@ There are three kinds of attributes: * Macro attributes * Derive mode helper attributes -## Dynamic and inert attributes +## Active and inert attributes -An attribute is either dynamic or inert. During attribute processing, *dynamic +An attribute is either active or inert. During attribute processing, *active attributes* remove themselves from the thing they are on while *inert attriutes* stay on. -The `cfg` and `cfg_attr` attributes are dynamic. The `test` attribute is inert -when compiling for tests and dynamic otherwise. Attribute macros are dynamic. -All other attributes are inert. +The `cfg` and `cfg_attr` attributes are active. The `test` attribute is inert +when compiling for tests and active otherwise. Attribute macros are active. +All other attributes are inert. --- From 981a1c378a8de4727c7c50e2df38d251783ac928 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Mon, 3 Sep 2018 21:56:49 -0700 Subject: [PATCH 22/24] macos => macros --- src/procedural-macros.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 58488dd5a..934fe5419 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -71,7 +71,7 @@ other functions (like `__internal_foo` instead of `foo`). Function-like procedural macros define new invokable macros. -These macros are defined by a [public] [function] with the `proc_maco` +These macros are defined by a [public] [function] with the `proc_macro` [attribute] and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is what is inside the delimiters of the macro invocation and the output [`TokenStream`] replaces the entire macro invocation. It may contain an @@ -119,7 +119,7 @@ define new items given the token stream of a [struct], [enum], or [union]. They also define derive mode helper attributes. Custom deriver modes are defined by a [public] [function] with the -`proc_maco_derive` attribute and a signature of `(TokenStream) -> TokenStream`. +`proc_macro_derive` attribute and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is the token stream of the item that has the `derive` attribute on it. The output [`TokenStream`] must be a set of items that are From f466d186dc840467063bbfb84e5a36f695f3c9e6 Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Tue, 4 Sep 2018 23:10:12 -0700 Subject: [PATCH 23/24] Proc macros: Fix tests, links, slight rewording, consistency --- src/procedural-macros.md | 65 ++++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/src/procedural-macros.md b/src/procedural-macros.md index 934fe5419..a5d078357 100644 --- a/src/procedural-macros.md +++ b/src/procedural-macros.md @@ -69,14 +69,15 @@ other functions (like `__internal_foo` instead of `foo`). ### Function-like procedural macros -Function-like procedural macros define new invokable macros. +*Function-like procedural macros* are procedural macros that are invoked using +the macro invocation operator (`!`). -These macros are defined by a [public] [function] with the `proc_macro` +These macros are defined by a [public] [function] with the `proc_macro` [attribute] and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is what is inside the delimiters of the macro invocation and the output [`TokenStream`] replaces the entire macro invocation. It may contain an -arbitrary number of [items]. The returned [`TokenStream`] cannot include any -[macro] definitions. +arbitrary number of [items]. These macros cannot expand to syntax that defines +new `macro_rule` style macros. For example, the following macro definition ignores its input and outputs a function `answer` into its scope. @@ -105,20 +106,18 @@ fn main() { ``` These macros are only invokable in [modules]. They cannot even be invoked to -make [item declaration statements]. Furthermore, they must either be invoked +create [item declaration statements]. Furthermore, they must either be invoked with curly braces and no semicolon or a different delimiter followed by a semicolon. For example, `make_answer` from the previous example can be invoked as `make_answer!{}`, `make_answer!();` or `make_answer![];`. -These macros cannot expand to syntax that defines new `macro_rule` style macros. - ### Derive mode macros -*Derive mode macros* define new modes for the `derive` attribute. These macros -define new items given the token stream of a [struct], [enum], or [union]. They -also define derive mode helper attributes. +*Derive mode macros* define new modes for the `derive` [attribute]. These macros +define new [items] given the token stream of a [struct], [enum], or [union]. +They also define [derive mode helper attributes]. -Custom deriver modes are defined by a [public] [function] with the +Custom deriver modes are defined by a [public] [function] with the `proc_macro_derive` attribute and a signature of `(TokenStream) -> TokenStream`. The input [`TokenStream`] is the token stream of the item that has the `derive` @@ -155,9 +154,9 @@ fn main() { #### Derive mode helper attributes -Derive mode macros can add additional [attributes] into the scope of the item +Derive mode macros can add additional [attributes] into the scope of the [item] they are on. Said attributes are called *derive mode helper attributes*. These -attributes are inert, and their only purpose is to be fed into the derive +attributes are [inert], and their only purpose is to be fed into the derive mode macro that defined them. That said, they can be seen by all macros. The way to define helper attributes is to put an `attributes` key in the @@ -167,7 +166,8 @@ the names of the helper attributes. For example, the following derive mode macro defines a helper attribute `helper`, but ultimately doesn't do anything with it. -```rust, ignore +```rust,ignore +# #[crate_type="proc-macro"] # extern crate proc_macro; # use proc_macro::TokenStream; @@ -179,7 +179,8 @@ pub fn derive_helper_attr(_item: TokenStream) -> TokenStream { And then usage on the derive mode on a struct: -``` +```rust,ignore +# #![crate_type="proc-macro"] # extern crate proc_macro_examples; # use proc_macro_examples::HelperAttr; @@ -193,24 +194,23 @@ struct Struct { *Attribute macros* define new [attributes] which can be attached to [items]. -Attribute macros are defined by a [public] [function] with the -`proc_macro_attribute` attribute that a signature of `(TokenStream, TokenStream) --> TokenStream`. The first [`TokenStream`] is the attribute's metaitems, not -including the delimiters. If the attribute is written without a metaitem, the -attribute [`TokenStream`] is empty. The second [`TokenStream`] is of the rest of -the item including other attributes on the item. The returned [`TokenStream`] -replaces the item. It may contain an arbitrary number of items. The returned -[`TokenStream`] cannot include any [macro] definitions. +Attribute macros are defined by a [public] [function] with the +`proc_macro_attribute` [attribute] that a signature of +`(TokenStream, TokenStream) -> TokenStream`. The first [`TokenStream`] is the +attribute's metaitems, not including the delimiters. If the attribute is written +without a metaitem, the attribute [`TokenStream`] is empty. The second +[`TokenStream`] is of the rest of the [item] including other [attributes] on the +[item]. The returned [`TokenStream`] replaces the [item] with an arbitrary +number of [items]. These macros cannot expand to syntax that defines new +`macro_rule` style macros. For example, this attribute macro takes the input stream and returns it as is, effectively being the no-op of attributes. -```rust -#![crate_type = "proc_macro"] - -extern crate proc_macro; - -use proc_macro::TokenStream; +```rust,ignore +# #![crate_type = "proc-macro"] +# extern crate proc_macro; +# use proc_macro::TokenStream; #[proc_macro_attribute] pub fn return_as_is(_attr: TokenStream, item: TokenStream) -> TokenStream { @@ -275,14 +275,21 @@ fn invoke4() {} [Derive mode macros]: #derive-mode-macros [Attribute macros]: #attribute-macros [Function-like macros]: #function-like-procedural-macros +[attribute]: attributes.html [attributes]: attributes.html [custom attributes]: attributes.html [crate type]: linkage.html +[derive mode helper attributes]: #derive-mode-helper-attributes +[enum]: items/enumerations.html +[inert]: attributes.html#active-and-inert-attributes [item]: items.html [item declaration statements]: statements.html#item-declarations +[items]: items.html [function]: items/functions.html [macro]: macros.html [module]: items/modules.html [modules]: items/modules.html [procedural macro tutorial]: ../book/2018-edition/appendix-04-macros.html#procedural-macros-for-custom-derive [public]: visibility-and-privacy.html +[struct]: items/structs.html +[unions]: items/unions.html From 1975f829c470b009fbc6d4299afcb74bfadeea6b Mon Sep 17 00:00:00 2001 From: "Havvy (Ryan Scheel)" Date: Wed, 5 Sep 2018 13:40:40 -0700 Subject: [PATCH 24/24] Whitespace: Extra newline in linkage.md --- src/linkage.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/linkage.md b/src/linkage.md index 49286cc6b..691e46584 100644 --- a/src/linkage.md +++ b/src/linkage.md @@ -210,6 +210,5 @@ a statically linked binary on MSVC you would execute: RUSTFLAGS='-C target-feature=+crt-static' cargo build --target x86_64-pc-windows-msvc ``` - [configuration option]: conditional-compilation.html [procedural macros]: procedural-macros.html \ No newline at end of file