Document custom derive. #38770

steveklabnik · 2017-01-02T01:37:21Z

These are some bare-bones documentation for custom derive, needed
to stabilize "macros 1.1",
#35900

The book chapter is based off of a blog post by @cbreeden,
https://cbreeden.github.io/Macros11/

Normally, we have a policy of not mentioning external crates in
documentation. However, given that syn/quote are basically neccesary
for properly using macros 1.1, I feel that not including them here
would make the documentation very bad. So the rules should be bent
in this instance.

So far, this PR includes only docs; @alexcrichton said in #35900 that he'd be okay with landing them before stabilization; I don't mind either way.

rust-highfive · 2017-01-02T01:37:25Z

@steveklabnik: no appropriate reviewer found, use r? to override

steveklabnik · 2017-01-02T01:37:43Z

/cc @rust-lang/docs @rust-lang/lang

est31 · 2017-01-02T01:41:52Z

Would it make sense to document it in the reference as well? Derive is documented there too.

steveklabnik · 2017-01-02T01:47:07Z

Would it make sense to document it in the reference as well? Derive is documented there too.

Ah ha! I knew I forgot something. Yes.

killercup

Nice. Added a few inline comments with suggestions :)

killercup · 2017-01-02T06:05:31Z

src/doc/book/procedural-macros.md

+With some kind of nice output, like `Hello, World! My name is Pancakes.`. 
+
+Let's go ahead and write up what we think our macro will look like from a user
+perspective. In `hello-world\main.rs` we write:


Shouldn't that be src/main.rs?

killercup · 2017-01-02T06:06:55Z

src/doc/book/procedural-macros.md

+```
+
+Great. So now we just need to actually write the procedural macro. Let's start
+a new crate called `hello-world-macro` inside our `hello-world` project.


Maybe add a sentence about naming convention? I've seen a bunch of crates that call these <name>_derive.

killercup · 2017-01-02T06:08:02Z

src/doc/book/procedural-macros.md

+```
+
+So there is a lot going on here.  We have introduced two new crates: `syn` and
+`quote`.  As you may have noticed, `input: TokenSteam` is immediately converted


Link "syn" and "quote" to their repositories or crates.io pages?

killercup · 2017-01-02T06:08:39Z

src/doc/book/procedural-macros.md

+to a String.  This String is a string representation of the rust code for which
+we are deriving `HelloWorld` for--that's it.  So what we really need is to be
+able to _parse_ rust code into something usable.  This is where `syn` comes to
+play.  `syn` is a `nom` based rust parser, and it's extremely useful.  The


People who don't know nom will be confused by this.

killercup · 2017-01-02T06:10:27Z

src/doc/book/procedural-macros.md

+
+So there is a lot going on here.  We have introduced two new crates: `syn` and
+`quote`.  As you may have noticed, `input: TokenSteam` is immediately converted
+to a String.  This String is a string representation of the rust code for which


Lowercase "rust", same a few times below

killercup · 2017-01-02T06:12:04Z

src/doc/book/procedural-macros.md

+}
+```
+
+So there is a lot going on here.  We have introduced two new crates: `syn` and


Double space after sentence here and a few times below; I don't think the rest of the book does this.

killercup · 2017-01-02T06:12:52Z

src/doc/book/procedural-macros.md

+
+So there is a lot going on here.  We have introduced two new crates: `syn` and
+`quote`.  As you may have noticed, `input: TokenSteam` is immediately converted
+to a String.  This String is a string representation of the rust code for which


Put "String" in back ticks?

killercup · 2017-01-02T06:14:03Z

src/doc/book/procedural-macros.md

+able to _parse_ rust code into something usable.  This is where `syn` comes to
+play.  `syn` is a `nom` based rust parser, and it's extremely useful.  The
+other crate we've introduced is `quote`.  It's essentially the dual of `syn` as
+it will make serializing rust code really easy.


Hm, maybe "serializing rust code" → "generating Rust code"?

killercup · 2017-01-02T06:17:17Z

src/doc/book/procedural-macros.md

+    let s = input.to_string();
+
+    // Parse the string representation
+    let ast = syn::parse_macro_input(&s).unwrap();


Error handling probably deserves a short paragraph as well, even if it's just "panicking the proc macro code will show it as a compiler error pointing to the derive".

withoutboats · 2017-01-02T06:25:48Z

How do you feel about making HelloWorld into a trait? Implementing traits is intended as the normal use case for derive, and it seems odd to document it with a non-trait impl (though of course that does work).

steveklabnik · 2017-01-02T16:59:41Z

Okay! I think I've fixed up everything, @withoutboats @killercup . Thanks for the review 😄

Still have to update the reference, should push that in a minute.

killercup · 2017-01-02T17:03:04Z

src/doc/book/procedural-macros.md


 ```bash
 error: the `#[proc_macro_derive]` attribute is only usable with crates of the `proc-macro` crate type
- --> hello-world-macro\src\lib.rs:8:3
+ --> hello-world-derive\src\lib.rs:8:3


FYI this path uses backslashes, the rest of the book probably not

frewsxcv · 2017-01-02T17:41:51Z

src/doc/reference.md


 * [Macros](book/macros.html) define new syntax in a higher-level,
  declarative way.
+* [Procedural Macros][procedural macros] can be used to implement custom derive.

 ## Macros


It's a little weird to me that ## Macros is a subheading within # Macros, but I see why you did it

yeah i would like to re-do this and other parts of the reference but i'm trying to be minimally invasive here

frewsxcv · 2017-01-02T17:44:35Z

src/doc/reference.md

@@ -2319,6 +2337,8 @@ impl<T: PartialEq> PartialEq for Foo<T> {
 }
 ```

+You can implement derive for your own type through [procedural macros](#procedural-macros).


Earlier in this section, "derive" is styled as

`derive`

with backticks, so might be good to be consistent.

frewsxcv · 2017-01-02T17:46:02Z

src/doc/reference.md

+* `--crate-type=proc-macro`, `#[crate_type = "proc-macro"]` - The output
+  produced is not specified, but if a `-L` path is provided to it then the
+  compiler will recognize the output artifacts as a macro and it can be loaded
+  for a program. If a crate is compiled with the proc-macro crate type t will


s/t will/it will/

It looks like crate types are styled with backticks, so also:

s/proc-macro/`proc-macro`/

frewsxcv · 2017-01-02T17:48:08Z

src/doc/reference.md

+  forbid exporting any items in the crate other than those functions tagged
+  `#[proc_macro_derive]` and those functions must also be placed at the crate
+  root. Finally, the compiler will automatically set the `cfg(proc_macro)`
+  annotation whenever any crate type of a compilation is the rustc-macro crate


s/rustc-macro/`rustc-macro`/

nikomatsakis

This seemed like a decent summary. The one thing I would say is that I would have expected a bit more up-front material discussing the 'basic model' for derive (string in, string out, you do some parsing and serialization in the model), but that's because I tend towards more of a "top-down" style, I think, whereas this is kind of "bottom-up". (i.e., here is the code to write, let's explain why you wrote it) In any case, all the important stuff seems to be there, and it reads pretty smoothly.

I left some nits.

nikomatsakis · 2017-01-02T19:49:42Z

src/doc/book/procedural-macros.md

+
+Rust includes several traits that you can derive, but it also lets you define
+your own. We can accomplish this task through a feature of Rust called
+"procedrual macros." Eventually, procedural macros will allow for all sorts of


typo: procedural

nikomatsakis · 2017-01-02T19:54:20Z

src/doc/book/procedural-macros.md

+type using `ast.ident`. The `quote!` macro let's us write up the Rust code
+that we wish to return and convert it into `Tokens`. `quote!` let's us use some
+really cool templating mechanics; we simply write `#name` and `quote!` will
+replace it with the variable named `name`. You can even do some repition


Nit: s/repition/repetition/

nikomatsakis · 2017-01-02T19:55:50Z

src/doc/reference.md

@@ -652,6 +648,28 @@ Rust syntax is restricted in two ways:

 [RFC 550]: https://github.com/rust-lang/rfcs/blob/master/text/0550-macro-future-proofing.md

+## Procedrual Macros


Nit: s/Procedrual/Procedural/

nikomatsakis · 2017-01-02T19:56:27Z

src/doc/book/procedural-macros.md

@@ -0,0 +1,209 @@
+% Procedrual Macros (and custom Derive)


Nit: s/Procedrual/Procedural/

i am... really bad at this spelling apparently, haha

nikomatsakis · 2017-01-02T19:58:44Z

src/doc/book/procedural-macros.md

+```
+
+Oh, so it appears that we need to declare that our `hello-world-derive` crate is
+a `proc-macro` crate type. How do we do this? Like this:


This stream of consciousness bit felt a little over the top to me, but it's fine. =)

steveklabnik · 2017-01-02T20:32:25Z

The one thing I would say is that I would have expected a bit more up-front material discussing the 'basic model' for derive

I'm not totally opposed to this; I was thinking about making it future proof. That is, right now it's string in string out because it's a hack. It's really TokenStream -> String -> process -> String -> TokenStream, and ideally those string bits are going away. But I guess it will be a long time...

chris-morgan · 2017-01-03T03:28:11Z

src/doc/book/SUMMARY.md

@@ -52,6 +52,7 @@
    * [Borrow and AsRef](borrow-and-asref.md)
    * [Release Channels](release-channels.md)
    * [Using Rust without the standard library](using-rust-without-the-standard-library.md)
+    * [Procedural Macros (and custom Derive)](procedural-macros.md)


I see we’re continuing the tradition of inconsistent capitalisation—nay, taking it to a new level, with mixed title and sentence case all in one line!

Yeah "Derive" felt like something that deserved a capital here, maybe not, idk

F001 · 2017-01-03T03:45:08Z

Thanks for writing the doc. I can get the basic concept of proc-macro now, but still haunted by some questions.

Is it possible to write this macro without any third party dependency? What if we create it from scratch? What kind of benefit we can get from syn&quote crate?
Why does TokenStream type exist, if the only purpose of it is to be converted to a String? It is weird to convert a structured type to a String, and then parse. I would like to see more explanation about what kind of API does the proc-macro feature provide for us, not the API of a third party lib.

withoutboats · 2017-01-03T06:19:07Z

src/doc/reference.md


 A number of minor features of Rust are not central enough to have their own
 syntax, and yet are not implementable as functions. Instead, they are given
 names, and invoked through a consistent syntax: `some_extension!(...)`.

-Users of `rustc` can define new syntax extensions in two ways:
-
-* [Compiler plugins][plugin] can include arbitrary Rust code that


We should probably continue to document compiler plugins in the reference for the time being; they still exist & will for some time, and of course some project are heavily reliant on them.

This isn't removing documentation; it's re-ordering it. I thought putting macros by example first was a better way of doing things.

It looks like it went from Compiler plugins, Macros to Macros, Procedural Macros, shouldn't it be Macros, Procedural Macros, Compiler plugins?

My understanding is that procedural macros are compiler plugins. That is, today "compiler plugins" are "using libsyntax to extend the compiler" but that's never going to be stable, only the new interface we're calling "procedural macros"

The names of all of this stuff has been very hard.

That's correct, but we've implemented (and now stabilized) the new procedural macros interface only for controlling derive macros. Users who want to define custom attributes or function style procedural macros (without using a hack) are still using the never-to-be-stabilized compiler plugin interface.

Until we've fully implemented procedural macros, compiler plugins will still be used by these users and so we shouldn't drop whatever docs we have on it (even if we don't focus any attention on improving it).

I didn't actually remove any of those docs generally, just the small description here. Let me fix it.

parir · 2017-01-03T12:57:54Z

src/doc/book/procedural-macros.md

+```
+
+To make sure that our `hello-world` crate is able to find this new crate we've
+crated, we'll add it to our toml:


s/crated/created/

steveklabnik · 2017-01-03T15:54:30Z

@F001

Is it possible to write this macro without any third party dependency? What if we create it from scratch? What kind of benefit we can get from syn&quote crate?

Sure, but it's much, much, much harder. The benefits are that syn and quote do parsing and generation for you.

Why does TokenStream type exist, if the only purpose of it is to be converted to a String? It is weird to convert a structured type to a String, and then parse.

This is the difference between macros 1.1 and macros 2.0, in a nutshell.

I would like to see more explanation about what kind of API does the proc-macro feature provide for us, not the API of a third party lib.

This is the entire API. That's it. There's nothing else to explain. It is extremely bare-bones.

steveklabnik · 2017-01-03T15:59:00Z

Okay, I believe that I've addressed everyone's comments now, both on GitHub and in the text. I've squashed, since this is going to need to be backported, and want to make that easy.

nikomatsakis · 2017-01-03T18:16:48Z

Let's do it.

@bors r+

bors · 2017-01-03T18:16:50Z

📌 Commit eb2c83f has been approved by nikomatsakis

nikomatsakis · 2017-01-03T18:16:53Z

@bors rollup

steveklabnik · 2017-01-03T22:17:06Z

@bors: r=nikomatsakis rollup (#38770 (comment))

bors · 2017-01-03T22:17:08Z

📌 Commit c0efdbf has been approved by nikomatsakis

@cbreeden

…ikomatsakis Document custom derive. These are some bare-bones documentation for custom derive, needed to stabilize "macros 1.1", rust-lang#35900 The book chapter is based off of a blog post by @cbreeden, https://cbreeden.github.io/Macros11/ Normally, we have a policy of not mentioning external crates in documentation. However, given that syn/quote are basically neccesary for properly using macros 1.1, I feel that not including them here would make the documentation very bad. So the rules should be bent in this instance. So far, this PR includes only docs; @alexcrichton said in rust-lang#35900 that he'd be okay with landing them before stabilization; I don't mind either way.

alexcrichton · 2017-01-04T22:01:44Z

@bors: rollup- p=1

Let's land this quickly so we can backport

bors · 2017-01-04T22:08:06Z

🔒 Merge conflict

bors · 2017-01-04T22:11:06Z

☔ The latest upstream changes (presumably #38783) made this pull request unmergeable. Please resolve the merge conflicts.

@cbreeden

These are some bare-bones documentation for custom derive, needed to stabilize "macros 1.1", rust-lang#35900 The book chapter is based off of a blog post by @cbreeden, https://cbreeden.github.io/Macros11/ Normally, we have a policy of not mentioning external crates in documentation. However, given that syn/quote are basically neccesary for properly using macros 1.1, I feel that not including them here would make the documentation very bad. So the rules should be bent in this instance.

steveklabnik · 2017-01-04T22:35:06Z

@bors: r=alexcrichton p=1

bors · 2017-01-04T22:35:07Z

📌 Commit 3075c1f has been approved by alexcrichton

sgrif · 2017-01-04T22:48:20Z

src/doc/book/procedural-macros.md

+moment, procedural macros need to be in their own crate. Eventually, this
+restriction may be lifted, but for now, it's required. As such, there's a
+convention; for a crate named `foo`, a custom derive procedural macro is called
+`foo-derive`. Let's start a new crate called `hello-world-derive` inside our


I think the convention is foo_derive not foo-derive.

package names are supposed to prefer - over _.

bors · 2017-01-04T23:04:55Z

⌛ Testing commit 3075c1f with merge 8b9bc29...

bors · 2017-01-04T23:08:51Z

💔 Test failed - status-travis

steveklabnik · 2017-01-04T23:21:10Z

@bors: retry

(travis is spurious)

steveklabnik · 2017-01-04T23:23:45Z

@bors: force

bors · 2017-01-05T00:11:18Z

⌛ Testing commit 3075c1f with merge 5d994d8...

@cbreeden

Document custom derive. These are some bare-bones documentation for custom derive, needed to stabilize "macros 1.1", #35900 The book chapter is based off of a blog post by @cbreeden, https://cbreeden.github.io/Macros11/ Normally, we have a policy of not mentioning external crates in documentation. However, given that syn/quote are basically neccesary for properly using macros 1.1, I feel that not including them here would make the documentation very bad. So the rules should be bent in this instance. So far, this PR includes only docs; @alexcrichton said in #35900 that he'd be okay with landing them before stabilization; I don't mind either way.

bors · 2017-01-05T02:10:38Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 5d994d8 to master...

killercup reviewed Jan 2, 2017

View reviewed changes

killercup approved these changes Jan 2, 2017

View reviewed changes

frewsxcv reviewed Jan 2, 2017

View reviewed changes

nikomatsakis approved these changes Jan 2, 2017

View reviewed changes

alexcrichton added the beta-nominated Nominated for backporting to the compiler in the beta channel. label Jan 2, 2017

chris-morgan reviewed Jan 3, 2017

View reviewed changes

withoutboats reviewed Jan 3, 2017

View reviewed changes

parir reviewed Jan 3, 2017

View reviewed changes

steveklabnik force-pushed the doc-custom-derive branch from eb2c83f to 16f76c4 Compare January 3, 2017 15:58

nikomatsakis added the beta-accepted Accepted for backporting to the compiler in the beta channel. label Jan 3, 2017

steveklabnik force-pushed the doc-custom-derive branch from 16f76c4 to c0efdbf Compare January 3, 2017 22:16

GuillaumeGomez mentioned this pull request Jan 4, 2017

Rollup of 9 pull requests #38826

Closed

steveklabnik force-pushed the doc-custom-derive branch from c0efdbf to 3075c1f Compare January 4, 2017 22:34

sgrif reviewed Jan 4, 2017

View reviewed changes

bors merged commit 3075c1f into rust-lang:master Jan 5, 2017

nikomatsakis mentioned this pull request Jan 6, 2017

Beta backports #38872

Merged

17 tasks

alexcrichton removed the beta-nominated Nominated for backporting to the compiler in the beta channel. label Jan 6, 2017

		@@ -652,6 +648,28 @@ Rust syntax is restricted in two ways:

		[RFC 550]: https://github.com/rust-lang/rfcs/blob/master/text/0550-macro-future-proofing.md

		## Procedrual Macros

Document custom derive. #38770

Document custom derive. #38770

Conversation

steveklabnik commented Jan 2, 2017

rust-highfive commented Jan 2, 2017

steveklabnik commented Jan 2, 2017

est31 commented Jan 2, 2017 • edited Loading

steveklabnik commented Jan 2, 2017

killercup left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

killercup Jan 2, 2017 • edited Loading

Choose a reason for hiding this comment

withoutboats commented Jan 2, 2017

steveklabnik commented Jan 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveklabnik commented Jan 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

F001 commented Jan 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveklabnik Jan 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveklabnik commented Jan 3, 2017

steveklabnik commented Jan 3, 2017

nikomatsakis commented Jan 3, 2017

bors commented Jan 3, 2017

nikomatsakis commented Jan 3, 2017

steveklabnik commented Jan 3, 2017

bors commented Jan 3, 2017

alexcrichton commented Jan 4, 2017

bors commented Jan 4, 2017

bors commented Jan 4, 2017

steveklabnik commented Jan 4, 2017

bors commented Jan 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Jan 4, 2017

bors commented Jan 4, 2017

steveklabnik commented Jan 4, 2017

steveklabnik commented Jan 4, 2017

bors commented Jan 5, 2017

bors commented Jan 5, 2017

est31 commented Jan 2, 2017 •

edited

Loading

killercup Jan 2, 2017 •

edited

Loading

F001 commented Jan 3, 2017 •

edited

Loading

steveklabnik Jan 3, 2017 •

edited

Loading