First pass at a guide to compiler plugins

[kmcallister]

@steveklabnik, are you interested in looking this over?

[steveklabnik]

I will check it out soon. I'm in Europe so there may be some lag.

[Gankra]

Plugins manipulate ASTs while macros manipulate tokens, right? Might be worth clarifying this distinction.

[kmcallister]

Eh, in both cases the base input is a sequence of token trees. A syntax extension plugin can parse to AST types using syntax::parse, but a MBE macro can do something similar with a metavariable like $x:expr. The result of a MBE macro is a token tree, while the result of a syntax extension is a thing that maybe knows how to make an expression, maybe knows how to make a pattern, and so forth.

So it's all a bit murky and I don't think there's a clear distinction worth highlighting.

[huonw]

A procedural macro really just manipulates token trees to possibly output some AST (e.g. https://github.com/huonw/compile_msg expands to nothing meaningful); there's no particular need for it to be interpreting any part of the input as an actual Rust AST.

(I'm happy with this phrasing though.)

[steveklabnik]

I think as a first step, writing wise, this is just fine. It needs some more refinement, but before we get into that, I have a slightly bigger concern.

Is this a guide we actually really want in-tree right now? I'm glad to have this documentation exist, but these APIs, as you say, are very unstable, and are not likely to be stable any time soon. I'm not sure that giving them an official "guide" right now is the right call.

Maybe could this work as module docs in libsyntax or something? Not totally sure.

[huonw]

That seems like a pretty good idea; the macro part could be a docs in syntax::ext, the plugin stuff in rustc::plugin and the lint-specific stuff in rustc::lint. (We could have guide or doc submodules, like other places in the compiler.)

[kmcallister]

@steveklabnik, @huonw: Hm, after your comments I gave some thought to moving all this into API docs. But I think basically nobody will find it on their own, without a more inviting starting place.

Basically I see this list as a key reference for "How do I do X in Rust?" for X = testing, FFI, macros, and other core areas where any language should have a good up-front answer. I think "procedural macros" should be on that list. Witness the number of systems projects with home-grown code generators; this is a capability many systems programmers need, even when it's a tremendous pain. We have a much better story here and we shouldn't bury it in internal compiler API docs.

[huonw]

How about linking to the internal doc page in that list?

[kmcallister]

That seems okay, but at that point how is it meaningfully different from a first-class Guide? There are plenty of compiler implementation details in the other guides (e.g. here). Even though plugins involve compiler internals, the audience for a "How to write plugins" document is users of the compiler who don't necessarily plan to contribute to rustc itself. So I'd like to style this as "user-facing" documentation, similar to the other Guides.

I suppose that eventually we will have documentation for the compiler itself, separate from the language reference. This would cover things like rustc command-like syntax, and might be a natural place to describe plugins as well.

[steveklabnik]

how is it meaningfully different from a first-class Guide?

Guides are intended (at least, by me, as of now) to be a sort of cross-cutting documentation. If something only involves one module, it should really be module-level documentation.

Furthermore, I could see an argument for maybe making this a guide, but given that 1: this is really about rustc and not about Rust and 2) compiler plugins are not stable and will not be soon, now may not be the appropriate time anyway.

(also, this needs a rebase regardless)

[kmcallister]

the macro part could be a docs in syntax::ext

This involves putting rustc-specific docs in libsyntax, for example the stuff about --pretty.

Guides are intended (at least, by me, as of now) to be a sort of cross-cutting documentation. If something only involves one module, it should really be module-level documentation.

I do think this is cross-cutting — the stuff about syntax extensions covers syntax::ext, syntax::ast, rustc::plugin, and the rustc command-line interface, as well as feature-gated language syntax (quasiquotes).

[kmcallister]

Thanks for the review everyone! I rebased with changes addressing most of the above comments.

I'm still trying to think of a more compelling example of a procedural macro. It's hard to strike the right balance of interesting, short, illustrative, and useful.

[kmcallister]

Rebased, with a new syntax plugin example, and red-box warnings as suggested by @steveklabnik.

[mdinger]

Macro guide line 105

rules of tokenization apply,
// extra comma             ^

[steveklabnik]

Looks good to me overall :)

[kmcallister]

Rebased.

[steveklabnik]

I'm happy with this after that one last nit regarding relative references, and with a squash.

[kmcallister]

Squashed, with a fix for check-docs in the macro guide.

[kmcallister]

NB: it's this commit that needs the r+, GitHub is weird.