RFC: Externally loadable syntax extensions #11151

sfackler · 2013-12-26T16:32:09Z

Example

Here's a silly example showing off the basics:

my_synext.rs

#[feature(managed_boxes, globs, macro_registrar, macro_rules)];

extern mod syntax;

use syntax::ast::{Name, token_tree};
use syntax::codemap::Span;
use syntax::ext::base::*;
use syntax::parse::token;

#[macro_export]
macro_rules! exported_macro (() => (2))

#[macro_registrar]
pub fn macro_registrar(register: |Name, SyntaxExtension|) {
    register(token::intern(&"make_a_1"),
        NormalTT(@SyntaxExpanderTT {
            expander: SyntaxExpanderTTExpanderWithoutContext(expand_make_a_1),
            span: None,
        } as @SyntaxExpanderTTTrait,
        None));
}

pub fn expand_make_a_1(cx: &mut ExtCtxt, sp: Span, tts: &[token_tree]) -> MacResult {
    if !tts.is_empty() {
        cx.span_fatal(sp, "make_a_1 takes no arguments");
    }
    MRExpr(quote_expr!(cx, 1i))
}

main.rs:

#[feature(phase)];

#[phase(syntax)]
extern mod my_synext;

fn main() {
    assert_eq!(1, make_a_1!());
    assert_eq!(2, exported_macro!());
}

Overview

Crates that contain syntax extensions need to define a function with the following signature and annotation:

#[macro_registrar]
pub fn registrar(register: |ast::Name, ext::base::SyntaxExtension|) { ... }

that should call the register closure with each extension it defines. macro_rules! style macros can be tagged with #[macro_export] to be exported from the crate as well.

Crates that wish to use externally loadable syntax extensions load them by adding the #[phase(syntax)] attribute to an extern mod. All extensions registered by the specified crate are loaded with the same scoping rules as macro_rules! macros. If you want to use a crate both for syntax extensions and normal linkage, you can use #[phase(syntax, link)].

Open questions

Does the macro_crate syntax make sense? It wraps an entire extern mod declaration which looks a bit weird but is nice in the sense that the crate lookup logic can be identical between normal external crates and external macro crates. If the extern mod syntax, changes, this will get it for free, etc. Changed to a phase attribute.
~~Is the magic name macro_crate_registration the right way to handle extension registration? It could alternatively be handled by a function annotated with #[macro_registration] I guess.~~ Switched to an attribute.
The crate loading logic lives inside of librustc, which means that the syntax extension infrastructure can't directly access it. I've worked around this by passing a CrateLoader trait object from the driver to libsyntax that can call back into the crate loading logic. It should be possible to pull things apart enough that this isn't necessary anymore, but it will be an enormous refactoring project. I think we'll need to create a couple of new libraries: libsynext libmetadata/ty and libmiddle.
Item decorator extensions can be loaded but the deriving decorator itself can't be extended so you'd need to do e.g. #[deriving_MyTrait] #[deriving(Clone)] instead of #[deriving(MyTrait, Clone)]. Is this something worth bothering with for now?

Remaining work

~~There is not yet support for rustdoc downloading and compiling referenced macro crates as it does for other referenced crates. This shouldn't be too hard I think.~~
This is not testable at stage1 and sketchily testable at stages above that. The stage n rustc links against the stage n-1 libsyntax and librustc. Unfortunately, crates in the test/auxiliary directory link against the stage n libstd, libextra, libsyntax, etc. This causes macro crates to fail to properly dynamically link into rustc since names end up being mangled slightly differently. In addition, when rustc is actually installed onto a system, there are actually do copies of libsyntax, libstd, etc: the ones that user code links against and a separate set from the previous stage that rustc itself uses. By this point in the bootstrap process, the two library versions should probably be binary compatible, but it doesn't seem like a sure thing. Fixing this is apparently hard, but necessary to properly cross compile as well and is being tracked in Don't promote binaries between stages #11145. The offending tests are ignored during check-stage1-rpass and check-stage1-cfail. When we get a snapshot that has this commit, I'll look into how feasible it'll be to get them working on stage1.
macro_rules! style macros aren't being exported. Now that the crate loading infrastructure is there, this should just require serializing the AST of the macros into the crate metadata and yanking them out again, but I'm not very familiar with that part of the compiler.
The macro_crate_registration function isn't type-checked when it's loaded. I poked around in the csearch infrastructure a bit but didn't find any super obvious ways of checking the type of an item with a certain name. Fixing this may also eliminate the need to #[no_mangle] the registration function. Now that the registration function is identified by an attribute, typechecking this will be like typechecking other annotated functions.
~~The dynamic libraries that are loaded are never unloaded. It shouldn't require too much work to tie the lifetime of the DynamicLibrary object to the MapChain that its extensions are loaded into.~~
The compiler segfaults sometimes when loading external crates. The DynamicLibrary reference and code objects from that library are both put into the same hash table. When the table drops, due to the random ordering the library sometimes drops before the objects do. Once Rewrite SyntaxEnv #11228 lands it'll be easy to fix this.

pcwalton · 2013-12-26T21:05:50Z

I would like to have macros in external crates be discovered and registered automatically by tagging them with an attribute.

Instead of macro_crate!, I would rather have something like #[phase(syntax)] extern mod foo; This makes the system more like "syntax phases" from Racket. What I had in mind was for #[phase(syntax)] to later be extended to allow functions to be tagged with #[phase(syntax)], which would allow compile time function evaluation.

pcwalton · 2013-12-26T21:17:06Z

I'm glad that you started on procedural macros instead of macro_rules!. I'd eventually like to deprecate macro_rules! in its current form and make it just a way to generate procedural macros.

sfackler · 2013-12-26T22:45:55Z

Sounds good to me. Would the corresponding phase if I want to use an external crate in the "normal" way be link? i.e. if I have a crate I want to use both at compile time and run time, I'd do #[phase(syntax, link)] extern mod foo;.

sfackler · 2013-12-27T00:46:37Z

@pcwalton updated

huonw · 2013-12-30T11:43:16Z

Item decorator extensions can be loaded but the deriving decorator itself can't be extended so you'd need to do e.g. #[deriving_MyTrait] #[deriving(Clone)] instead of #[deriving(MyTrait, Clone)]. Is this something worth bothering with for now?

I've been thinking about how to do this for a while now, and I think the best strategy is to have a hashmap of trait name -> deriving function stored in the ExtCtxt, and allow phase(syntax) things to add to this map. (I don't think the current deriving interface is good enough though, nor do I think that this needs to be done in this PR.)

huonw · 2013-12-30T11:45:09Z

src/librustc/metadata/creader.rs

+    let should_load = i.attrs.iter().all(|attr| {
+        "phase" != attr.name() ||
+            attr.meta_item_list().map_default(false, |phases| {
+                phases.iter().any(|phase| "link" == phase.name())


This can be syntax::attr::contains_name(phases, "link");.

Done, thanks

huonw · 2013-12-30T11:50:03Z

This needs a rebase. Also I think this should either be behind a -Z flag, or behind a #[feature] (e.g. #[feature(phases)] will allow the use of the #[phase] attribute on extern mods) since it's so easy for users to get it wrong (e.g. slightly incorrect definition of macro_registration_function and you'll get segfaults and other weirdness on some platforms, but maybe not on others).

sfackler · 2013-12-30T16:02:36Z

@huonw agreed. I think that #[feature(phases)] would be the way to go since once-fn support went from a -Z flag to a feature gate.

pcwalton · 2013-12-31T17:53:35Z

This is really awesome work! Sorry for not noticing the update earlier.

sfackler · 2014-01-04T16:21:43Z

The stack of commits was becoming a huge pain to rebase so I've squashed them. The full history is still up here: https://github.com/sfackler/rust/commits/ext-crate-history

This is necessary for rust-lang#11151 to make sure dtors run before the libraries are unloaded.

This is necessary for #11151 to make sure dtors run before the libraries are unloaded.

cartazio · 2014-01-06T17:34:56Z

Question: does the current design play nice with cross compilation?

I'm mostly asking because in some other languages, the metaprogramming facilities (sometimes)have some challenges in a cross compilation context.

sfackler · 2014-01-06T18:07:29Z

It should work just fine. The library with the macros will need to be compiled for the host architecture as opposed to the target architecture since it'll be loaded into the compiler, but it operates on the AST which doesn't depend on the target architecture.

cartazio · 2014-01-06T18:48:53Z

That's the thing: what about in the case of generating target dependent code? The ast certainly supports platform dependent operations. Is there a simple way to expose the host vs target info to the macro writer?

sfackler · 2014-01-06T18:57:18Z

You can have the extension generate code that uses #[cfg(...)] to case on different architectures.

pnkfelix · 2014-01-07T19:05:43Z

cc me

sfackler · 2014-01-16T03:01:38Z

This should be ready to go.

nikomatsakis · 2014-01-17T00:03:49Z

💃 this is so seriously cool

This is a first pass on support for procedural macros that aren't hardcoded into libsyntax. It is **not yet ready to merge** but I've opened a PR to have a chance to discuss some open questions and implementation issues. Example ======= Here's a silly example showing off the basics: my_synext.rs ```rust #[feature(managed_boxes, globs, macro_registrar, macro_rules)]; extern mod syntax; use syntax::ast::{Name, token_tree}; use syntax::codemap::Span; use syntax::ext::base::*; use syntax::parse::token; #[macro_export] macro_rules! exported_macro (() => (2)) #[macro_registrar] pub fn macro_registrar(register: |Name, SyntaxExtension|) { register(token::intern(&"make_a_1"), NormalTT(@SyntaxExpanderTT { expander: SyntaxExpanderTTExpanderWithoutContext(expand_make_a_1), span: None, } as @SyntaxExpanderTTTrait, None)); } pub fn expand_make_a_1(cx: &mut ExtCtxt, sp: Span, tts: &[token_tree]) -> MacResult { if !tts.is_empty() { cx.span_fatal(sp, "make_a_1 takes no arguments"); } MRExpr(quote_expr!(cx, 1i)) } ``` main.rs: ```rust #[feature(phase)]; #[phase(syntax)] extern mod my_synext; fn main() { assert_eq!(1, make_a_1!()); assert_eq!(2, exported_macro!()); } ``` Overview ======= Crates that contain syntax extensions need to define a function with the following signature and annotation: ```rust #[macro_registrar] pub fn registrar(register: |ast::Name, ext::base::SyntaxExtension|) { ... } ``` that should call the `register` closure with each extension it defines. `macro_rules!` style macros can be tagged with `#[macro_export]` to be exported from the crate as well. Crates that wish to use externally loadable syntax extensions load them by adding the `#[phase(syntax)]` attribute to an `extern mod`. All extensions registered by the specified crate are loaded with the same scoping rules as `macro_rules!` macros. If you want to use a crate both for syntax extensions and normal linkage, you can use `#[phase(syntax, link)]`. Open questions =========== * ~~Does the `macro_crate` syntax make sense? It wraps an entire `extern mod` declaration which looks a bit weird but is nice in the sense that the crate lookup logic can be identical between normal external crates and external macro crates. If the `extern mod` syntax, changes, this will get it for free, etc.~~ Changed to a `phase` attribute. * ~~Is the magic name `macro_crate_registration` the right way to handle extension registration? It could alternatively be handled by a function annotated with `#[macro_registration]` I guess.~~ Switched to an attribute. * The crate loading logic lives inside of librustc, which means that the syntax extension infrastructure can't directly access it. I've worked around this by passing a `CrateLoader` trait object from the driver to libsyntax that can call back into the crate loading logic. It should be possible to pull things apart enough that this isn't necessary anymore, but it will be an enormous refactoring project. I think we'll need to create a couple of new libraries: libsynext libmetadata/ty and libmiddle. * Item decorator extensions can be loaded but the `deriving` decorator itself can't be extended so you'd need to do e.g. `#[deriving_MyTrait] #[deriving(Clone)]` instead of `#[deriving(MyTrait, Clone)]`. Is this something worth bothering with for now? Remaining work =========== - [x] ~~There is not yet support for rustdoc downloading and compiling referenced macro crates as it does for other referenced crates. This shouldn't be too hard I think.~~ - [x] ~~This is not testable at stage1 and sketchily testable at stages above that. The stage *n* rustc links against the stage *n-1* libsyntax and librustc. Unfortunately, crates in the test/auxiliary directory link against the stage *n* libstd, libextra, libsyntax, etc. This causes macro crates to fail to properly dynamically link into rustc since names end up being mangled slightly differently. In addition, when rustc is actually installed onto a system, there are actually do copies of libsyntax, libstd, etc: the ones that user code links against and a separate set from the previous stage that rustc itself uses. By this point in the bootstrap process, the two library versions *should probably* be binary compatible, but it doesn't seem like a sure thing. Fixing this is apparently hard, but necessary to properly cross compile as well and is being tracked in #11145.~~ The offending tests are ignored during `check-stage1-rpass` and `check-stage1-cfail`. When we get a snapshot that has this commit, I'll look into how feasible it'll be to get them working on stage1. - [x] ~~`macro_rules!` style macros aren't being exported. Now that the crate loading infrastructure is there, this should just require serializing the AST of the macros into the crate metadata and yanking them out again, but I'm not very familiar with that part of the compiler.~~ - [x] ~~The `macro_crate_registration` function isn't type-checked when it's loaded. I poked around in the `csearch` infrastructure a bit but didn't find any super obvious ways of checking the type of an item with a certain name. Fixing this may also eliminate the need to `#[no_mangle]` the registration function.~~ Now that the registration function is identified by an attribute, typechecking this will be like typechecking other annotated functions. - [x] ~~The dynamic libraries that are loaded are never unloaded. It shouldn't require too much work to tie the lifetime of the `DynamicLibrary` object to the `MapChain` that its extensions are loaded into.~~ - [x] ~~The compiler segfaults sometimes when loading external crates. The `DynamicLibrary` reference and code objects from that library are both put into the same hash table. When the table drops, due to the random ordering the library sometimes drops before the objects do. Once #11228 lands it'll be easy to fix this.~~

brson · 2014-01-17T02:46:55Z

Awesome effort here. 🐯

bharrisau · 2014-01-20T13:48:15Z

Does this work with modules exported from the crate?

inner.rs

  #[macro_export]
  macro_rules! inner_exported_macro (() => (2))

lib.rs

  #[feature(managed_boxes, globs, macro_registrar, macro_rules)];

  pub mod inner;

  #[macro_export]
  macro_rules! exported_macro (() => (1))

I'v got exported_macro visible, but inner_exported_macro isn't. Haven't had a scan through the exported metadata yet (a quick google didn't show up a quick command to export the metadata). Am I forgetting something obvious?

sfackler · 2014-01-20T17:04:52Z

@bharrisau inner_exported_macro should be exported, but something must be a bit broken. I'll look into what's going on tonight.

sfackler · 2014-01-20T17:24:34Z

@bharrisau PR #11687 fixes it.

seanmonstar · 2014-01-22T05:27:09Z

with this, would it make sense to move some syntax extensions into related std modules? Like moving log!, debug!, info!, etc to std::logging?

sfackler · 2014-01-22T05:32:18Z

It'll at least need a new snapshot which I don't think has happened yet.

sfackler · 2014-01-22T05:38:10Z

Oops, turns out there is a new snapshot. It would be possible to do it now, though there are some span issues I'd like to work out first.

pnkfelix · 2014-05-15T10:21:43Z

@pcwalton You said:

I'd eventually like to deprecate macro_rules! in its current form and make it just a way to generate procedural macros.

While I do not object to this in principle, I would like to note that generating procedural macros within the same crate that those macros are used will require rustc to be able to essentially do JIT compilation: expand a macro definition to its procedural form, compile that form to a dylib, load the dylib into the running rustc, and continue compilation.

That's not a problem in principle, but I do think there is some value in a compilation model where the procedural macros are defined in distinct crates from where they are actually used, and the only kind of macro you can generate and use in the same compilation unit are ones that use a simple term-rewriting system language like that of macro_rules!. I've been having some discussions about this with @alexcrichton in a side-bar on PR #13948 and on irc

huonw reviewed Dec 30, 2013
View reviewed changes

sfackler mentioned this pull request Jan 2, 2014

Less painful range construction sfackler/rust-postgres#25

Closed

sfackler mentioned this pull request Jan 6, 2014

Add FromRow and ToRow traits sfackler/rust-postgres#28

Closed

sfackler added a commit to sfackler/rust that referenced this pull request Jan 6, 2014

Use ~-objects instead of @-objects for syntax exts

bb49916

This is necessary for rust-lang#11151 to make sure dtors run before the libraries are unloaded.

sfackler mentioned this pull request Jan 6, 2014

Use ~-objects instead of @-objects for syntax exts #11332

Merged

bors added a commit that referenced this pull request Jan 6, 2014

auto merge of #11332 : sfackler/rust/de-at-se, r=huonw

bae091e

This is necessary for #11151 to make sure dtors run before the libraries are unloaded.

erickt mentioned this pull request Jan 9, 2014

Add a regular expressions library to the distribution #3591

Closed

pnkfelix mentioned this pull request Jan 9, 2014

Don't promote binaries between stages #11145

Closed

emberian mentioned this pull request Jan 14, 2014

Docopt library #4658

Closed

Load macros from external modules

328b47d

bors closed this Jan 17, 2014

bors merged commit 328b47d into rust-lang:master Jan 17, 2014

erickt mentioned this pull request Jan 18, 2014

Move syntax extensions and macros out of compiler #1762

Closed

sfackler deleted the ext-crate branch January 20, 2014 17:03

pnkfelix mentioned this pull request Jan 22, 2014

[UR] local storage inside function #11726

Closed

This was referenced Jan 25, 2014

Macro import/export #3114

Closed

RFC: Allow external #[deriving(...)] implementations #11813

Closed

lifthrasiir mentioned this pull request Mar 16, 2014

add a macro rule system! #12915

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Externally loadable syntax extensions #11151

RFC: Externally loadable syntax extensions #11151

sfackler commented Dec 26, 2013

pcwalton commented Dec 26, 2013

pcwalton commented Dec 26, 2013

sfackler commented Dec 26, 2013

sfackler commented Dec 27, 2013

huonw commented Dec 30, 2013

huonw Dec 30, 2013

sfackler Dec 30, 2013

huonw commented Dec 30, 2013

sfackler commented Dec 30, 2013

pcwalton commented Dec 31, 2013

sfackler commented Jan 4, 2014

cartazio commented Jan 6, 2014

sfackler commented Jan 6, 2014

cartazio commented Jan 6, 2014

sfackler commented Jan 6, 2014

pnkfelix commented Jan 7, 2014

sfackler commented Jan 16, 2014

nikomatsakis commented Jan 17, 2014

brson commented Jan 17, 2014

bharrisau commented Jan 20, 2014

sfackler commented Jan 20, 2014

sfackler commented Jan 20, 2014

seanmonstar commented Jan 22, 2014

sfackler commented Jan 22, 2014

sfackler commented Jan 22, 2014

pnkfelix commented May 15, 2014

RFC: Externally loadable syntax extensions #11151

RFC: Externally loadable syntax extensions #11151

Conversation

sfackler commented Dec 26, 2013

Example

Overview

Open questions

Remaining work

pcwalton commented Dec 26, 2013

pcwalton commented Dec 26, 2013

sfackler commented Dec 26, 2013

sfackler commented Dec 27, 2013

huonw commented Dec 30, 2013

huonw Dec 30, 2013

Choose a reason for hiding this comment

sfackler Dec 30, 2013

Choose a reason for hiding this comment

huonw commented Dec 30, 2013

sfackler commented Dec 30, 2013

pcwalton commented Dec 31, 2013

sfackler commented Jan 4, 2014

cartazio commented Jan 6, 2014

sfackler commented Jan 6, 2014

cartazio commented Jan 6, 2014

sfackler commented Jan 6, 2014

pnkfelix commented Jan 7, 2014

sfackler commented Jan 16, 2014

nikomatsakis commented Jan 17, 2014

brson commented Jan 17, 2014

bharrisau commented Jan 20, 2014

sfackler commented Jan 20, 2014

sfackler commented Jan 20, 2014

seanmonstar commented Jan 22, 2014

sfackler commented Jan 22, 2014

sfackler commented Jan 22, 2014

pnkfelix commented May 15, 2014