Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-compile macro output as include! #114

Open
overlookmotel opened this issue Sep 12, 2024 · 1 comment
Open

Pre-compile macro output as include! #114

overlookmotel opened this issue Sep 12, 2024 · 1 comment

Comments

@overlookmotel
Copy link

overlookmotel commented Sep 12, 2024

@rzvxa and I have been discussing various ways to make macros cheaper on compile time, for example the #[ast] macro.

I had another idea in this vein. How about this:

  • For each type marked #[ast], ast_tools would output a separate file.
  • If the type is in oxc_ast/src/ast/js.rs, then the codegen would write to oxc_ast/.macro_expansions/src/ast/js.rs/<line number>.rs.
  • Content of those files would be the complete output after processing (i.e. alter the type def in same way #[ast] macro does now).
  • #[ast] macro would expand to:
include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));

Example

Original type def:

// oxc_ast/src/ast/js.rs line 123
#[ast]
enum Foo<'a> {
    Bar(Box<'a, Bar>),
    Qux(Box<'a, Qux>),
    @inherits Donkey,
}

ast_tools writes to file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
#[repr(C, u8)]
enum Foo<'a> {
    Bar(Box<'a, Bar>) = 0,
    Qux(Box<'a, Qux>) = 1,
    BigDonkey(Box<'a, BigDonkey>) = 2,
    SmallDonkey(Box<'a, SmallDonkey>) = 3,
}

#[ast] macro's implementation is very simple indeed:

#[proc_macro_attribute]
pub fn ast(_: TokenStream, _: TokenStream) -> TokenStream {
    // No `syn` here!
    r#"include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));"#.parse().unwrap()
}

Spans

I've not tested this out, so I don't know for sure it'll work. In particular, I wonder if spans/hygiene will get mashed up so Rust Analyser may lose "jump to definition" etc.

Maybe we can work around that by outputting a macro_rules! macro. e.g. ast_tools writes this to the file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
macro_rules! macro_123 {
    (
        #[ast]
        enum $Foo:ty $life:lifetime {
            $Bar:ident($Box1:ident<'a, $Bar:ident>),
            $Qux:ident($Box2:ident<'a, $Qux:ident>),
            @$inherits:ident $Donkey:ident,
        }
    ) => {
        #[repr(C, u8)]
        enum $Foo $life {
            $Bar($Box1<'a, $Bar>) = 0,
            $Qux($Box2<'a, $Qux>) = 1,
            macro_123!(@$inherits $Donkey,)
        }
    }

    (@inherits Donkey,) => {
        BigDonkey(Box<'a, BigDonkey>) = 2,
        SmallDonkey(Box<'a, SmallDonkey>) = 3,
    }
}

Then #[ast] macro would output a token stream:

include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));
macro_123! {
<original input TokenStream>
};

That's a bit more involved, but still can be done just by concatenating TokenStreams, which is much cheaper than syn.

Actually I'm not sure how to get proc macro to output macro_123!, as the proc macro doesn't know the line number. Maybe proc macro has to hash the input stream and use that as the UID instead of line number. Hashing with FxHash is fairly cheap.

Or maybe it searches through the input TokenStream for the token struct or enum, and then the UID is the token that follows (which will be the type name).

What do you think, @rzvxa?

@rzvxa
Copy link
Collaborator

rzvxa commented Sep 12, 2024

This one is fairly complex. Give me some time to digest the ideas. But I'm kind of on board with the "include" already; Just need some time to process it and come back with comments.

Edit

I'm mostly worried about the manual import of these files. We should make them internal and unexposable to the users somehow. Maybe we can have them outside of the src directory so nobody can use them as valid Rust modules. But having source files outside of src makes me worried about crate and how the publish works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants