feat(ast_tools): add dedicated Derive trait.

[rzvxa]

In an effort toward the implementation of #5256, this PR allows us to have a separately generated "derive" file for each crate.
This also eliminates a bunch of boilerplate when writing new "derive" generators and generally makes it more approachable.

[rzvxa]

• refactor(linter): use "parsed pattern" in no_div_regex rule. #5417
• feat(ast, parser): add oxc_regular_expression types to the parser and AST. #5256
• refactor(syntax): use generate_derive for CloneIn in types outside of oxc_ast crate. #5280
• feat(ast_tools): add dedicated Derive trait. #5278 👈
• feat(regular_expression): implement Display for RegularExpression type. #5304
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rzvxa and the rest of your teammates on Graphite

[codspeed-hq]

CodSpeed Performance Report

Merging #5278 will not alter performance

_{Comparing 08-28-feat_ast_tools_add_dedicated_derive_trait (68a1c01) with main (be1a6d4)}

Summary

✅ 28 untouched benchmarks

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[overlookmotel]

Just to say sorry I've not got to reviewing this yet. Have been going hell-for-leather on the transformer and semantic. Hope I can get to it this evening.

[overlookmotel]

This is really good. It simplifies writing a new #[generate_derive] a lot. I can imagine us writing a bunch more in future - it's a really fast way to build something out which would take days and days to do by hand. The comments I've made are largely nits.

[overlookmotel]

One broader thing...

The one thing that could confuse people is visibility. We're used to code generated by #[derive] sitting next to the struct/enum it's derived on, and having access to everything you can "see" from that position. Obviously #[generated_derive] doesn't work like that.

I don't think much we can do about that, but we should document that you have to make structs/enums/fields pub(crate) if you're using #[generated_derive], so people at least understand how to use it.

Or (and very possibly this is a stupid idea), could we make it work like #[derive]? e.g.:

• Generate 1 file of generated code for each source file (not one for each crate).
• Introduce a #[derive_generator] macro.

Then you use #[generated_derive] as follows:

// src/foo.rs
#[derive_generator]
use oxc_allocator::CloneIn;

#[generated_derive(CloneIn)]
struct Foo { // Not pub
    blah: u64 // Not pub
}

#[derive_generator] would expand to:

include!("generated/derive_clonein_foo.rs");

Then the generated code has access to everything in the source file without having to make anything pub (still wouldn't work for a struct which is defined inside a private module {} block, but that's a niche case).

This approach has pluses and minuses... maybe it solves one problem but creates another!

[overlookmotel]

One other thing. At some point, maybe we should consider breaking #[generated_derive] out into a stand-alone thing, separate from #[ast]. We may well want to derive traits on types which are not part of the AST.

And... maybe this is completely bananas, but with this Derive trait, is there anything stopping us defining the derives in the source crates along with the traits they generate code for? e.g.:

// oxc_allocator/src/clone_in.rs

#[cfg(feature = "generate")]
use oxc_ast_tools::{Derive, TypeDef, LateCtx, TokenStream};

pub trait CloneIn {
    type Cloned;
    fn clone_in(&self, allocator: &'new_alloc Allocator) -> Self::Cloned;
}

#[cfg(feature = "generate")]
impl Derive for CloneIn {
    fn derive(&mut self, def: &TypeDef, _: &LateCtx) -> TokenStream {
        match &def {
            TypeDef::Enum(it) => derive_enum(it),
            TypeDef::Struct(it) => derive_struct(it),
        }
    }
}

// etc...

It's always been impossible to define your #[derive] macro in same crate as the trait it derives, but that's just due to a quirk of how Rust's proc macros work. I don't think we're constrained in the same way.

[overlookmotel]

Since #5304 is blocked at present, would you like to split this and #5280 into a separate stack, so we can get them merged now?

Concerning my idea of using include!() to generate code in situ, I guess the other option is one we've discussed before - the codegen passing a serialized TokenStream to the macro to insert. Maybe that's a better solution (when we can get it working) - though I do like that with generated code you have it committed to the repo and it's visible, rather than being hidden away behind a macro expansion.

[rzvxa]

Since #5304 is blocked at present, would you like to split this and #5280 into a separate stack, so we can get them merged now?

Let's not do that, I don't want to mess up the Graphite stack. The down stack should be ready to merge. I'll request a review after running actions.

---

What I like about include is the fact that we can have our code outside of the src directory(I'm 90% sure, Haven't tested it), This would allow us to put these stuff in a hidden directory so it isn't accessible from our code base in any other way. If we want to hide away the process then having these files in the generated directory might be more confusing than helpful. It's not like accessing oxc_crate_name/.generated_derives/ is hard.

It is worth having an issue tracking this idea in the backlog. We have to do a quick test to make sure it's practical. There might be some quirks with how crates work.

Regarding the latter idea, My personal take is having derives in a separate project is "cleaner" than having them with a conditional feature flag and bloating our core projects with bootstrapping flags. We could use a third person pitching in, Technically speaking it is possible to implement I'm not sure if it would result in a better DX.

[overlookmotel]

Regarding the latter idea, My personal take is having derives in a separate project is "cleaner" than having them with a conditional feature flag and bloating our core projects with bootstrapping flags. We could use a third person pitching in, Technically speaking it is possible to implement I'm not sure if it would result in a better DX.

I suppose the reason I thought of this is I recently added another generated file to oxc_traverse: #5119

This generator is pretty simple and lightweight, and it felt like little generators like this would sit quite naturally in the crate in which the generated code sits, so you have the generator and generated code next to each other.

I can imagine writing many more of these (but in Rust, not JS!), and the codebase ending up peppered with them. At that point oxc_ast_tools could become a bit of a monolith.

I'm not 100% pushing that we should do this, just explaining where I'm coming from.