feat(tasks/ast_codegen): prototype for codegen AST related code

[rzvxa]

Part of #3819

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[rzvxa]

• feat(ast)!: generate ast_builder.rs. #3890
• feat(ast_codegen): generate ast_kind.rs. #3888
• fix(ast)!: rename all instances of BigintLiteral to BigIntLiteral. #3898
• feat(ast_codegen): add ImplGetSpanGenerator. #3852
• feat(tasks/ast_codegen): prototype for codegen AST related code #3815 👈
• refactor(ast)!: revert adding span field to the BindingPattern type. #3899
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rzvxa and the rest of your teammates on Graphite

[codspeed-hq]

CodSpeed Performance Report

Merging #3815 will not alter performance

_{Comparing 06-22-feat_ast_gen_add_ast_code_gen (f6c4ec4) with main (92c21b2)}

Summary

✅ 28 untouched benchmarks

[overlookmotel]

What I had in mind for the "schema" is more abstracted, and would not use syn's types (kind of an IR). e.g.:

// `TypeId` is an index into `Types`
type TypeId = usize;
struct Types(Vec<TypeDef>);

enum TypeDef {
  Struct(StructDef),
  Enum(EnumDef),
  Box(BoxDef),
  Vec(VecDef),
  Option(OptionDef),
  Cell(CellDef),
  Primitive(PrimitiveDef),
  StrRef(StrRefDef),
  // I think that's all of them?
}

struct StructDef {
  name: String,
  fields: Vec<StructFieldDef>,
  has_lifetime: bool,
}

struct StructFieldDef {
  name: String,
  type_id: TypeId,
}

struct EnumDef {
  name: String,
  variants: Vec<EnumVariantDef>,
  // For `@inherits` inherited enum variants
  inherits: Vec<TypeId>,
  has_lifetime: bool,
}

struct EnumVariantDef {
  name: String,
  type_id: Option<TypeId>, // `None` for fieldless variants
  discriminant: Option<u8>,
}

// ... and so on ...

My thought was codegen proceeds in 2 steps:

1. Parse .rs files and build Types.
2. Generate code from Types (without needing original syn AST).

The types above are similar to what layout_inspect uses (and is what AST transfer will need): https://github.com/overlookmotel/layout_inspect/blob/master/layout_inspect/src/defs.rs

StructDef etc can have more fields added later on if we need them, but I think initially this is all that's needed.

Types could alternatively be a hashmap keyed by type name, but I thought a Vec which is indexed into would be simpler. It may come in useful later for every AST type to have a simple integer ID.

Does this make any sense? What do you think?

[rzvxa]

What I had in mind for the "schema" is more abstracted, and would not use syn's types (kind of an IR). e.g.:

// `TypeId` is an index into `Types`
type TypeId = usize;
struct Types(Vec<TypeDef>);

enum TypeDef {
  Struct(StructDef),
  Enum(EnumDef),
  Box(BoxDef),
  Vec(VecDef),
  Option(OptionDef),
  Cell(CellDef),
  Primitive(PrimitiveDef),
  StrRef(StrRefDef),
  // I think that's all of them?
}

struct StructDef {
  name: String,
  fields: Vec<StructFieldDef>,
  has_lifetime: bool,
}

struct StructFieldDef {
  name: String,
  type_id: TypeId,
}

struct EnumDef {
  name: String,
  variants: Vec<EnumVariantDef>,
  // For `@inherits` inherited enum variants
  inherits: Vec<TypeId>,
  has_lifetime: bool,
}

struct EnumVariantDef {
  name: String,
  type_id: Option<TypeId>, // `None` for fieldless variants
  discriminant: Option<u8>,
}

// ... and so on ...

My thought was codegen proceeds in 2 steps:

1. Parse `.rs` files and build `Types`.

2. Generate code from `Types` (without needing original `syn` AST).

The types above are similar to what layout_inspect uses (and is what AST transfer will need): https://github.com/overlookmotel/layout_inspect/blob/master/layout_inspect/src/defs.rs

StructDef etc can have more fields added later on if we need them, but I think initially this is all that's needed.

Types could alternatively be a hashmap keyed by type name, but I thought a Vec which is indexed into would be simpler. It may come in useful later for every AST type to have a simple integer ID.

Does this make any sense? What do you think?

I'll add the simple output, I keep the syn type in the building process so we can expand macros and generate AST in the future if needed.

you are looking at the Module sturct not the Schema.
I basically drop the syn types and keep their meta after the build as the schema which then can be outputed as either json or binary.

[overlookmotel]

you are looking at the Module sturct not the Schema.
I basically drop the syn types and keep their meta after the build as the schema which then can be outputed as either json or binary.

Sorry if I misunderstood. I don't quite understand what you're doing, but no matter. I'm sure you have a plan!

[rzvxa]

you are looking at the Module sturct not the Schema.
I basically drop the syn types and keep their meta after the build as the schema which then can be outputed as either json or binary.

Sorry if I misunderstood. I don't quite understand what you're doing, but no matter. I'm sure you have a plan!

It is totally on me, Currently, the code is full of unused types and dirty prototypes, I'm focused on getting the whole pipeline to work and then going back to clean it up.

I also opted for interior mutability and cloning where needed since there aren't many performance concerns in this crate. I thought it would make development easier if we could be a little careless with how we pass around stuff, After all this script only runs once in a while and the runtime is minimal so not having to fight with borrow checker constantly is much more favorable than having a well-optimized build script.

[rzvxa]

@Boshen @overlookmotel This is almost ready, You may want to give it a spin.

[Boshen]

@overlookmotel can you help me review and merge this one? This is beyond me 😅

I'll chime in when the edit - save workflow is up and running.

[rzvxa]

@overlookmotel I suggest you only review #3852 there is some stuff I've changed after this PR and I was too lazy to cherry-pick and merge them here.

[rzvxa]

@overlookmotel Just to make your life easier, I'm adding comments to help with the review process, I'll mention you in up stack PR as soon as I submit my docs commit.

[graphite-app]

Merge activity

• Jun 25, 9:54 AM EDT: Boshen added this pull request to the Graphite merge queue.
• Jun 25, 9:59 AM EDT: Boshen merged this pull request with the Graphite merge queue.