Prototype: Macro expansion / lowering #329

c42f · 2023-07-12T20:29:21Z

This is the start of trying out some ideas from #327

Here I'm prototyping JuliaSyntax.eval2() and JuliaSyntax.include2() which are some baby steps toward a "new frontend".

Currently this code achieves the following

All expressions carry precise source location
Preserve precise source information through macro expansion
Automatic hygiene in the style of RFC: Improvement to hygienic macros julia#6910
Fixing macro expansion edge cases like Macro hygiene is hard to use correctly in nested macro expansion julia#37691
Code using the new frontend can call old macros
Modules using eval2() are just normal Julia modules which can interoperate with code from old modules.
User defined macros can emit precise error messages by throwing MacroExpansionError

Fixing these issues with the frontend implies changing the representation of expression literals. Here I have a SyntaxLiteral type rather than Expr.

All these things appear to work in this proof of concept so far. Things I'd like to play with...

Port "some representative parts" of lowering to this system; enough that we can get a CodeInfo out directly without calling into the old frontend.
Implement some nontrivial macros and think about a better API for SyntaxLiteral

Begin working on macro expansion. So far this just expands backquotes. Also some changes to SyntaxNode make it possible to construct ASTs from these programmatically, without the source being available. And some basic support to allow us to track provenance through syntax transformations.

Allow `macroexpand` to call macros which only have an implementation in terms of the `Expr` data structure. Also add a `__macroname__` argument to pass the expression of the macro name as it appeared at macro invocation. Implement `@__FILE__` `@__LINE__` and `@__COLUMN__` in terms of this.

Macros expansion of macros which emit macros as part of their AST needs care when that AST contains interpolations: We don't want the second macro expansion to see the hygenic_scope expression. One way to do this is to transform hygenic_scope back into `SyntaxLiteral` on the fly. Also fix auto-escaping for expressions arising from the original scope by adding the necessary number of Expr(:esc)

codecov · 2023-07-12T20:33:08Z

Codecov Report

Merging #329 (81c5dc2) into main (296cd5e) will decrease coverage by 3.67%.
Report is 4 commits behind head on main.
The diff coverage is 59.65%.

@@            Coverage Diff             @@
##             main     #329      +/-   ##
==========================================
- Coverage   96.54%   92.88%   -3.67%     
==========================================
  Files          14       17       +3     
  Lines        4140     4608     +468     
==========================================
+ Hits         3997     4280     +283     
- Misses        143      328     +185

Files Changed	Coverage Δ
src/JuliaSyntax.jl	`100.00% <ø> (ø)`
src/lowering.jl	`0.00% <0.00%> (ø)`
src/match.jl	`0.00% <0.00%> (ø)`
src/parse_stream.jl	`93.69% <0.00%> (-0.20%)`	⬇️
src/kinds.jl	`78.57% <50.00%> (-2.03%)`	⬇️
src/macroexpand.jl	`78.92% <78.92%> (ø)`
src/syntax_tree.jl	`93.88% <83.33%> (-1.48%)`	⬇️
src/diagnostics.jl	`95.23% <100.00%> (+0.23%)`	⬆️
src/expr.jl	`100.00% <100.00%> (+0.31%)`	⬆️

... and 1 file with indirect coverage changes

One of the many internal Base things we're using is likely different on <= 1.8

Plus subtle fixes to macro expansion prototype

c42f · 2023-08-07T02:50:35Z

@maleadt I've implemented a prototype of @ccall macro expansion here, based on porting the implementation of @ccall from Julia 1.9. It turns out to be an excellent test case because it:

Has interesting error reporting cases
Constructs expressions for which there's no surface syntax (Expr(:foreigncall))
Pattern matching on the input
However is not overly complex in the scheme of things

Quick benchmarks suggest ~7x speed improvement over Base's current macro expansion compared with the @ccall implementation from Julia 1.9 (implementation I coped into the module CCall19 for this test. ie, before JuliaLang/julia#50077)

julia> ex = parsestmt(SyntaxNode, "CCall.@ccall libfoo.bar(a::A, b::B, c::C, d::D, e::E)::X")
       @benchmark JuliaSyntax.macroexpand(Main, ex)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  35.980 μs …  33.100 ms  ┊ GC (min … max): 0.00% … 99.77%
 Time  (median):     38.453 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   43.957 μs ± 330.734 μs  ┊ GC (mean ± σ):  7.51% ±  1.00%

  ▅▇██▄▄▂                                                      ▂
  ████████▇▆▅▄▄▂▄▄▆▆▆█▇▇▆▇▇▇▇▇▆▆▆▅▆▆▆▇▆▄▄▆▆▆▇▆▅▅▆▅▅▅▂▄▄▅▇▆▄▄▄▄ █
  36 μs         Histogram: log(frequency) by time      88.2 μs <

 Memory estimate: 48.84 KiB, allocs estimate: 1163.

julia> ex = :(CCall19.@ccall libfoo.bar(a::A, b::B, c::C, d::D, e::E)::X)
       @benchmark macroexpand(Main, ex)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  217.853 μs …  10.244 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     227.139 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   293.611 μs ± 602.926 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

  █▇▄▃▄▂▂▂▂▁▁▁                                                  ▂
  ████████████▇██▇▇▇▇▆▅▇▆▅▆▅▅▆▅▄▁▄▄▁▁▃▃▄▃▄▃▁▃▄▁▁▃▁▁▁▁▁▁▁▁▁▄▄▃▃▅ █
  218 μs        Histogram: log(frequency) by time        798 μs <

 Memory estimate: 18.57 KiB, allocs estimate: 365.

It's hard to get a perfectly fair comparison here but these kind of speedups are generally what I've observed in JuliaSyntax's parser vs the flisp as well.

c42f · 2023-08-08T06:46:49Z

It looks like that benchmark above is measuring hygenic-scope resolution rather than macro expansion and this is the real cost. Hacking libjulia to add a function which doesn't call jl-expand-macroscope shows that macro expansion itself is really not a problem. As might be expected given that expansion itself is written in C and just calls Julia code:

julia> ex = :(CCall19.@ccall libfoo.bar(a::A, b::B, c::C, d::D, e::E)::X)
       @benchmark macroexpand_no_scope(Main, ex)
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  12.125 μs … 73.399 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     13.049 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   13.501 μs ±  2.584 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

   ▃█▅▁                                                        
  ▃█████▆▄▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▁▁▂▂▂▁▁▁▁▂▁▂▁▂▂▂▂▂▂▂▂▂▂▁▂ ▃
  12.1 μs         Histogram: frequency by time        28.4 μs <

 Memory estimate: 14.98 KiB, allocs estimate: 295.

This also means it's very hard to do a fair benchmark of this in the new prototype without also reimplementing the parts of lowering which do scope resolution. (Which I want to do! It's just a fair bit of work.)

maleadt · 2023-08-08T09:24:32Z

This also means it's very hard to do a fair benchmark of this in the new prototype without also reimplementing the parts of lowering which do scope resolution.

I see; thanks for clarifying! I had trouble myself too figuring out where the real cost of the @ccall expansion was, and happy to see this considered for future work. And for the time being, I can just go back to plain ccall anyway.

c42f · 2024-06-04T02:24:53Z

This work is continuing over at https://github.com/c42f/JuliaLowering.jl so I'll close this.

Over there I've got enough things working that we can produce a CodeInfo directly for some simple cases without ever calling into Julia's existing lowering implementation. Also an entirely new and much more flexible data structure to replace SyntaxNode. Automatic hygiene also works, including weird cases like macros recursively expanding to calls to themselves, and the old scope resolution mechanism is entirely deleted and replaced with something better. There's lots and lots to do though - many very basic things like branching don't even work yet (ie, come back in half a year ;-) )

It looks like that benchmark above is measuring hygenic-scope resolution rather than macro expansion and this is the real cost.

At some point I'll port the @ccall implementation from this PR over to there and we'll finally be able to understand the performance implications. But the new implementation basically deletes the separate hygenic-scope resolution pass entirely so I expect good things 😊

c42f added 7 commits July 9, 2023 07:22

Prototype macro expansion - works without esc()!

9edd433

Add forgotten macroexpand_inc.jl

cd846e0

Various fixes and cleanups for scopes in macro expansion

b43c443

Precise diagnostics from macros; MacroExpansionError

fe69bf0

c42f mentioned this pull request Jul 12, 2023

Macro expansion and lowering #327

Open

c42f added 8 commits July 13, 2023 07:00

Only test macroexpand on 1.9+

d672db2

One of the many internal Base things we're using is likely different on <= 1.8

Remove srcref kwarg usage in SyntaxNode construction

d5193ca

WIP lowering

fa096e8

lowering WIP

4ffae03

WIP

9a54e2e

Various macro expansion fixes + @ccall prototype implementation

f72a1d0

Fix ccall prototype implementation

657a23c

Plus subtle fixes to macro expansion prototype

Minor fix for ccall prototype

81c5dc2

c42f mentioned this pull request Apr 15, 2024

"Bindings as sets of scopes" c42f/JuliaLowering.jl#1

Open

c42f closed this Jun 4, 2024

c42f mentioned this pull request Jun 4, 2024

Port parts of JuliaSyntax.jl#329 c42f/JuliaLowering.jl#4

Open

3 tasks

c42f deleted the caf/macro-expansion branch August 9, 2024 05:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prototype: Macro expansion / lowering #329

Prototype: Macro expansion / lowering #329

c42f commented Jul 12, 2023 •

edited

Loading

codecov bot commented Jul 12, 2023 •

edited

Loading

c42f commented Aug 7, 2023 •

edited

Loading

c42f commented Aug 8, 2023 •

edited

Loading

maleadt commented Aug 8, 2023

c42f commented Jun 4, 2024

Prototype: Macro expansion / lowering #329

Prototype: Macro expansion / lowering #329

Conversation

c42f commented Jul 12, 2023 • edited Loading

codecov bot commented Jul 12, 2023 • edited Loading

Codecov Report

c42f commented Aug 7, 2023 • edited Loading

c42f commented Aug 8, 2023 • edited Loading

maleadt commented Aug 8, 2023

c42f commented Jun 4, 2024

c42f commented Jul 12, 2023 •

edited

Loading

codecov bot commented Jul 12, 2023 •

edited

Loading

c42f commented Aug 7, 2023 •

edited

Loading

c42f commented Aug 8, 2023 •

edited

Loading