RFC: WIP: Make macro hygiene easier to use and less error-prone #10940

vtjnash · 2015-04-22T03:09:30Z

This is a rebase and squash (plus updates) of PR #6910

TODO items:

handle new in type definitions in macros
handle fieldnames in type definitions in macros
move export of @hygienic to exports.jl
re-import doc changes from original PR
solicit feedback

@cmd

esc function and making hygiene automatic instead. This is a rebase and squash (plus updates) of the following commits: commit + c3af49c Author: Dave Moon <david-moon@rcn.com> Date: Wed Jun 4 15:41:52 2014 -0400 Update the entry for let in vars-introduced-by-patterns to use hygienic-symbol? instead of symbol?. env-for-expansion needs to use difference rather than diff to remove the globals, because hygienic-symbols are not interned, thus must be compared with equal? rather than eq?. commit + 850746a Author: Dave Moon <david-moon@rcn.com> Date: Wed Jun 4 15:19:43 2014 -0400 Change :x to quote x end in lots of places so it is hygienic, also fix a merge error commit + 5bba56d Author: Dave Moon <david-moon@rcn.com> Date: Wed Jun 4 15:18:34 2014 -0400 difference is to diff as assoc is to assq commit + 97c11fe Author: Dave Moon <david-moon@rcn.com> Date: Fri May 23 13:42:17 2014 -0400 Use quote instead of Expr in help macro so it gets automatic hygiene the same as before my changes commit + ac7240b Author: Dave Moon <david-moon@rcn.com> Date: Fri May 23 13:41:17 2014 -0400 Fix quoting problem in my previous commit commit + 9e7e186 Author: Dave Moon <david-moon@rcn.com> Date: Fri May 23 13:40:38 2014 -0400 Avoid symbol("@cmd") which is not hygienic commit + dddb999 Author: Dave Moon <david-moon@rcn.com> Date: Fri May 23 13:39:36 2014 -0400 Tweak hygiene doc commit + 755fca0 Author: Dave Moon <david-moon@rcn.com> Date: Fri May 23 11:38:10 2014 -0400 Fix error message interpolation in gen_call_with_extracted_types() commit + 4dfd75c Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 16:01:10 2014 -0400 Improve documentation of esc, per code review commit + 7ae26dd Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 15:56:25 2014 -0400 Add documen tation of splatting expression interpolation, make quote doc easier to find commit + 467549c Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 15:54:59 2014 -0400 Remove some tabs I accidentally added in the last commit commit + 5b2f162 Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 15:31:50 2014 -0400 Fix this to work with the new auto-hygienic macros. Specifically: gen_call_with_extracted_types() is declared hygienic, and uses quote instead of manually constructing Expr's so the result will be hygienic as it was before the new hygienic macros. Fix it so "@which(a[i]=x)" will work. Fix the too-complex error message not to assume @which is the caller. Gratuitous rearrangement of code in the for fname loop. Also tests have been added to make testall to verify all this. The tests failed with the new auto-hygienic macros before these fixes were put in. commit + 5a7d3b8 Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 15:31:16 2014 -0400 Add some tracing code (commented out) and fix some comments commit + 9713b2b Author: Dave Moon <david-moon@rcn.com> Date: Thu May 22 15:29:46 2014 -0400 Add tests for some macros commit + baeeb03 Author: Dave Moon <david-moon@rcn.com> Date: Wed May 21 13:05:50 2014 -0400 Update documentation for new macro hygiene mechanism commit + 8e07097 Author: Dave Moon <david-moon@rcn.com> Date: Tue May 20 16:17:15 2014 -0400 Make macro hygiene easier to use. PURPOSE: The problem with Julia's current implementation of hygienic macros is that it depends on the author of a macro to put calls to esc() in all the right places. Mistakes are hard to detect by testing because no failure will occur until there is an accidental name clash. Macros are already hard to think about, no need to add unnecessary complications. STRATEGY: Deprecate esc() and make it not do anything. Instead, quote marks all symbols that appear directly in the quote, rather than being interpolated, as hygienic. Post-processing of the macro expansion will replace these with the appropriate gensym, symbol, or module reference. Make a distinction between hygienic and non-hygienic quote, because quote is not only used by macros. Only hygienic quote does the above. Quote is hygienic when it is in the body of a macro or inside an @hygienic annotation. The output of hygienic quote will only work as part of a macro expansion, it will not work with eval. Unary : always behaves the same as quote. As a special exception, quoting a single symbol never makes it hygienic. This seems to be the best compromise to maximize usability without adding a new construct to reflect the fact that quoting a single symbol is sometimes used to construct executable code and other times used just to quote data. Fix the few macros in the base code that were broken by these changes. In the interest of simplicity, there are NO changes to the C code. TESTING: A clean build and make testall works on 32-bit Linux. RISKS: Did not test any packages that are not tested by make testall. There could be complex macros out there that will stop working because of these changes, analogous to @printf or @uv_write. The distinction between hygienic and non-hygienic quote could be confusing to users, although not as confusing as esc() was. esc() is deprecated but not marked as such. Calls to esc() are not removed from all over the source code. That could be cleaned up later. Another problem is that the macro system has its own separate code walker to decide where to insert gensyms for hygiene. This creates complexity and risk. These changes are not fixing that, because to do so would require changing much more of the Julia implementation to allow symbols marked as hygienic to be used directly without replacement by gensyms. However, if we did those changes, eval() would respect hygiene too and it would no longer be necessary to have distinct hygienic and non-hygienic versions of quote. PERFORMANCE IMPACT: Using assoc instead of assq is slower, but it only affects the macro expander. There is also slightly more consing than there was before. I doubt that the difference will be large enough for anyone to notice. I did not give scm_to_julia() a special node type for hygienic symbol markers. I doubt that not optimizing that costs much performance. PORTABILITY IMPACT: None expected. DOCUMENTATION IMPACT: Remove documentation of esc() and all mention of it. Document the distinction between hygienic and non-hygienic quote. Document @Hygiene. Document that $:x is a convenient way to insert a non-hygienic x into a hygienic quote. I will see if I can provide a suggested rewrite of the metaprogramming chapter to reflect these changes and make it easier to understand. All macro examples should be checked to remove esc and see if anything needs to be changed to keep them working. FILES CHANGED: src/julia-syntax.scm - New design for hygienic macros: - (hygienic <symbol>) represents an occurrence of <symbol> in a hygienic quote. It will be replaced with the appropriate gensym, symbol, or module reference when the macro is expanded. - allow hygienic symbols as argument names, type parameter names, variables in pair-with-gensyms, and keyword argument names. Keywords cannot be hygienic because the same symbol is both the name used by the caller and the variable bound in the callee (same as before). - Add an argument to expand-backquote() to tell it whether the quote is hygienic or not. If hygienic, add hygienic markers to all symbols that are variable names, but don't add them to Expr heads. Don't use copyast of inert in hygienic mode since we need to add hygienic markers to symbols. - Add an argument to julia-expand-macros-() to tell it whether quotes are hygienic or not. Support the @hygienic macro which uses the new with_hygiene Expr head. - The hygienic renaming env has to be searched with assoc rather than assq because uniqueness of (hygienic <symbol>) objects cannot easily be preserved through julia_to_scm() and scm_to_julia(). - resolve-expansion-vars-() only replaces symbols marked as hygienic by wrapping in (hygienic <symbol>). Plain symbols must have come from an interpolation of code supplied by a macro argument. base/LineEdit.jl - The bindings introduced by @keymap are supposed to be non-hygienic, make it so. base/REPLCompletions.jl - There is no longer an escape Expr, remove the check for it so the code will still work. base/base.jl - Neuter esc() but for convenience don't remove it yet. It's deprecated. base/expr.jl - Add @hygienic macro which makes any quotes in its body provide hygiene. This means those quoted expressions will only work as part of a macro expansion, they will not work with eval. base/printf.jl - Add @hygienic to all subroutines of the @printf and @sprintf macros so those macros will still be hygienic. Fix code in gen_d() to work around the fact that quote of just a symbol is no longer hygienic; the replacement code is clearer to read anyway. Fix @sprintf to use an explicit gensym to work around the same issue. base/process.jl - @setup_stdio and @cleanup_stdio are supposed to introduce non-hygienic bindings. base/stream.jl - @uv_write is supposed to introduce a non-hygienic binding of uvw.

tkelman · 2015-04-22T06:04:32Z

test/macros.jl

+# Test gen_call_with_extracted_types's alternate path for keyword arguments
+@test Which_test.test2(1.0) == Tuple{Real}
+@test Which_test.test2(123) == Tuple{Int}
+


whitespace here

also would've been nice to preserve @david-moons git authorship of a large part of this

To preserve authorship this approach can be used: http://blog.spreedly.com/2014/06/24/merge-pull-request-considered-harmful/

i can also run git commit --amend --author <original author> to fix this up, and had intended to do so before pushing. thanks for the reminder.

JeffBezanson · 2015-04-22T17:40:12Z

I think this needs to be merged early in a release cycle.

tkelman · 2015-04-22T21:51:08Z

Sounds familiar #6910 (comment)

Packages that make heavy use of macros (JuMP as an especially notable example) might need to make major changes to adapt to this. This would be a good candidate to run PkgEvaluator on prior to merging.

pao · 2015-04-22T21:52:27Z

Should the milestone get pushed out to 0.5, so there's some chance this doesn't slip through the cracks in (fingers crossed) a few months?

mlubin · 2015-04-23T01:12:32Z

Well, if we merge it soon then it will break JuMP, we'll fix it, and then we'll be able to quickly drop the old-style code after the 0.4 release. If we merge it at the start of the 0.5 cycle, it'll be less disruptive for users but we'll have to keep maintaining the old-style code for about a year until the 0.5 release. I'm ambivalent as to which is better, but either way please keep us in the loop and give us a chance to evaluate this before merging.

JeffBezanson · 2015-04-23T01:35:22Z

I'm not convinced that this is a large improvement. @hygienic introduces a strange lack of referential transparency, where it suddenly matters where your quote expressions are, and not just what expressions they yield. You also end up wrapping some expressions in :hygienic. That may be better in some ways, but at the end of the day it's very similar to how we have to wrap some things in :escape now. My impression is that this is roughly as fiddly as what we have now.

vtjnash · 2015-04-23T04:45:25Z

I think the argument is that fiddling with esc is non-intuitive. Whereas this actually interprets expressions in macros using rules that are closer to the context they are used in.

My understanding from the original PR is that @hygienic is a bit of a side-effect from not adding support to eval for unwrap hygienic Exprs, but that it too can go away with a bit more work.

vtjnash · 2015-04-23T04:47:23Z

@mlubin do you have any sense for how badly this would break JuMP? one of the original goals of the PR was to be as non-disruptive to existing codebases as possible.

JeffBezanson · 2015-04-23T04:57:37Z

a side-effect from not adding support to eval for unwrap hygienic Exprs, but that it too can go away with a bit more work

Then I think we should really not rush this, and take the time to do it right. My gut feel is that we are at capacity with things to worry about right now.

vtjnash · 2015-04-23T05:07:52Z

sounds good. i wanted to rebase this, in part, just so it wouldn't get too far behind

JeffBezanson · 2015-04-23T05:08:20Z

Yes, it's very nice to have it rebased.

toivoh · 2015-04-23T05:45:34Z

Agreed, let's not rush this but think through how we really want things to work.

Having written macros whose implementations effectively span multiple modules I must say though, that the hygiene aspect that depends on where the quote block is in the code is highly compelling; I've basically had to fall back to the old pre-hygiene tricks in a number of such cases.

JeffBezanson · 2015-12-19T18:16:17Z

@david-moon @toivoh I wonder if hygienic quote should resolve globals to the module containing the quote? I suppose to implement that we'd have to mark symbols as (hygienic ,module ,sym)?

david-moon · 2015-12-20T00:33:53Z

@david-moon @toivoh I wonder if hygienic quote should resolve globals to the module containing the quote? I suppose to implement that we'd have to mark symbols as (hygienic ,module ,sym) ?
—

Yes, symbols in the mark expansion that come from the macro definition rather than from the macro call should be looked up in the environment (including module) where the macro was defined. Did I forget to do that part when I prototyped this?

JeffBezanson · 2016-02-15T16:28:05Z

We seem to be missing the manual section that was written for this.

tkelman · 2016-04-21T20:49:04Z

Should the milestone get pushed out to 0.5, so there's some chance this doesn't slip through the cracks in (fingers crossed) a few months?

1 year later, this happened. Don't think this is still 0.5.

stevengj · 2016-08-14T22:55:50Z

Any update on this? Would be good to have hygiene finalized for 1.0.

StefanKarpinski · 2016-08-16T14:14:50Z

I think that a system like Clojure's with fully scoped symbols could be the simplest. It wouldn't even need to be particularly breaking since symbol literals like :foo could be "relative symbols" while if you wrote :(a + b) in a module M you would implicitly have M.a and M.b symbols. Of course, this forces us to stop punning on :a versus :(a) but I think that's for the best anyway.

JeffBezanson · 2016-08-16T14:30:22Z

I'm not sure about that. It pretty drastically changes what a symbol is. How does it work out for, say, keyword arguments and field names?

StefanKarpinski · 2016-08-16T15:05:38Z

That's a good question. I guess we should look at what Clojure does.

StefanKarpinski · 2016-08-16T15:14:51Z

If you continue to consider symbols to be implicitly relative but introduce a notion of fully qualified identifiers then keyword arguments and field names could continue to be represented by symbols (implicitly relative) while when you write something like :(a) it would be resolved to :(M.a). Maybe this isn't much different than what this PR does anyway.

StefanKarpinski · 2016-08-16T15:48:37Z

@david-moon sent me a message:

Stefan Karpinski wrote:

If you continue to consider symbols to be implicitly relative but introduce a notion of fully qualified identifiers then keyword arguments and field names could continue to be represented by symbols (implicitly relative) while when you write something like :(a) it would be resolved to :(M.a). Maybe this isn't much different than what this PR does anyway.

Yes, I think your choices for properly hygienic macros are either to draw a distinction between symbols (run-time objects) and identifiers (compile-time objects), or to do what I did (assuming I remember correctly, it's been years) and allow symbols to be interned in a dictionary specific to a particular macro expansion as well as in the usual global dictionary. This prevents symbols introduced by a macro from being eq to symbols in the macro call, but it requires normalizing those local symbols to global symbols when they end up as literals, keyword arguments, field names, etc. in the macro expansion. When those local symbols end up as bound variables, everything just works. When they end up as unbound variables, they need to be looked up in the scope where the macro was defined rather than in the scope where it was called.

Whatever you do, it shouldn't have a built-in restriction that a macro can only be defined in module scope, not in a scope nested inside a module. Even if for implementation reasons you need that restriction at first, it shouldn't be built deeply into the language. An identifier introduced by a macro should be in the scope where the macro was defined regardless of the nature or lifetime of that scope.

[I have a feeling this email reply is going to be prevented from going to the correct mailing list. Could you please forward it?]

--Dave Moon

JeffBezanson · 2016-08-16T16:52:10Z

For globals, we do have a representation for fully-qualified identifiers: GlobalRef objects. I'm not sure it will help much to resolve :(a) to :(M.a) (a GlobalRef) very early, since you still need the surrounding context to know what the symbol actually refers to. But it does seem likely some version of this is similar to what's here, especially if we include #10940 (comment)

stevengj · 2016-12-27T17:20:03Z

I guess this will not make 0.6, but it would be nice to have for 1.0

vtjnash · 2018-10-02T17:09:43Z

don't think this helpful anymore to have open, future work can just as easily start from the original (#6910)

mauro3 · 2018-11-21T13:00:24Z

Should the original be re-opened? And its milestone updated to 2.0?

vtjnash added this to the 0.4.0 milestone Apr 22, 2015

vtjnash mentioned this pull request Apr 22, 2015

RFC: Improvement to hygienic macros #6910

Open

vtjnash force-pushed the moon/hygienic-macros branch from 8f9d259 to 4fb0c21 Compare April 22, 2015 03:32

tkelman reviewed Apr 22, 2015
View reviewed changes

JeffBezanson modified the milestones: 0.5, 0.4.0 May 28, 2015

tkelman mentioned this pull request Aug 2, 2015

Doc hard, and with a vengeance #11943

Merged

GlenHertz mentioned this pull request Aug 6, 2015

Improve documentation for generated functions #12488

Closed

JeffBezanson mentioned this pull request Feb 15, 2016

Symbols passed to macros in a module improperly scoped #15085

Closed

tkelman modified the milestones: 0.6.0, 0.5.0 Apr 21, 2016

Sacha0 mentioned this pull request Aug 13, 2016

Deprecate functions vectorized via @vectorize_(1|2)arg in favor of compact broadcast syntax #17302

Merged

4 tasks

StefanKarpinski assigned vtjnash and JeffBezanson Sep 13, 2016

JeffBezanson modified the milestones: 1.0, 0.6.0 Dec 28, 2016

JeffBezanson modified the milestones: 2.0+, 1.0 May 2, 2017

JeffBezanson mentioned this pull request Jun 9, 2017

macro hygiene bug #22307

Closed

vtjnash mentioned this pull request Jul 27, 2017

fix some macro expansion scope bugs #22985

Merged

JeffBezanson mentioned this pull request Aug 6, 2017

some deprecated methods in the wrong module #23146

Closed

c42f mentioned this pull request Aug 14, 2017

macro hygine escape stripped late for macros calling macros #23221

Closed

stevengj mentioned this pull request Jun 6, 2018

fix cfunction escaping bug JuliaLang/Compat.jl#566

Merged

vtjnash closed this Oct 2, 2018

DilumAluthge deleted the moon/hygienic-macros branch March 25, 2021 22:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: WIP: Make macro hygiene easier to use and less error-prone #10940

RFC: WIP: Make macro hygiene easier to use and less error-prone #10940

vtjnash commented Apr 22, 2015 •

edited

Loading

tkelman Apr 22, 2015

mauro3 Apr 22, 2015

vtjnash Apr 22, 2015

JeffBezanson commented Apr 22, 2015

tkelman commented Apr 22, 2015

pao commented Apr 22, 2015

mlubin commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

vtjnash commented Apr 23, 2015

vtjnash commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

vtjnash commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

toivoh commented Apr 23, 2015

JeffBezanson commented Dec 19, 2015

david-moon commented Dec 20, 2015

JeffBezanson commented Feb 15, 2016

tkelman commented Apr 21, 2016

stevengj commented Aug 14, 2016

StefanKarpinski commented Aug 16, 2016

JeffBezanson commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

JeffBezanson commented Aug 16, 2016

stevengj commented Dec 27, 2016

vtjnash commented Oct 2, 2018

mauro3 commented Nov 21, 2018

RFC: WIP: Make macro hygiene easier to use and less error-prone #10940

RFC: WIP: Make macro hygiene easier to use and less error-prone #10940

Conversation

vtjnash commented Apr 22, 2015 • edited Loading

tkelman Apr 22, 2015

Choose a reason for hiding this comment

mauro3 Apr 22, 2015

Choose a reason for hiding this comment

vtjnash Apr 22, 2015

Choose a reason for hiding this comment

JeffBezanson commented Apr 22, 2015

tkelman commented Apr 22, 2015

pao commented Apr 22, 2015

mlubin commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

vtjnash commented Apr 23, 2015

vtjnash commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

vtjnash commented Apr 23, 2015

JeffBezanson commented Apr 23, 2015

toivoh commented Apr 23, 2015

JeffBezanson commented Dec 19, 2015

david-moon commented Dec 20, 2015

JeffBezanson commented Feb 15, 2016

tkelman commented Apr 21, 2016

stevengj commented Aug 14, 2016

StefanKarpinski commented Aug 16, 2016

JeffBezanson commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

StefanKarpinski commented Aug 16, 2016

JeffBezanson commented Aug 16, 2016

stevengj commented Dec 27, 2016

vtjnash commented Oct 2, 2018

mauro3 commented Nov 21, 2018

vtjnash commented Apr 22, 2015 •

edited

Loading