Consider extensible parser API

[kpdecker]

There are a number of integrations that are out there (HTMLBars, Ember Handlebars, etc) or desired (i18n preprocessor, html-aware minimizer, html-aware linter).

We should strongly consider formalizing the parser/compiler APIs into a system that would be consumable and pluggable at the various levels.

This may take the form of AST/Opcode API formalization or similar. The SpiderMonkey AST model comes to mind.

[kpdecker]

@ebryn @mmun is there anything that we could change here that would simplify the HTMLbars integration?

[wycats]

Do you mean formalizing the AST (and the APIs that produce and consume ASTs) or something more ambitious?

I'm pretty happy with JISON as the parsing engine, but it doesn't look like you're talking about changing that, right?

[mmun]

@kpdecker The parser/AST changes you've already merged have been helpful and everything seems to be working smoothly. I'd be happy to draft a formalized AST if you'd like.

A couple side notes: HTMLBars only uses the generated Handlebars parser, and none of the compiler (though it is structured similarly).

Basically <p>Hello!<em>{{name}}</em></p> runs through the handlebars parser and produces the Handlebars AST:

- root program
  - content: `"<p>Hello!<em>"`
  - mustache: `name`
  - content: `"</em></p>"`

This AST gets fed into the HTMLBars parser (which is a combined HTML tokenizer / Handlebars AST walker) and produces the HTMLBars AST:

- root program
  - element: `p`
    - text: `"Hello!"`
    - element: `em`
      - mustache: `name`

[kpdecker]

@wycats a bit hand wavy right now. Want to try to think about how how the compile lifecycle can have points that are extensible. What would the easiest way for someone to interact with the AST before the code is generated? This without having to write their own tool and create handlebars -> handlebars compilers and other sorts of unpleasantness.

One step there would be formalizing these APIs for sure but I think there may be a little bit more in terms of how plugins could be registered to interact with such a flow.

No issues with jison, although we may want to consider updating that at some point, I think we are on a fairly old version.

[wycats]

@kpdecker I totally agree with this. I like the idea of having the CLI support intermediate forms, like dumping out a JSON version of the AST for consumption by other tools.

What kind of plugins do you have in mind?

[mmun]

cc @saponifi3d. I noticed you've been working on Handlebars 2.0 support for Emblem.js. You may have some thoughts on this topic.

[mmun]

I started drafting a formalism: https://gist.github.com/mmun/4e498b46b70c38978181. It's slightly more flexible than what the parser will accept today but the Handlebars compiler can enforce stronger semantics (and give better errors) where necessary.

[kpdecker]

Closing this out as I think #915 implements the desired API but further comments welcome.

[kpdecker]

Released in 3.0.0