Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for simple import *statement* to identifier #2

Merged
merged 5 commits into from
Aug 1, 2016

Conversation

JimPanic
Copy link
Collaborator

I couldn't get import as an expression to work, so I resorted to define it as a
statement for now.

I'm pretty sure multi-line imports don't work either and there's no alias
functionality yet. So this is still heavily WIP.

I couldn't get import as an expression to work, so I resorted to define it as a
statement for now.

I'm pretty sure multi-line imports don't work either and there's no alias
functionality yet. So this is still *heavily* WIP.
@GeoffreyBooth
Copy link
Owner

I have very similar code that I haven't committed yet. But based on lydell's comment I think I need to backtrack. I had created a moduleToken function in lexer.coffee just like in that PR, but I think that that won't work (lydell says as much). I think just about the only modifications we'll make to lexer.coffee will be to recognize from and as and treat them as special tokens, similar to yield from. Notice how from is not a reserved word, even though it's treated as a keyword when used with yield. I think we need to do the same thing for our from, and as.

@@ -353,6 +353,7 @@ grammar =

Import: [
o 'IMPORT Expression', -> new Import $2
o 'IMPORT Assignable FROM Expression', -> new Import $4, $2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to replace Assignable with something new that represents the ES2015 import syntax, and Expression with something that only matches ' and " strings (also keep in mind that interpolation cannot be used here.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this was a destructured assignment, just like CS already supports (syntactically). Is it not?

@lydell
Copy link
Collaborator

lydell commented Jul 27, 2016

Tip: Pretend that CoffeeScript's import/export is entirely different from JavaScript.

Alexander Pánek added 4 commits July 28, 2016 15:10
I think I'm getting a hang of how it's supposed to work. 🙌

What works now, as you can see in all the tests that are not commented out and
should run through:

```coffee
import foo
import bar from lib
import { foo } from lib
import { foo as boo, bar } from lib
import { foo, bar } from lib
```
@JimPanic
Copy link
Collaborator Author

I think I'm getting a hang of how it's supposed to work. 🙌

Dunno how much time I can spend with it the rest of this week though, because I will be in a place without internet (at all, not even mobile) over the weekend.

What works now, as you can see in all the tests that are not commented out and should run through:

import foo
import bar from lib
import { foo } from lib
import { foo as boo, bar } from lib
import { foo, bar } from lib

@JimPanic
Copy link
Collaborator Author

Sure!

  • from foo: yes, the tests all use 'module-name', I just wrote this out
    by hand and just forgot tbh. :)
  • OptComma is an optional comma that is also used the same way in array and
    object literals (I pretty much copied a lot of the array code).
  • default didn't work at the time, I wanted to see if it was because it's a
    reserved keyword or because the general construct doesn't work. In theory, the
    only thing needed to make it work is to add an if clause like for
    import, from and as to the identiferToken function.

lexer.coffee

identifierToken basically takes one word or symbol (read: @chunk) at a
time, assigns it a name or type and creates a token in the form of a token tuple
[ tag, value, offsetInChunk, length, origin ]. This is what the functions
token and subsequently makeToken create.

In identifierToken there are a few key variables and functions that are needed:

  • @chunk: the current string to handle, this is split up into
    [input, id, colon] with the IDENTIFIER regular expression at the
    bottom
  • id: in case of import, this is literally 'import'
  • @tag(): gets the tag (first value of the token tuple) of the last
    processed token. When processing foo (as in the second chunk of
    import 'foo'), @tag() will return 'IMPORT'.
  • @value(): gets the value (second value of the token tuple) of the last
    processed token. When processing foo (as in the second chunk of
    import 'foo'), @value() will return import, the very string that
    was held in id in the last chunk's handling.

So basically what I added to identifierToken was the tags IMPORT,
IMPORT_AS, IMPORT_FROM as well as the variable @seenImport to know
that when I encounter an as or a from, this will be from an import and
not a yield or similar. This also means in theory that from can still be
used as an identifier as well. We have to test that though. :)

These three tags are then used in grammar.coffee.

There's also code the reset @seenImport when the statement is terminated (in
lineToken iirc).

grammar.coffee

For this part I took a look at
the spec for imports
and basically copied the structure from there.

The DSL used here basically mixes and matches tags and named grammar forms. In
this case the tags are 'IMPORT', 'IMPORT_AS', 'IMPORT_FROM' as
replaced in lexer.coffee's identifierToken. The other parts of those
strings are just other named grammar forms (ImportsList, OptComma,
Identifier, etc.).

The structure builds up through references to other grammar forms and functions
that create and return data structures, like -> new Import $2. $n
variables are just references to the nth word in the string.

This process leads to an AST that is passed to the Import class defined in
nodes.coffee.

Off the top of my head this should look as follows:

# import 'foo' will yield something like:

new Import(Value { value: 'foo' })

# import { foo } from 'foo' will yield something like:

new Import(Value { value: 'foo' }, ImportsList { .... })

You can look at this AST quite easily by just prepending a console.log
before calling new Import:

Import: [
  o 'IMPORT String',                          -> console.log($2); new Import $2
  o 'IMPORT ImportClause IMPORT_FROM String', -> console.log($4, $2); new Import $4, $2
]

nodes.coffee

Taking the AST from grammar.coffee, the classes in nodes.coffee are
supposed to create tupels of "code" through @makeCode and compileNode
functions. I'm not entirely clear on this part yet, but each node is compiled to
a string by calling compileNode or compileToFragments. What
Import.compileNode basically does is just look at the AST and either return
an array of strings passed through @makdeCode directly OR it calls the
token's compileNode function.

This part is a bit of magic for me still, as there function names and processes
don't line up with my way of thinking it seems.


I hope this helps a bit. If there's anything else you don't understand, please
just ask!

@GeoffreyBooth GeoffreyBooth merged commit de78f48 into import-export Aug 1, 2016
@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 1, 2016

I have no idea why this is in twice now, it seems Github didn't send it three days ago but tried again now after I did so myself… neat, would be nice if it told me, though. Sorry for the double-post. ;)

@GeoffreyBooth
Copy link
Owner

No worries, it's been very helpful. I put your explanation here so it doesn't get lost. Let's keep editing that page as we figure more parts out?

I merged your branch into mine and then kept going. I didn't get too far, but everything I've committed I think is an improvement :) especially the more detailed tests and the syntax definition in the comments in test/modules.coffee. I also did some general cleanup, putting grammar definitions in order, etc. Please merge my branch into yours before you continue?

And we should maybe find a better place to keep a thread going than this now-closed pull request . . .

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 1, 2016

Awesome 👍

Yeah, we could keep hijacking the thread over at coffeescript6/discuss maybe? Or the issue/PR in the jashkenas repo.

@GeoffreyBooth
Copy link
Owner

Hey @JimPanic, I think this is probably a better place to discuss than hijacking either of those two threads.

I made a few more commits on the import-export branch. I think I have asterisks working, as in import * as foo from 'lib'. I also fixed some minor bugs.

We’re down to only two failing import tests. Both have to do with parsing commas:

  • import foo, { bar, baz as qux } from 'lib' (compiles to import foo from 'lib';)
  • import foo, * as bar from 'lib' (compiles to import foo from 'lib';)

If you look in grammar.coffee:359, ImportClause, you’ll see I’ve added a few more grammars for comma-delimited import members. That at least prevents the compiler from choking on the comma, but for some reason only the first element is ever output. I’m wondering if ImportClause, NamedImports and ImportsList need to be merged somehow; the list in ImportsList really should also apply to top-level members too, not just ones wrapped in {}. Any thoughts?

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 2, 2016

Awesome work! 🙌

I tried to find out what's going on, but I'm not entirely sure. It seems the grammar is correct. I changed a small detail so that it matches the spec to the point and it actually parses the tokens properly. It just seems to not generate the appropriate code from it. I don't have the code handy right now as I've been in meetings half of the day and don't have my work laptop here right now. I'll follow up tomorrow with the debug output I got, maybe you can make sense of it more than I do.

Btw, I like the pace we eventually got to! 👍 Feels productive :)

@GeoffreyBooth
Copy link
Owner

Okay, take a look at 05aa40b. All the import tests now pass.

This is actually simpler than the previous version. I realized that everything between the import and the from is just a comma-delimited list, some items of which are “wrapped” (enclosed in braces) and some which are not. As far as I can tell you can only destructure once and it must be either the first element or the last element, i.e.:

import foo, { bar as baz } from 'lib' # or
import { foo as bar }, baz from 'lib'

but not:

import foo, { bar }, baz from 'lib' # bad!
import * as foo, { bar }, { baz } from 'lib' # bad!

Because of this, I can simplify the arguments that ImportList can accept—it gets either one array or two, and each one is specified as being either wrapped or not. If my two “bad” examples turn out to be valid syntax, ImportList can be updated to accept an array of objects or something more elaborate. But basically most ImportList objects will have just the initial array, not two arrays, as I don’t think foo, { bar } happens all that often.

We need many more tests, for longer and more complicated (but still valid) variations; and we need tests that cover multiline import statements. But I think we’re well on our way.

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 3, 2016

Niiiiice! 👏

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 9, 2016

Okay, take a look at 0bb6dcd. Most of the export syntaxes are now implemented:

  • export { foo, bar }
  • export { foo as bar, baz as qux }
  • export default foo = 'bar'
  • export default ->
  • export { foo as default, bar }
  • export * from 'lib'
  • export { foo, bar } from 'lib'
  • export { foo as bar, baz as qux } from 'lib'

What doesn’t work, because of ambiguities in the grammar:

  • export foo, bar
  • export foo = 'bar', baz = 'qux'

I’m thinking of excluding these two syntaxes. First, they look suspiciously similar to var foo, bar which CoffeeScript doesn’t support. More importantly, though, neither one are allowed by the Babel REPL, and I’m not sure what the benefit is of this syntax over export { foo, bar }. I assume there can’t be any functionality lost by not supporting these variants, since if Babel doesn’t support them then no one is using them yet?

There are some bigger fish that don’t work, though, namely multiline imports or exports. I commented out the multiline tests that are failing; @JimPanic, do you have time to take a look at the multiline-related code and see if you can figure it out? That code is now living here.

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 9, 2016

Oooh, you just beat me to it. I wanted to start implementing the export grammar just now haha.

This is great progress! I will take a look and see if I can figure it out.

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 9, 2016

Yeah, we should probably figure out a way to not overlap. You know, communication 😃 Though I'm so many time zones behind you that we don't overlap our working hours much. It's all yours for 18 hours at least 😃

@GeoffreyBooth
Copy link
Owner

Okay, I’ve made some progress:

If an import or export token is detected in a file, bare is set to true. This resolves the issue I discussed with @lydell over here. 858c7b5

Exporting a block, or at least the one example I’ve tested, now works:

export default (foo) ->
  console.log foo

becomes

export default function(foo) {
  return console.log(foo);
};

Though I’m confused by how this works. It’s detected by the grammar EXPORT EXPORT_DEFAULT Expression, but not EXPORT EXPORT_DEFAULT Block or EXPORT EXPORT_DEFAULT Expression Block. I would think that a multiline function should show up as a Block, but it doesn’t 😕

Still to do: multiline import. Have you looked into that at all @JimPanic? I think we could consider it optional, if it proves too difficult to implement.

@JimPanic
Copy link
Collaborator Author

I haven't looked into it yet, no. I had too many meetings to focus on anything the last days. :\

Re Expression/Block: that might be because the grammar sees a function as Expression (=Code) which in turn has ParamList and Block. Maybe Code or Expression should be used instead of Block?

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 11, 2016

Two things cause the multiline to break it seems:

  1. lineToken in lexer.coffee has @seenImport = no in it
  2. the codegen strips all newlines of imports:
~/p/coffeescript> bin/cake test
coffee input:  import foo, {
  bar,
  baz as qux
} from 'lib'
compiled input:  import foo, { bar, baz as qux } from 'lib';
failed 2 and passed 677 tests in 2.39 seconds 

@JimPanic
Copy link
Collaborator Author

I have absolutely no idea why those whitespace characters are stripped; shouldn't they be part of the AST? Or maybe we should opt to only outputting single-line imports, even though the source might be multi-line? Because multi-line imports get parser and generated correctly besides the whitespace issue as soon as @seenImport = no is removed from lineToken in lexer.coffee.

@lydell
Copy link
Collaborator

lydell commented Aug 11, 2016

A function is an expression. The grammar is export default should be followed by an expression. That's why it works. :)

AST:s usually strip or irrelevant information, such as whitespace.

Having entire import statements on one line in the output is fine. Perhaps we should do it like object literals, though, which are always multiline in the compiled ouptut, regardless of the input.

Don't forget to add support for export <statement>, such as export class Foo.

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 12, 2016

@lydell, per the export spec on MDN, export default can take only an expression, function, function* or class.

I don’t think it’s practical to test for export default class at the moment, though it does work (I think):

export default class foo extends bar
  baz: ->
    console.log 'hello, world!'

becomes:

var extend = function(child, parent) { for (var key in parent) { if (hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor(); child.__super__ = parent.prototype; return child; },
  hasProp = {}.hasOwnProperty;

export default foo = (function(superClass) {
  extend(foo, superClass);

  function foo() {
    return foo.__super__.constructor.apply(this, arguments);
  }

  foo.prototype.baz = function() {
    return console.log('hello, world!');
  };

  return foo;

})(bar);

(b211a51) Technically this is export default function, which is on the approved list. But there’s so much generated code here that I don’t want to create a test that compares input/output strings when the output is so subjective. Almost any changes to how CoffeeScript class is output (to say nothing of when ES2015+ class is supported) could break such a test.

We have a test for export default foo = 'bar', are there any other types of expressions worth writing tests for?

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 12, 2016

@JimPanic, thank you for finding the extraneous @seenImport = no! I took @lydell’s suggestion and went in the opposite direction, always outputting destructured members across multiple lines (8c08ae4), similar to how CoffeeScript currently outputs object definitions. So now:

import { foo, bar as baz } from 'lib'

becomes:

import {
  foo,
  bar as baz
} from 'lib';

which is pretty similar to what you’d get from foo = { bar, baz }. This is a double win: we get consistency with CoffeeScript’s current output, and this solves the multiline parsing problem. The ModuleList.compileIdentifiers method no longer needs to detect whether the input is multiline or not; it just always outputs the identifiers as a multiline block. Now all the tests pass!

@lydell, I took your suggestions and simplified our tokens: 43b0248, b211a51, 6b27e23.

What else remains to be done? SimpleString? I’m sure we need more tests—what for? Should we bring Babel into the project and redo the tests so that our compiler’s output goes through Babel and then gets evaluated? What mistakes should we catch and output error messages for—perhaps import/export statements that aren’t at the top level? Anything else?

@GeoffreyBooth
Copy link
Owner

Also I want to submit this as a PR soon, even if these edge cases aren’t solved just yet. Should I commit the .js files in lib? Should I write documentation?

@rattrayalex
Copy link
Collaborator

Hmm, how would this handle

foo = 1
export foo = 2

and

export foo = 2
foo = 3

?

I imagine that the first case will fail, which I think it should (though we might want to ensure a helpful error message) and the second case will work just fine, which it also should.

Worth adding tests?

@GeoffreyBooth
Copy link
Owner

The first example becomes:

foo = 1;

export var foo = 2;

which Babel converts to:

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.foo = foo = 1;

var foo = exports.foo = 2;

The second example becomes:

export var foo = 2;

foo = 3;

which Babel converts to:

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
var foo = exports.foo = 2;

exports.foo = foo = 3;

So both are valid JavaScript, but both end up with foo declared in the global scope, which isn’t good.

@lydell
Copy link
Collaborator

lydell commented Aug 15, 2016

It’s fine to create a PR even if you’re not 100% finished. That’ll attract more reviewers.

Yes, do commit the compiled .js files.

Documentation is usually written right before a release. I’d leave that out from your PR.

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 15, 2016

Okay, 1e0714e passes all the from and as edge cases. The lexer is now much more discerning about when it recognizes from as a 'FROM' token and as as an 'AS' token.

I also added several more tests regarding variable assignment, like:

foo = 5
import { bar } from 'lib'
baz = 7

which should produce just a var foo, baz up at the top, but doesn’t. My code regarding insideModuleStatement inside Assign.compileNode in nodes.coffee:1376 clearly needs improvement. It would help if I had any idea how that function works. @lydell, do you know how I might be able to add an identifier to the list of “seen identifiers” without that identifier getting output in a var line? That’s what I was trying to do with my insideModuleStatement code, but clearly my test is too broad and I’m excluding everything and not just the import line in this example. I can’t split on terminators, either, since we’re supporting multiline import and export statements.

Also I don’t think @seenImport/@seenExport are getting reset properly. They can’t get assigned to no on every 'TERMINATOR' token; how do I detect that the module statement is truly finished and therefore tell the lexer to reset them back to no?

@rattrayalex
Copy link
Collaborator

how do I detect that the module statement is truly finished and therefore tell the lexer to reset them back to no?

my hunch is setting a flag at an IMPORT and unsetting it at a String might work, since all import statements must end with a string, and can't have any strings in the middle. I'm looking into doing that now. Hopefully you'll finally see a patch from me on this soon.

So both are valid JavaScript, but both end up with foo declared in the global scope, which isn’t good.

hmm, doesn't seem desirable. per @lydell 's comments above, seems fine to submit the PR without that working, but might be good to write failing tests describing better behavior first. I'll try to do that today if I can.

@GeoffreyBooth
Copy link
Owner

GeoffreyBooth commented Aug 15, 2016

@rattrayalex, there are failing tests in 1e0714e.

The code inside lexer.coffee StringIdentifier, where the 'FROM' is set, could indeed reset @seenImport; and @seenExport for the ExportImport grammars. But what about plain Export and ExportDefault?

@rattrayalex
Copy link
Collaborator

But what about plan Export and ExportDefault?

Assuming you meant plain there. Good question; sorry I missed that.

Let's see, the grammar is EXPORT Identifier = Expression or EXPORT DEFAULT Expression, so we want to find the end of an expression... I'm not sure how to do that but digging into it now...

@rattrayalex
Copy link
Collaborator

rattrayalex commented Aug 15, 2016

there are failing tests in 1e0714e.

I see import tests there, but not export tests... Sorry if I'm being dense/unhelpful here 😥

I know I at least have gotten my imports and exports mixed up a few times in this conversation, especially around assignment

@rattrayalex
Copy link
Collaborator

ah, as for detecting EXPORT Identifier = Expression, I am going to try adding setting the context of the Assign to 'export', and looking at that when deciding whether to declare the var

@rattrayalex
Copy link
Collaborator

(made a first stab at resolving this, linked to a patch in 7a3113e#commitcomment-18643729 )

@JimPanic
Copy link
Collaborator Author

JimPanic commented Aug 16, 2016

Wow, amazing how much all of you have done since I left for vacation just on Friday! Kudos! 🔝

@GeoffreyBooth
Copy link
Owner

cfc18c4 applies @rattrayalex’s patch and makes a few tweaks. All the tests now pass. But I’m not sure all of this is the best way to implement things.

@lydell, perhaps you could provide some guidance here. So @rattrayalex got this to work:

o 'EXPORT Identifier = Expression', -> new Export new Assign $2, $4, 'export'

but further down there’s syntax like this (as if I applied it to our line):

o 'EXPORT Identifier = Expression', -> new Export new Assign LOC(1)(new Value $2), $4, 'export',
                                                      operatorToken: LOC(2)(new Literal $3)

What is going on here? Do we need to do any of this—perhaps for proper source maps? What is LOC? I don’t even see that defined anywhere.

Also, is context the parameter we should be using to tell Assign that this assignment is within an import or export statement? I question using it since two blocks expect @context to be falsy for a simple assignment like var foo = 5.

@GeoffreyBooth
Copy link
Owner

@lydell my questions in the previous comment still stand, but I revised things in c3893d7. Now instead of using the context parameter, I created a new option moduleStatement that can be passed to Assign:

exports.Assign = class Assign extends Base
  constructor: (@variable, @value, @context, options = {}) ->
    {@param, @subpattern, @operatorToken, @moduleStatement} = options

Since I think what we’re doing here is more similar to how @param can be true, than a new context like object. This also lets me modify the original code a lot less, as I don’t have weird exceptions where things happen if @context is falsy or @context is import or export, which feels wrong.

I also set Assign.isStatement to return true if @moduleStatement is set. I’m not sure what effect isStatement really has, and if I should be saying an import or export statement really is a statement. Please advise 😄

@lydell
Copy link
Collaborator

lydell commented Aug 17, 2016

Remember that a home made grammar DSL is used in grammar.coffee. All sorts of crazy hacks are used to get it going. All of that is set up at the top of the file. For example, LOC doesn’t “exist.” When the functions passed to the o functions are turned into strings, LOC is replaced with other code. I don’t really know exactly what LOC does. Haven’t really worked with it (I think).

operatorToken was something that I added some time last year or so to improve some error messages. Don’t really remember the details now, though.

Regarding @context, I think it has to do with the fact that Assign is also used for foo: bar in object literals.

I really care about import/export coming to CoffeeScript, so I gladly help out here. Unfortunately, I haven’t really had the time lately to code on it myself. But remember that I am by no means an expert on the CoffeeScript compiler. I started contributing relatively late (the end of 2014 or something) and I’ve learned most things by hacking around, grepping and really trying to understand stuff. Sometimes there are really helpful comments that help, too.

So many times when I read your questions I’m like “oh yeah, I remember that thing, but I’m unsure of the details now – I’ll just go grep for it in the source code before I answer – oh wait I need to this first…”

@GeoffreyBooth
Copy link
Owner

Well @lydell I appreciate whatever wisdom you have, it’s much more than I have. 😄 Better to at least ask than find out when people point out bugs.

I feel like I shouldn’t be so mystified since there’s so much documentation and so many detailed comments, yet grepping through all the comments I don’t see anything that explains a block like this for the Module class:

  isStatement: YES
  jumps:       THIS
  makeReturn:  THIS

These are just guesses based on an assumption that Module should use the same values as Return. But I have no idea.

I guess it’s about time to submit as a PR, and maybe the greater wisdom of a greater crowd might uncover more mistakes?

@rattrayalex
Copy link
Collaborator

As sorta-discussed on another thread, I'm not so sure the @moduleStatement is the right approach... it also doesn't seem to be used with import yet?

@rattrayalex
Copy link
Collaborator

Oh and thanks very much @GeoffreyBooth for cleaning up + integrating my patch!! Very gratifying 😄

@GeoffreyBooth
Copy link
Owner

@rattrayalex it doesn’t need to be used with import by definition, since imported members can’t be assigned. See discussion here. I added “support” for import inside Assign mainly so that we could potentially throw our own error about assigning to a read-only variable, if we don’t want to delegate that error to Babel or the runtime.

@lydell
Copy link
Collaborator

lydell commented Aug 17, 2016

One thing I just come to think of: Do we disallow import/export in non-toplevel scopes yet? Otherwise that's something that needs to be done.

@GeoffreyBooth
Copy link
Owner

@lydell, 3eb4533 adds a check for non-top-level scope. It’s just checking that o.indent.length is 0, which feels like a brittle check, but I’m not sure how else to do it. It passes all these tests, at least.

I have some ideas for more robust testing for this soon-to-be-PR:

  • https://github.com/pauc/piqu-ember is an Ember app that uses ember-cli-coffees6, which means it’s one of the rare open-source projects that has import and export statements in CoffeeScript, unbackticked. I plan to diff its current generated JavaScript against what gets generated by removing ember-cli-coffees6 and replacing coffee-script with the version in this PR; and of course see if the app still runs.
  • I’m responsible for the coffeescript fork of the Meteor Todos demo app. I’ll update that fork to use import and export statements instead of CommonJS, and somehow swap out the build of CoffeeScript used by the Meteor compiler, and see if the app still compiles and runs.

Does anyone else have any more serious CoffeeScript apps that currently use modules in some way (CommonJS, backticks, something else)? Especially if your app has tests. I think trying this code on a real app with real modules is the way to really shake out whatever edge cases and bugs remain.

@rattrayalex
Copy link
Collaborator

@GeoffreyBooth that sounds sufficient to me, and probably something that can happen once you've submitted the PR – but if you want more, I might suggest finding some well-tested demo apps out there and replacing require()s with imports, and just see if the tests still pass. That's likely how most people will be using it in any case. Obviously it's also a bit more work, and unfortunately I don't personally have / know of repos that would qualify.

@GeoffreyBooth
Copy link
Owner

Okay, I recompiled piqu-ember using my branch, and here’s the diff between the original output via ember-cli-coffees6 and via this branch. Basically, it seems to have worked. The app still loads without errors. Here’s the repo so you can try it for yourself (and see how I did this test, if you follow the instructions in the README).

The exercise did catch one bug—I was inadvertently gobbling up all *s within an export expression, even *s within an export default function, as IMPORT_ALL tokens. Now I check that not only are we inside an import or export statement but the indentation level is 0, so we’re not inside a block (857b603). Again it feels hacky to check @indent, but I don’t see a better way (or a case where this wouldn’t work). I wish I could replace all IMPORT_ALL tokens in grammar.coffee with *s, but that doesn’t seem to work for some reason.

@GeoffreyBooth
Copy link
Owner

I finished the other test, of rewriting the Meteor Todos CoffeeScript example app to use unbackticked import and export statements and seeing if Meteor (patched to use this PR) could compile and run the app. It does. Try it yourself:

git clone git@github.com:GeoffreyBooth/coffeescript-modules-test-meteor-todos.git \
  && cd coffeescript-modules-test-meteor-todos \
  && ./init.sh

Of course, read init.sh first so you’re not just blindly trusting me. But it’s safe, it doesn’t modify any files outside of the folder created by cloning the repo.

@rattrayalex
Copy link
Collaborator

Nice! Need any help before submitting a PR?

@GeoffreyBooth
Copy link
Owner

I don’t think so, just need to find the time to write a proper description. I’ll do it today. Thanks!

lydell pushed a commit to jashkenas/coffeescript that referenced this pull request Sep 14, 2016
This pull request adds support for ES2015 modules, by recognizing `import` and `export` statements. The following syntaxes are supported, based on the MDN [import](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import) and [export](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export) pages:

```js
import "module-name"
import defaultMember from "module-name"
import * as name from "module-name"
import { } from "module-name"
import { member } from "module-name"
import { member as alias } from "module-name"
import { member1, member2 as alias2, … } from "module-name"
import defaultMember, * as name from "module-name"
import defaultMember, { … } from "module-name"

export default expression
export class name
export { }
export { name }
export { name as exportedName }
export { name as default }
export { name1, name2 as exportedName2, name3 as default, … }

export * from "module-name"
export { … } from "module-name"
```

As a subsitute for ECMAScript’s `export var name = …` and `export function name {}`, CoffeeScript also supports:
```js
export name = …
```

CoffeeScript also supports optional commas within `{ … }`.

This PR converts the supported `import` and `export` statements into ES2015 `import` and `export` statements; it **does not resolve the modules**. So any CoffeeScript with `import` or `export` statements will be output as ES2015, and will need to be transpiled by another tool such as Babel before it can be used in a browser. We will need to add a warning to the documentation explaining this.

This should be fully backwards-compatible, as `import` and `export` were previously reserved keywords. No flags are used.

There are extensive tests included, though because no current JavaScript runtime supports `import` or `export`, the tests compare strings of what the compiled CoffeeScript output is against what the expected ES2015 should be. I also conducted two more elaborate tests:

* I forked the [ember-piqu](https://github.com/pauc/piqu-ember) project, which was an Ember CLI app that used ember-cli-coffeescript and [ember-cli-coffees6](https://github.com/alexspeller/ember-cli-coffees6) (which adds “support” for `import`/`export` by wrapping such statements in backticks before passing the result to the CoffeeScript compiler). I removed `ember-cli-coffees6` and replaced the CoffeeScript compiler used in the build chain with this code, and the app built without errors. [Demo here.](https://github.com/GeoffreyBooth/coffeescript-modules-test-piqu)
* I also forked the [CoffeeScript version of Meteor’s Todos example app](https://github.com/meteor/todos/tree/coffeescript), and replaced all of its `require` statements with the `import` and `export` statements from the original ES2015 version of the app on its `master` branch. I then updated the `coffeescript` Meteor package in the app to use this new code, and again the app builds without errors. [Demo here.](https://github.com/GeoffreyBooth/coffeescript-modules-test-meteor-todos)

The discussion history for this work started [here](#4160) and continued [here](GeoffreyBooth#2). @lydell provided guidance, and @JimPanic and @rattrayalex contributed essential code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants