Asynchronous coffeescript made easy, part III

[timbertson]

previous issues:
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=narrative#issue/241
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=defer#issue/287

So, it's back!

For those who didn't read or can't remember the previous discussions, I have been working on adding a "defer" semantic into coffee-script. This is aimed at making asynchronous coffee-script less painful, by providing a continuation-like callback object for the current function (at compile-time).

For example:

myfunc: (cb) ->
  data: defer call_some_ajax('/path/to/relevant/data')
  alert("got $data")
  cb(data)

Would be transformed into the following javascript (or its equivalent):

function myfunc(cb) {
  call_some_ajax('/path/to/relevant/data', function(data) {
    alert("got" + data);
    cb(data);
  });
}

Special care has been taken such that the following use-cases work as expected:

• multiple return arguments ([err, result]: defer some_call())
• defer calls within:
  • if/else statements
  • switch statements
  • for loops
  • while loops
  • nested expressions.

These are non-trivial, so there may be bugs if you do something utterly weird - let me know! ;)

So, please do check out my deferred branch (http://github.com/gfxmonk/coffee-script/tree/deferred) and let me know what you think. Is it a good idea? Should it be (eventually, after some more cleanup) merged into master? Are there any glaring omissions or bugs? Be sure to look at test/test_deferred.coffee, it has over 30 different tests ensuring that as many language features as I could think of work when defers appear in various locations.

(note that the tests in this branch rely on my coffee testing project, coffee-spec (http://github.com/gfxmonk/coffee-spec). I couldn't have managed this many complex tests without it, and it's hopefully coming to coffee-script itself officially sometime soon)

Stuff not yet done

• I have not yet added the ability to use defer in place of a positional argument - it shouldn't be too hard though.
• There's a fair bit of cleanup that can be done on the code, as well as some possible optimisations on the output of code - I was going for correctness, not efficiency.
• Some form of trampolining to prevent the stack growing indefinitely. This is easy enough to do manually, and I still have no idea how often it will be necessary.
• (potentially) adding a callback argument to a function definition when the compiler detects that one is required. This was a controversial suggestion in my first proposal, so it's been left out for now.

So... thoughts?

[jashkenas]

Here's a direct link to the test_deferred.coffee suite, for folks who'd like to take a peek:

http://github.com/gfxmonk/coffee-script/blob/deferred/test/test_deferred.coffee

[mrjjwright]

Exciting, I hope to try this soon, great work!

[mrjjwright]

I couldn't get this branch to compile, http://gist.github.com/387224. Any ideas? I am sick of nesting code and using 3rd party libs like step and flow for this so I am anxious to try this.

[timbertson]

@mrjjwright: might have been a bad merge, I'll check it out tonight and let you know.

[timbertson]

done! (just pushed). Not entirely sure what the problem was, but it seems to be sorted now. Apologies for that.

[mrjjwright]

It compiles now, thanks.

[mrjjwright]

I tried it on a place today where I was planning to use flow.js. It worked fine. I liked it because I didn't need a library to pull this off. I am always worried that there is a hidden bug in flow.js or step.js. There could of course be bugs in the defer code as well but I am assured that all the code is right there to inspect if it is wrong, and it clear what it is doing. Both flow.js and step.js use underlying state machines that are harder to debug.

The defer keyword definitely illustrates at least theoretically the beauty of CoffeeScript to provide a language solution to an awkward JS usage problem. A developer would otherwise use a library but I think directly generated code is a lot easier to inspect and trust than libraries. I don't understand why developers are reluctant to try CoffeeScript but will gladly pop in any third party library that is barely documented.

I don't want to get too addicted to this yet but I can see how I could easily do so.

+1

[jashkenas]

mrjjwright: for the sake of discussion, it would be great if you could paste in a bit of one of your real-world conversions to the "defer" syntax. I think a lot of the discussion that we'll have will be better informed by "before" and "after" examples with real code...

[mrjjwright]

Ok I am going to try this on what is surely a great test case, a SQLite table migration where you dump all the results of one table to another and then re-import them. This involves about 15 levels of nesting as well as a nested function definition in a very deep defer. I have this working in flow.js and tried migrating it to defer but running into a compilation issue that I have pasted in the comments below the gist (I could be doing something really naive but couldn't find it).

http://gist.github.com/388519

I will probably need your help gfxmonk to get this compiling.

[DanielVF]

If this were in coffeescript, I'd be using it now. Even with the all the virtues of asynchronous programming there are still many of things that need to be done serially. This makes clear, readable code out of it.

+1

[mrjjwright]

gfxmonk helped me get it to compile and it worked like a charm.

http://gist.github.com/388519

A few things to summarize from my experience.

• My code has virtually no nesting now. It didn't have much before because of the flow.js library but now I know the underlying javascript is simply async nested code.
  As I kept adding defer after defer I was wondering if the code generator could handle it. It felt weird but I went with it and it generated all the right levels. This saves tons of keystrokes over plain JS with brackets.
• There is no built-in error handling. I still need to check for a return error from each async call after each invocation and handle them. Not a big deal but just saying.
• Will future readers of my code easily understand defer? Sure if it's documented but it's going to be a bit odd at first. It's a lot better than following nested calls. All you have to do is look up to see where the variables are declared and you can see that the variable was set as a return of an async call.
• I personally would prefer a word like async. I just think it would read cool and make me kind of smug in a good way every time I type it. defer sounds a bit inaccurate and weird to me.

jashkenas I am wanting this, it's going to be hard to not lobby you. But this is my first day with this and I haven't heard or read all the naysayers yet.

[weepy]

I made a suggestion about syntax in an old thread - I'll just repeat it here ...
.... instead of

[err, data] = defer get "/", {}
my_async(x, y)
[err, data] = defer more a, b
setTimeout(defer, 100)
final()    

use a loong arrow

[err, data] = get "/", {}, -->
my_async(x, y)
[err, data] = more a, b, -->
setTimeout -->, 100
final()

I think it neatly encapsulates the idea of 'look to the next line' and also that it's a bit like a function.

[timbertson]

mrjjwright - you might be interested reading the old threads for discussions we had around naming. async has been suggested, but I don't really feel it explains what the keyword does. "call-with-current-continuation" is the best name for it - that comes from smalltalk, but even there it was contracted to call_cc in actual code (which is almost meaningless)

I've been discussing this with a friend, and a couple things should also be pointed out as shortcomings:

• loops containing a defer statement will be transformed into recursive calls. If the deferred function you call is not actually asynchronous, you may well blow the stack. But since an asynchronous return (via ajax, or some other browser callback mechanism) throws away the existing stack anyways, this should not be a problem too often. It can be fixed manually if need be, with appropriately-placed deferred calls to setTimeout.
• as I think we've discussed before, there is no plan to handle exceptions (as you've seen, mrjjwright). They can't easily be handled on a global level - errors in callback-based functions must be caught and explicitly returned into the callback. I've been thinking about ways to make this easier, but I haven't come up with anything compelling yet.

weepy Being a python guy, I typically prefer meningful words over punctuation. Plus, I think your suggestion makes it too easy to accidentally start a function (by forgetting a "-").

But if we move to having the "defer" keyword replacing positional arguments (rather than prefixing function calls) then defer wouldn't necessarily be a great choice, so suggestions are welcomed.

[timbertson]

oh, and:

Will future readers of my code easily understand defer?

Most people dealing with javascript callbacks would already understand continuation-passing-style, and hopefully most have longed for or thought about a way to make that automatic. So hopefully this feature should seem like a fairly natural transformation to most javascript programmers that would encounter it.

[timbertson]

by way of further example, here's an actual javascript file I wrote using lawnchair (a heavily async datastore). I've ported it to both coffe and coffee+defer (reasonably idiomatically, aside from some unimportant string concatenation):

http://gist.github.com/389375
it's easier to see the difference if you open them side by side in a text editor - the bottom "run" method in particular is far more straightforward with defer.

[mrjjwright]

Hey gfxmonk,

Ok, defer is fine as a keyword. I defer.

Looping issues, no big deal either, I am always really careful about async calls in a loop. (Btw, ever seen this: http://glathoud.easypagez.com/publications/tailopt-js/tailopt-js.xhtml. Any value here?)

Will you be merging all major bug fixes into deferred from the main line for a while so I can use this branch? I am kind of hooked. If not, I will go back to my old crack async dealer, flow.js.

[weepy]

How about wrapping setTimeout for numerical values? E.g.

defer 50
# do something in 50ms

You might counter that it wouldn't work for variables that are numeric, but 99% of the time setTimeout's are used with a hard coded numeric value.

[timbertson]

weepy: I think that's more of a library issue - special casing in the parser to deal with deferred numbers instead of deferred calls seems a bit of a strange way to handle it. You could do this with a simple library function:

sleep: (delay, cb) -> setTimeout(cb, delay)

then just use it as:

defer sleep 50

mrjjwright: I plan to keep the deferred branch up to date (maybe at a lag of a week or so, because merging is not exactly my top priority). However I can't promise that the syntax won't change (that is, after all, part of the point of this issue / discussion).
Having said that, I plan to port my currently in-progress javascript-heavy webapp to using defer when I get the time.

[drnic]

I'm so giddy with excitement over the possibilities for readable nodejs code (as an example)

[timbertson]

Okay, I've just pushed a fairly significant change, which cleans up and clarifies the tricky bits of the original attempt. I now believe my branch is in a good state to consider merging into the master. Pity that jeremy has just disappeared for a couple weeks, but at least I shouldn't have to do too many more big merges while he's gone (they are not much fun).

If you want to see the differences, you can take a peek here:
http://github.com/gfxmonk/coffee-script/compare/master...deferred#diff-8
(nodes.coffee is where most of the changes have been made)

It's a big diff, but it's also the first change of its kind. Most of the CS compiler translates one thing into a more primitive version of it (i.e javascript). The deferred machinery, on the other hand, does a lot of rewriting nodes into a manner that will work when calls are "resumed" after an asynchronous call.

The first part of the machinery is basically to pull out deferred calls to the beginning of an expression. That is, if a complex expression contains a deferred call then that call is executed first, and the rest of the expression gets evaluated inside the callback provided to the function. This makes all deferred results available at the time when they are needed - because you can't just pause execution in the middle of an addition operation, for example.

The second (and quite unsightly) part of the machinery is the rewriting. Just as you can't pause execution in the middle of an addition operation, you also can't fire off a call from within an if block and expect your control-flow to still work when your callback is called. In fact, the callback would have to include the rest of the if branch, as well as any operations that sit after the end of the if block in the original code.

So that's where make_control_flow_async comes in. This method is implemented on all nodes. By default it just propagates the call to its children. But in the case of nodes that affect control flow (IfNode, WhileNode, ForNode), it does something pretty invasive. Basically, there's a way to transform each of these nodes into a "flattened" version which manages control-flow by explicit continuations. This method (for each of those nodes) generates the CoffeeScript node objects after that transformation is applied. Because they build up a reasonably different source tree, I've introduced a couple of builder objects (down the bottom of nodes.coffee). I can move them into another file if people feel they clutter the already-huge nodes.coffee file. These builders have named methods for common rewriting operations that make it more obvious what transformation is happening, and why.

I'm happy to explain specific parts of the implementation, if people are interested.

[mckeed]

I just came across this; it's very exciting. I'd like to share my vote for positional argument syntax:

defer setTimeout(continue, 100)

The defer introduces the call that is being deferred to, and (conceptually) binds continue to the continuation that begins on the next line. If you are using this syntax, it is because the function being called focusses on the callback, and I think continue is the clearest description of what the continuation-as-callback does.

Obviously, this could be a problem since continue is already a statement, but I don't think there will be overlap, and it is kind of similar, both being control constructs. Other possibilities are resume, continuation, or cc.

[timbertson]

mckeed: continue could be confusing because of the existing use (even if it's not ambiguous to the compiler).

jashkenas: any thoughts on the possibility of merging this in sometime? You haven't said a word on this since you came back, not sure if you're busy elsewhere...

[weepy]

I was wondering about an alternative syntax using the <- symbol. Something like

err, data:  mongo.find {user: 'nicky'}, <-
throw err if err
process data

So the idea is that the arrow points the other way indicate that the tree branch is being unwound, but keeps parity with the normal CoffeeScript.
It also allows for arbitrary positioning of the callback, e.g.

setTimeout <-, 100
run_delayed_code()

I think it works nicely because it still mostly looks like normal coffeescript (I found defer to be a bit ugly)

[jashkenas]

Hey gfxmonk. I've been trying to knock out most the smaller tickets before tackling the big ones, like defer.

There are a couple things you could do to facilitate this...

• Bring the patch up-to-speed with the latest master (a big pain in the ass, I know, I'm sorry).
• Make sure the tests that are marked as "these ones cause compile errors at the moment" are now working.
• Comment in here about some of the parts of the implementation that you think look tricky and/or debatable, and explain why they're necessary.
• Although test_deferred.coffee is supposed to test all of the possible scenarios, it would be good to have a reference Gist that shows the ways you are supposed to use deferred.

[weepy]

I use => quite alot, so it would be important (for me) to be able to be able to specify that the function should be bound.

[timbertson]

@weepy: I think it works nicely because it still mostly looks like normal coffeescript (I found defer to be a bit ugly)
It's probably unwise to make it look too much like coffee-script, as it still has different semantics (return, for example). I personally get too confused by a language having every imaginable type of ascii arrow, so I'd prefer to steer away from "<-".

As for "=>", I'm not sure what it would mean for a deferred function to be bound - that's about function definition, not a function call...

@jashkenas: geez, that's a lot of changes. I get distracted writing android apps and all the underscores die ;)

• merging: I've had a go at it. but am finding it difficult to pinpoint why tests are now breaking. I'll keep going...
• The ones that cause compiler errors aren't actually written yet. the functionality as it stands is complete, the only optional bit is to have a keyword stand in for a positional argument in the unlikely case of not having the callback argument as the last one. To be honest, I don't think that functionality is necessary, so I'd rather not spend time on it and all the merge hell it entails. If people want it, I would suggest adding it after the branch has been merged in (it should be a non-breaking change...)
• as for the tricky parts of the code, I've detailed how the compiler transforms work in my comments above ( http://github.com/jashkenas/coffee-script/issues#issue/350/comment/242851 ); I don't think there's any tricky or debatable stuff that I haven't mentioned. Feel free to look at the diff and point out any bits that look suss to you though...
• in terms on ways to use deferred, the actual usage is very simple (the tests are just for completeness). Here's a gist: http://gist.github.com/445525

[weepy]

ah - after seeing your examples, i can see that my comment about => isn't really appropriate.
so am i right that:

• comprehensions work
• returns work
  also what is the value of this ?
  it would be great to see some more of those examples you've got there - perhaps along with the exact output JS ? -- so we can get a good feel for it.

[jashkenas]

gfxmonk: Can you explain how defer changes return semantics in your branch? (Presumably other semantics like break and continue) after a return as well... I can see how they'd be problematic, but I'm not precisely sure how you're handling it.

[timbertson]

weepy: yep, that is correct.
As for this, it's what you'd expect - it's restored after a defer - that is, whatever this is at the start of the function is maintained after making a deferred call. this inside a deferred function call will be whatever it normally is (i.e you can use the fat arrow to bind this at definition time, as you already do).

The exact JS output is not exactly beautiful, but I've now added it to the gist for illustration's sake ( http://gist.github.com/445525 )

jashkenas; regarding return. With defer, you're still writing your functions in a callback style. That is, you take a callback, and you call it with one or more arguments instead of returning a value. The defer machinery makes this look nicer on the call side, so that you don't have to pass an actual function as the callback, it is constructed for you.

Keeping that in mind, there's no difference in return semantics to asynchronous coffee-script. If you return instead of calling your callback, execution will silently cease. So it's the same as normal async code, but that's obviously different to plain-old-procedural coffee-script, which is why I was discouraging making that the defer resemble procedural code too closely.

I'm happy to report that continue and break are indeed problematic, but as far as I know, they are handled properly in all cases :)

[jashkenas]

gfxmonk: looks like a couple of things need to be cleaned up in the generated code. To quote:

        return _h(undefined);            //continue;
        return undefined;
      });
    };
    return _h(undefined);
    return _g(undefined);

No need to double-return in either of those places, is there?

Also, for the common-case defer, it would be great to write this:

myFunc = function(callback) {
  var _a;
  someFunction(1, 2, 3, function(_a) {
    var result;
    result = _a;
    doSomeThingWith(result);
    return callback(true);
  });
};

As this, without the variable juggling:

myFunc = function(callback) {
  someFunction(1, 2, 3, function(result) {
    doSomeThingWith(result);
    return callback(true);
  });
};

[thejefflarson]

gfxmonk:

Let me first say that the defer branch is a fantastic feat, there's some nice stuff in there.

But I will say that my example was the final api of a library, just as defer would be a final feature of coffeescript. I should have thought it out a bit more and I'd write what I meant, but I see karl has beaten me to the punch.

Three things:

1. I'm not saying to use another language. I know we're all polyglots, but by the very fact that we're here, we each love javascript a whole bunch. What I am saying is that hiding the nature of javascript because other languages have a feature is a non-argument.

2. In a list comprehension coffeescript combines primitives to accomplish its task. From the docs:

globals: (name for name of window)[0...10]

becomes this coffeescript:

var _a, _b, globals, name;
var __hasProp = Object.prototype.hasOwnProperty;
globals = (function() {
  _a = []; _b = window;
  for (name in _b) { if (__hasProp.call(_b, name)) {
    _a.push(name);
  }}
  return _a;
})().slice(0, 10);

There's nothing in there that is not explicitly defined in the ECMAscript spec. Now while the ECMAscript spec does talk about callbacks (forEach for example), it does not define a true nature for callbacks. To put it a little more clearly: callbacks are a corollary to the language not a primitive.

3. And this is the most important part. I still believe that it's a mistake to make code look iterative when it is recursive, which is the reason for being for defer. A traditional (iterative) program runs like so:

statement
statement
statement
statement
alwaysExecuted()
exit # end of program
neverExecuted()

A javascript (recursive-functional) program with callbacks runs like so:

var _cb = function(){ neverExecuted() }
func(args, function(){
  process.exit(); // end of program
  _cb()
});
alwaysExecuted(); // as long as func is truly async

Now with defer it runs like so:

defer func args
process.exit()
return neverExecuted()

alwaysExecuted() // uh oh

This is a cognitive problem. When I write async code, I assume that sometime in the future the code will be executed, so often I'll go about other business while I wait for it to execute. But what defer does is change the meaning of the code, it takes what I know to be async, and changes it into an iterative call which is a slight of hand I'm not entirely sure is necessary. I'd further propose that it's a special case (albeit a popular one), and to change the behavior of language for a special case seems extreme.

[timbertson]

thejefflarson::

I'm convinced that a library is destined to be too noisy and awkward. The above example of map has not one but two callbacks, which one would typically write inline (they're not likely to be named functions, just steps to perform). A function call that takes two inline callable blocks is far from readable. And that still only solves a very specific case - map. Implementing filter, and if branches and everything else is all possible, but it will look hideous and will be a pain to use. I'm happy to be proved wrong, but I'm tired of people telling me I should do it in a library when I have already tried, and do not consider it acceptable. (yes, I have written something identical to the map function above. actually using it is horrible)

1. I'm not saying to use another language. I know we're all polyglots, but by the very fact that we're here, we each love javascript a whole bunch. What I am saying is that hiding the nature of javascript because other languages have a feature is a non-argument.

The fact that a feature has been implemented and shown to be immensely useful in other programming languages is no guarantee that it's a good idea, but it's hardly irrelevant either.

As an aside, I don't love javascript in the least - and if everyone loved javascript, there would be little point in coffeescript...

2. In a list comprehension coffeescript combines primitives to accomplish its task. From the docs:

(snip)

There's nothing in there that is not explicitly defined in the ECMAscript spec. Now while the ECMAscript spec does talk about callbacks (forEach for example), it does not define a true nature for callbacks. To put it a little more clearly: callbacks are a corollary to the language not a primitive.

functions are a primitive. CS uses primitives to turn a for loop into a map / filter operation. defer uses primitives to provide a continuation to a function call. We're providing abstractions over the core primitives where it makes sense to do so, in order to create more meaningful syntax - sounds pretty similar to me.

The problem of deferred code appearing to be sync is only true if you ignore the defer keyword. Trying to understand code by its syntax alone will not get you very far if you don't understand the semantics. defer makes much nicer syntax for mostly the same (but simplified) semantics. Again, you don't have to use it - or you can use it only when it makes sense to. Just like list comprehensions don't remove the ability to write a for loop, defer doesn't prevent you from managing your callbacks yourself.

[josh]

I really love the idea of defer and I'm patiently waiting for it in CS master.

My syntactic proposal is to change the assignment operator instead of the defer keyword. Haskell has something very similar regarding sequencing events with its do block. They have a special <- assignment-like operator that cleans up staircasing.

with defer:

append: (filename, str) ->
  [err, data]: defer fs.readFile filename
  err: defer fs.writeFile filename, data + 'bar'
  return;
  console.log 'done'

<- operator:

append: (filename, str) ->
  [err, data] <- fs.readFile filename
  err <- fs.writeFile filename, data + 'bar'
  return;
  console.log 'done'

translates to:

append: (filename, str, cb) ->
  fs.readFile filename, (err, data) ->
    fs.writeFile filename, data + 'bar', (err) ->
      cb()
      console.log 'done'

Whats nice is that <- is not a valid function name like defer could be. It also makes it clear this isn't a normal assignment. And it has this nice parity with the -> lambda shorthand.

(Sorry if you already discussed something like this, I'm jumping into this discussion for the first time and there's alot of backlog ;)

[jashkenas]

Plus one to Josh's suggestion.

<- used to be a different CoffeeScript operator, serving the equivalent purpose to ECMA5's Function#bind. But we've removed it from recent versions of CoffeeScript, freeing it up to be put to other uses -- and in my opinion this is an excellent use indeed.

It visually signifies the inverted call order of the deferred function, and makes it obvious that normal assignment isn't happening. In fact, we could lose the destructuring array assignment, and just do this:

(err, data) <- fs.readFile filename

It doesn't solve our return problem, but it brings things very close...

[hen-x]

Another +1 for the <- syntax. I like that it emphasizes how the assignment itself is asynchronous, and is intermediated by a callback function. Also agree that we use parameter-list-style parens (potentially with splats) instead of array brackets to capture the return value/s.

getOne: (callback) ->
    callback 'one'
getTwo: (callback) ->
    callback 'one', 'two'
getMany: (callback) ->
    callback 'one', 'two', 'three'
(x) <- getOne() # x is 'one'
(x, y) <- getTwo() # y is 'two'
(x...) <- getMany() # x is ['one', 'two', 'three']

[josh]

Had an observation about some simple CPS statements

I've seen this pattern frequently to curry on arguments to an async call.

getPeople: (callback) ->
  ajax.get "/people", callback

You could write something that accomplishes the same thing with the new syntax.

getPeople: () ->
  data <- ajax.get "/people"
  return data

# compiled
getPeople: (callback) ->
  ajax.get "/people", (data) ->
    callback data

The compiled code isn't quite the same because of the extra function wrapper. I'm not sure how you'd want to write it to take advantage of the implicit continuation argument. Maybe we could optimize away these identity functions?

More involved example

getAuthor: () ->
  post <- posts.get "1"
  person <- people.get post.author_id
  return person

# compiled
getAuthor: (callback) ->
  posts.get "1", (post) ->
    people.get post.author_id, (person) ->
      callback person


# Something similar would
getAuthor: () ->
  post <- posts.get "1"
  people.get post.author_id

# generate this ???
getAuthor: (callback) ->
  posts.get "1", (post) ->
    people.get post.author_id, callback

I really like the idea of making return work nicely, but it brings up alot of issues like this.

[timbertson]

I was a fan of <- for currying, but if it's free now then I think it fits well with defer (especially since continuation is a monad in haskell, so we can all pretend to be using haskell ;))

josh: there's certainly some amount of work that can be done towards optimisation. Personally, I'm not that interested in it (I think you should probably just use a JS optimiser), but it's certainly possible. Although without a clear separation between the AST and code generation, the optimisations are likely to make for ever-more-confusing compiler code. But yes, we can hopefully tackle the most obvious of inefficiencies at least.

[timbertson]

hmm, some problems just occurred to me with "<-". It's nice for an assignment, like so:

foo <- defer someOperation()

however when not used in an assignment, it would look pretty odd. e.g:

someFunc(<- db.get(id))

whereas

someFunc(defer db.get(id))

is still pretty readable.

And also, "<-" would cause havok with implied calls (the ones without parens). The following code:

foo <- someFunc()

would actually parse (naively) as:

foo(<-someFunc)

but foo is likely to be undefined, and not at all callable. I think the easiest way to fix this would be to restrict "<-" to not be an expression (make it only valid as a statement), but I'm pretty sure that's not what we want at all :/

or you could require the assignment still occur:

foo: <- someFunc

but that's super easy to forget, and doesn't look like haskell any more ;)

[DanielVF]

-1 for arrow.

Making a simple, often used feature (like defining a function) extremely short is wonderful. It's why I use coffeescript. But making every advanced language feature into punctation is bad.

[skybrian]

I've been watching on the sidelines (and don't even use CoffeeScript yet) so take this with a grain of salt, but based on using Python generators, I'd like to see #1 with some special syntax:

get_document: (id, *) ->
  document: defer db.get(id)
  return document

The * stands for the compiler-generated callback parameter that will be called instead of returning. (Justification: it's concise, obvious that there's an extra parameter, obvious that it's a special parameter, and you don't have to scan the function body for 'defer'. The naive user will likely realize that there's something funny going on and read the CoffeeScript manual before calling this function.)

For consistency, it seems like we should do the same thing for the special function calls that defer makes:

get_document: (id, *) ->
  document: defer db.get(id, *)
  return document

I also think 'defer' is a somewhat unfortunate name since it doesn't sound like a return statement. (It also does something entirely different in Go.) If we can't have 'yield', maybe 'pause' would get the point across?

get_document: (id, *) ->
  document: pause db.get(id, *)
  return document

'pause' implies both that the function will stop running and resume later, and I think that's the most important thing to get across about an async function call. (It also becomes obvious why pausing within a loop might not be good for performance. Since the generated code isn't that good yet, it might be better to disallow that in the initial implementation.)

Passing along an error callback then becomes straightforward if we allow the '*' to appear at any position:

get_document: (id, *, error) ->
  document: pause db.get(id, *, error)
  return document

[skybrian]

Also, on the client side, perhaps 'pause' with no argument could expand to:

 pause window.setTimeout(*, 0)

[gfodor]

I'm sadly going to have to come out against the incorporation of defer into the language. Not that I'd not want to use it, but it would legitimately decrease the main argument that I'm using to push CS in my workplace: it's javascript with some great sugar. The semantics are 1-1 and the code generation is intuitive. It's has an hour long learning curve.

Up to this point the new syntax has been both intuitive and welcome. However, if we introduce this new defer mechanism, the learning curve for any body of code using it jumps up by an order of magnitude compared to what it is now, since it's a programming construct that would be hard to digest in any language. For this reason alone I feel like it will stand out as a feature that doesn't belong with the others, and would be a inflection point in the overall barrier to entry of CS for Javascript programmers.

[timbertson]

Hi all,

So I've spent a while on this, and obviously I'd love to see it merged in to coffeescript core. But it's looking like that won't happen, at least not by me. There are a few reasons:

• differences in approach: jaskenas doesn't want it on core until it's finished, whereas maintaining a diff becomes much harder the more optimisations I make to defer (and the code around it)
• difficulty / scope. defer is a pretty significant departure from the rest of CoffeeScript, which mostly deals with localised transformation of x into y, rather than function-wide transformations of control flow. The AST is unfortunately somewhat tangled up with the compilation output; the content of a node can change depending on how it is compiled.
• forgiveness of the language. Features like implicit return and trying hard to enable more things to be statement takes its toll on the compiler, and make code transformations like defer have to deal with many more corner-cases.

So it really seems that perhaps defer isn't as good a fit for coffeescript as I had hoped. I'd love for someone to take over and prove me wrong, but I've reached the point of thinking it's not worth trying to squeeze defer into a language that has such different goals.

Although I no longer have a vested interest in seeing defer adopted (since I have no plans to write anything big in javascript any more), I'll certainly be taking a closer look at oni and stratified javascript - perhaps in the future stratified JS might be an optional compile target for coffeescript?

Anyways, if anyone wants to take over defer, please do get in touch and I'll do what I can to help out. But I don't expect I'll be doing any more work on it.

[nrstott]

That is very sad. A language that does not add new constructs to help with async programming is 'missing the point' of what a compile-to-javascript language should address imo.

When I saw coffeescript I thought, "That's kind of neat, but doesn't address the problem that I wish it did." When I saw defer, I was excited.

[gfodor]

To me, the way something like this could come into fruition would be a CoffeeScript fork that takes the project in a different direction with different goals. There's been more than a few things Jeremy has avoided doing with good reason (for example, anything involving compilation into eval), but that's not to say there isn't room for a Javascript-compiled language that breaks these rules to put more power in the hands of the compiler. Defer would fit nicely into such a project, as would other enhancements that introduce fundamentally new concepts on top of Javascript. These are large tradeoffs of course, but I see it as a worthy project once there are more than a few of these types of features that would greatly increase programmer productivity.

[jashkenas]

It's not sad at all -- defer was a valiant and long-running effort to prove that code with sync semantics can be written in a sync style, and transformed transparently into an async implementation. If it didn't make it in the end, I believe it's because you can't accomplish the transformation and preserve semantics, while at the same time generating something close to the async code you would have written in the first place.

Case in point, Stratified JS: A virtuoso performance of JavaScript compilation, but look at what it compiles into. Taking the Stratified JS version of our getDocument function above...

var getDocument = function(){ 
  waitfor(document) { 
    resume(db.get(id)); 
  } 
  return document; 
};

It compiles into this JavaScript:

var getDocument;
__oni_rt.exec(__oni_rt.Seq(0,__oni_rt.Seq(0,__oni_rt.Nblock(
function(arguments){
  getDocument=function (){
    return __oni_rt.exec(__oni_rt.Seq(1,__oni_rt.Suspend(
    function(arguments, resume){
      return __oni_rt.exec(__oni_rt.Seq(0,__oni_rt.Fcall(0,__oni_rt.Nblock(
      function(arguments){
        return resume;
      }),__oni_rt.Nblock(function(arguments){
        return db.get(id)
      })
    )),arguments,this)}, 
    function() {
      document=arguments[0];
    }),__oni_rt.Fcall(0,__oni_rt.Return,__oni_rt.Nblock(
    function(arguments){
      return document;
    })
  )),arguments, this)};
}))), this.arguments, this);

I don't think we want to take CoffeeScript down that path. Open the Pandora's box of injecting special functions into the runtime, and ... suddenly you have to worry about being orders of magnitude slower than normal JS.

So, pour one on the ground for defer. This series of tickets will continue to be a reference point.

[sveisvei]

To bad, I would have really loved if some kind of language feature like defer would help me solve/save daily problems/loc.

[hen-x]

An exact analogue of defer is slated for inclusion in _C#_5, as the await keyword.

[rameshpallikara]

I'm wondering if try/catch/finally could be used to achieve the readble callback-less sync flow attempted here - as in

try
    something_first()
catch error
    console.log error
finally
    do_something()

I'm assuming a bunch of stuff here but
Q1. Doesn't "finally" always enforce completion of the "try" block or are there any exceptions? (pardon the pun)
Q2. Any pros/cons or gotchas with this approach if it should indeed work?

any ideas?

[timbertson]

Nope, finally does not apply to asynchronous events, which is why asynchronous exception handling is so painfully difficult. This section of my post explains async programming as it relates to control flow more thoroughly: http://gfxmonk.net/2010/07/04/defer-taming-asynchronous-javascript-with-coffeescript.html#callbacks

[amirshim]

I find myself wishing CS had defer everyday... it's could be such a valuable paradigm, and simplify the way async code is written. Given that, I have a comment and question...

Jeremy: I'm not sure what the issue is with the size and/or speed of the generated code.

1. For speed, I don't think it's much of an issue, since (at least on client side), the places you would user defer would usually be where either you want to wait for IO or user interaction. Both of those are orders of magnitude slower than the speed of the generated code.
2. For size, you end up writing almost as much (albeit cleaner) code if you want to handle many of the edge cases. Plus... if someone is trying to save every last bit, then they can simply decide not to use defer.

Given that, I do understand the need to keep CS's learning curve low. Being able to learn it in one hour is one of it's (many) great strengths.

Unfortunately, I haven't had a chance to look at much of CS's internals, but was wondering if there are enough hooks to implement defer as a compiler plugin. And... whether the compiler architecture is stable enough (meaning it's not gonna change much) to make it worthwhile to create such a plugin now.

[yang]

Is there a .js that I can include in my pages to just use the latest coffeescript-fork-with-defer (whichever fork that might be)?

[sveisvei]

@yang This was not included after all, after long discussions. Coffee-script is staying close to js core features...

[yang]

Right, that's why I'm looking past coffeescript and asking about forks.

[weepy]

Check out Kaffeine

Tapped on my fone

On 21 Jul 2011, at 17:27, yang
reply@reply.github.com
wrote:

Right, that's why I'm looking past coffeescript and asking about forks.

Reply to this email directly or view it on GitHub:
#350 (comment)

[timbertson]

@yang: my fork is the most up to date that I know of for defer specifically (https://github.com/gfxmonk/coffee-script/tree/deferred), but it hasn't been touched in over a year (and I have no plans to do so), so Kaffeine (or tame, or stratified JS) is probably a better bet.

[amirshim]

You can also use coffeescript and then tamejs: https://github.com/maxtaco/tamejs
although it won't be pretty... you'll have to do something like:
twait {` ; setTimeout defer(), 100; `}

[amirshim]

The main drawback (IMHO) with using tamejs, as was one of the main reasons it didn't get into coffeescript, is debugging ability. But putting that aside, it works really well.

I created a gist with a very basic tamejs example written in coffeescript, compiled to tamejs:
https://gist.github.com/1098605

It does get pretty mangled. 3 lines of coffee script -> 7 lines in "tamejs" javascript -> 44 lines in "expanded" javascript