async/await semantics (diamod arrows as demonstration)

[Artazor]

I'm aware of #3757 but I don't think that we should speak in terms of the future ES7, let stay in the pragmatic domain. This PR is intended to start a discussion and research in this field.
I've discovered that await semantics can be achieved by a tiny utility, much more simple than that of #3757
Here it is:

    function (fn) {
      return function () {
        var gen = fn.apply(this, arguments);
        try {
          return resolved();
        } catch (e) {
          return Promise.reject(e);
        }
        function resolved(res) { return next(gen.next(res)); }
        function rejected(err) { return next(gen.throw(err)); }
        function next(ret) {
          var val = ret.value;
          if (ret.done) {
            return Promise.resolve(val);
          } else try {
            return val.then(resolved, rejected);
          } catch (_) {
            throw new Error('Expected Promise/A+');
          }
        }
      };
    }

Also small tweaks in compiler enable async'ed functions to be used as inheritable methods of CS classes, and even more - as constructors (in both cases super is available), so the inheritance and extends will not cause any issues (like they do for ordinary generators when used with co).

I've chosen a bit ridiculous syntax for async'ed functions: -<> and =<> and kept yield for resolving promises (instead of await). This is not critical and easily can be refactored into await and await return style as of @GabrielRatener (and keep -> and =>).

You can review the test (/test/generators/async).

Also I've moved all generator-related test stuff into separate directory, and removed --harmony flag sniffing (is my way of detecting generators valid enough?) - this is separate commit.

Also in the separate commit I've extended testing framework to be capable for dealing with promises.

The main question is: is the proposed semantics useful enough to be included into language in some way?

I think that generators+promises coding patterns will be very popular at the server side (especially with io.js)
and there are so tiny conservative changes needed from the language to enable brand new type of libraries/packages to emerge.

@GabrielRatener, @jashkenas, @alubbe, @michaelficarra and other guys - what are you thinking about it?

P.S. All discussed things are already implemented in this PR

[GabrielRatener]

My personal belief is that it is best for generator and async functions to be explicit, but @jashkenas seems to have leaned towards implicit generator syntax, so I try to stay consistent with that in #3757, by using await or await return to make functions async. In my PR having an await operator instead of the same yield has two advantages. First it distiguishes coroutines, and async functions syntactically, which I believe is good as they are fundamentally different. Secondly, the await operator has higher precedence than yield. Since its operand is always an object, I give it precendence over all arithmetic operators, cause arithmetic operations in JS never result in an object.

[Artazor]

@GabrielRatener no, they don't. The semantics of EcmaScript allows replacing the result of the constructor [if you are using explicit return of non-primitive type inside the constructor, then the result of the new will be overriden]. Here result of the constructor is replaced by a promise that when fullfilled will return the instance of the appropriate class.

[GabrielRatener]

My bad, that does actually seem useful.

[Artazor]

@GabrielRatener I've considered await as unary operator, but found that any new keyword will break backward compatibility, because it will be indistinguishable from the implicit call. I think there are tons of packages where await is the name of the function. I believe that if we choose the unary operator syntax, then we should use one of the existing CS keywords in the manner that is prohibited at present moment. Or even combine an identifier with keyword. I can easily imagine solutions like

asyncFn1 = ->
    result =  wait for myPromise            # awaitng for the promise
    results = all of [promise1, promise2]   # shorhand for Promise.all()
    winner = race of [promise1, promice2]   # shorthand for Promise.race()
    ...

asyncFn2 = (a, b) ->
       x = wait for someOtherAsync a
       y = wait for someOtherAsync b
      return x + y

where wait for, all of, race of are essentially compound keywords. I believe that in present syntax these constructs are illegal, that means we can add them without introducing any breaking changes, though this will introduce additional complexity into lexer/normalizer/parser.
Somebody like @jashkenas or @michaelficarra can correct me if I'm wrong here.
In this case I can't find appropriate compound keyword for modelling await return.
Any Ideas?

[vendethiel]

"await" might be a bit too widely used (after a very quick github search, it seems moslty used as part of compound names, though..), yes; but I'm not sure we want to use another name. If we add it as a keyword, await = will be disallowed anyways.

[GabrielRatener]

@Artazor The 'race' and 'all' examples are not backwards compatible because of the of keyword, Currently this is a boolean expression in CS, right?

[GabrielRatener]

What about keeping await and having a completely backwards compatible alias, maybe ~<?

[Artazor]

@GabrielRatener yes, unfortunately of leads to ambiguities.

[Artazor]

Let's brainstorm!
What about single postfix exclamation immediately after expression (without a space)?
Looks very minimalistic and allows natural chaining.
Parenless chaining is impossible with a prefix form of the await. Isn't it?

x = promise!                                                     # resolution of a single promise
y = promise!.promiseField!.promiseFunc(a,b)!.nonPromiseField     # nice chaining

And looks that it will not produce any ambiguities. Will it?

[bcherny]

@Artazor that might be confusing, since in ruby ! means the method mutates its arguments.

[GabrielRatener]

what about promise<>.chainedPromise<>?

[Artazor]

@bcherny I know about this mnemonic. I've supposed that being reminiscent of mutating modifier would be relevant for the resolution of promises in some sense. Why? Promise being resolved changes it's own state from pending to resolved. So I'd stay with postfix !. Another one-character alternative is postfix ~ which is visually hard to distinguish from minus and it has less relevant connotation (only negation).

@GabrielRatener, I appreciate your trial to keep diamonds in a language, but I'm afraid that <> as a postfix operator is much more frustrating than the original -<> and =<>. However, I think that Jeremy will approve only implicit asynchronous functions (if he'll approve them at all). Thus, I agree that we may implement await feature as an unary operator, but there are still unclear for me

A. if an await will be a keyword — how will we ensure the backward compatibility?

B. which forms of the await should be implemented (prefix, postfix, both)?

prefix operator — better for standalone calls without chaining. I don't like ~ in ~<. Any other alternatives? I'll use ~< in the examples below, but I don't like it:

x = ~< (~< (~< a).b).c    # ugly
x = ~< fn x, y, z         # nice (except the ~< itself)
~< sleep 100              # nice

postfix operator — better for chaining, but ugly for standalone calls:

x = a!.b!.c!        # nice
x = fn(x, y, z)!    # ok
x = (fn a, y, z)!   # strange
sleep(1000)!        # ok      
(sleep 1000)!       # ugly      

both?
Here I'd propose ! as postfix and (!) as prefix (to keep track of the exclamation sign - a bit haskelish)

x = a!.b!.c!         # nice
x = (!) fn x, y, z   # ok
(!) sleep 100        # ok

or kinda inconsistent

x = a!.b!.c!        # nice
x = ~< fn x, y, z   # nice but inconsistent
~< sleep 100        # the same

Any other variants, thoughts?

[GabrielRatener]

@Artazor Yes, finding a solution to backwards compatibility is not easy. I am starting to think that supporting bacwards compatibility of the await keyword would only very rarely cause issues though, based on a keyword search.

[vlad-p]

+1 for postfix !

[dev-rke]

@Artazor
I am just a user of CoffeeScript and would like to see a real keyword rather than a new operator.
CoffeeScript always tried to focus on clarity of its expressions and keywords, and i dont think it would break existing applications to use new keywords like await and async. Developers using a new software version should accept new features and things that might break in favor of clarity and usability. Using operators could improve the chainability, but how often will this feature really used?
Furthermore a single exclamation will be hard to read, it might easily beeing overlooked.
If it is hard to read and hard to understand, people won't use it.
If it is clear and understandable, people will use it, that's why i am using CoffeeScript.

I tried the IcedCoffeeScript implementation from maxtaco, which uses just the additional keywords await and defer, maybe helpful for further discussion: https://maxtaco.github.io/coffee-script/

Some ideas for the planned implementation you are talking about:

E.g. when waiting for multiple async methods to be finished:

await [
   method1: ->
   method2: ->
]
async 
  # code beeing executed after all methods have been finished

E.g. to chain methods

currentState = getIn() await order() await getBeer() await drink() async getOut()

where currentState will have the result of getOut after all methods have been chained.
Another approach would be to introduce an additional keyword like achain (abbreviation for "asynchronous chain"):

currentState = await getIn() achain order() achain getBeer() achain drink() async getOut()

This approach would propably keep the syntax clean to use cascaded structures like

currentState = await getIn() achain (await getMenu() achain order() async pay()) achain getBeer() achain drink() async getOut()

This could also be expanded to

await
  achain [
    getIn()
    order()
    getBeer()
  ]
async
  currentState = getOut()

or as alternative notation:

currentState = await
  achain [
    getIn()
    order()
    getBeer()
  ]
async
   getOut() # which implicitly returns and its return value will be applied to currentState

For a race promise currently i have no idea without an additional keyword.

There might be some things i missed out, e.g. parameters for the async callback.
But i am not here to solve this, i am just here to introduce some ideas. :-)

I would really like to see a stable async feature in CoffeeScript, awaiting it since i found IcedCoffeeScript. ;-)

What do you think about this?

[GabrielRatener]

Lets all remember that coffeescript broke backwards compatibility when introducing implicit generator syntax. Considering @jashkenas pushed for this approach over an explicit syntax as in JS, it seems backwards compatibility is a low priority for the devs.

[dev-rke]

I do not think it broke backwards compatibility. There are only a few brackets, commas and no semicolons required when writing CS code. Developers using CS are interested in writing code with less effort, using just reachable keys without wrenching fingers to type special characters. So in scope of CS everything seems to be clear, there are no underscores in variable names, no dollar signs, nor questionmarks. So introducing new keywords like await might break something (whoever calls his own variables or methods 'await'), but in prospect of usability and clarity it would be an enrichment. And developers already using an async library should be interested in getting an implementation by the language.

[Artazor]

I think that await keyword ideally suits for the mentioned role. The only thing I'm afraid of is creating serious difficulties for that guys who already created libs with the same intention, like async/await on top fibers, and those who created their systems on top of such libraries. Ok, maybe we should sacrifice them.

Other thing that I've started to think of is the performance of the proposed semantics, since @jashkenas blamed me for being "not smart" in usage of the new JS features. The point within current discussion is the fact that code inside generator functions is not optimized at all (in v8). As well it is not optimized when the function contains the try/catch block. Maybe await can be implemented on top of the pure promises and very straigtforward code transformation. There are couple of patterns (loop, branch, sequence) that allow you rewrite your async/await code with promises in that way that the resulting code will have the same visual structure and will be familiar to the original author (we can even keep the original line numbers fore every original statement/exoression). This approach is conservative (it is not the way of @maxtaco), and keeps the proposed semantics: every function that uses await will return a promise.

@jashkenas, what are you thinking of the generator-less awaits? They will work on the client-side as well without any 6to5 postprocessing...

[alubbe]

To clarify your discussion, generator support never broke backwards compatibility, "yield" was a reserved word and could not be used in the past.

[Artazor]

@alubbe, exactly! I think if await will be a keyword then CS should change it's version to 2.0 since it will be a real backward compatibility break.

[alubbe]

since CS 1.3.0

coffeescript/src/lexer.coffee

Line 584 in de511e0

'private', 'protected', 'public', 'static', 'yield'

[Artazor]

Guys, let the concrete syntax for the later stages of the discussion.

What I want to discuss here too is the possibility to implement await on top of the promises only. I know that @maxtaco's transformations were rejected by @jashkenas and @maxtaco created its own language. But there was a serious semantic improvement + runtime + generated code was surely generated (no one manually writes in the style of the javascript generated by IcedCoffee). Here I propose the style that is already used by many developers when they deal with promises without generators.

Here is an example how await operator can be desugared into Promise-based code without ES6 generators. The only assumption is that there is Promise.resolve (to create an empty promise for the loop finalization) among the globals.

Examples are written with "throw early" policy adoption: if promisified code has a synchronous preamble that throws then we prefer to throw instead of creating rejected Promise.

Note, that patterns below don't cover control flow breaks like return, break, continue and throw (for these breaks we should use a bit more complex structures).

Asynchronous sequence

function() {
  var b, e;
  a();
  b = await c;
  d();
  e = (await f[3].f()).x.y.z;
  g();
  return h;
}

when desugared into Promises becomes

function() {
  var b, e;
  a();
  return c.then(function(_ref){ 
    b = _ref;
    d();
    return f[3].f().then(function(_ref){
      e = _ref.x.y.z;
      g();
      return h;
    });
  });
}

Note, that we can reformat the code to make it line-to-line prefect:

Asynchronous selection

function() {
  var d, n;
  a()
  if (x < y) {
    b();
    d = await e();
    f();
  } else {
    m();
    n = await p();
    q();
  }
  r();
  return x;
}

when desugared to promises becomes:

function() {
  var d, n;
  a();
  function _if() { if (x < y) {
    b();
    return e().then(function(_ref){ 
      d = _ref;
      f();
    });
  } else {
    m()
    return p().then(function(_ref){ 
      n = _ref;
      q();
    });
  }
  return _if().then(function(){
    r();
    return x;
  });
}

Line-to-line:

Asynchronous repetition

function() {
  var i, c;
  a();
  i = 0;
  while (i < 10) {
    b();
    c = await d();
    e();
    i++;
  }
  f();
  return g;
}

when desugared to promises becomes

// Repetition
function() {
  var i, c;
  a();
  i = 0;
  function _while() { 
    if(i < 10) {
      b();
      return d().then(function(_ref){
        c = _ref;
        e();
        i++; 
        return _while();
      });
    } else { 
      return Promise.resolve();
    }
  } 
  return _while().then(function(){
    f();
    return g; 
  });
}

Line-to-line comparison:

@jashkenas, is it possible for CS to generate such structures? Or you don't like this?

[benzen]

I would be happy to see async await land in cs.
I didn't see how we could polyfill the Promise object in your PR, is this intentional ?

[Artazor]

@benzen, yes, it is intentional, you should polyfill the Promise by yourself, as it is already landed in io.js and soon will be a standard feature. Also you can assign an alternative implementation into global Promise (e.g. Bluebird)

[DylanPiercey]

-1 for postfix. Hard to skim for promises.

await (await fetch(...)).json()

Seems fine.

[drewhamlett]

@michaelficarra, async / await is moving right a long. Currently at Stage 3. Stage 4 ("Finished") by the end of November

http://tc39.github.io/ecmascript-asyncawait/

[brandonaaskov]

@drewhamlett thanks for the update! Can't wait 👍

[davibe]

Coffeescript should support await and compile to js code containing async/await just like it supports yield and compiles to generator*/yield. Then if you want to run it on an engine that does not support ES7 you can transpile the js code as a second stage. It would be strange if coffee compiler tried to polyfill new features.

That being said in the mean time we can do something very similar to await with things that are already supported with little syntax overhead like this https://github.com/davibe/async-js-coffee/blob/master/src/4_generators_promises.litcoffee . Then once await will be supported its going to be simple to adapt things

[GabrielRatener]

I would like to see a backwards compatible syntax introduced to CS < 2.0. Essentially an alternative to the await keyword for awaiting a promise, such as a postfix ! perhaps. This could compile to generators which are now widely supported. Then, when CS 2.0 is released the implementation could switch to using JS async functions, and the await keyword would be added to CS as well.

???

[barretron]

I like the idea of tapping into CoffeeScript's functional ancestry and going with a reflected "Monad sequencing operator" from Haskell.

text = ->
  (JSON.parse << http.get "/service.json").text

Resulting in:

var text;

text = async function() {
  return JSON.parse(await http.get("/service.json")).text;
};

The nice touch here is that assignment to a variable is also a reflection of the Haskell monad sequencing with assignment (>>=) operator, e.g.:

textA = ->
  service =<< http.get "/service.json"
  (JSON.parse service).text

[mikeyhew]

We should start compiling async/await pretty soon - at the time of writing (December 2015), it has been implemented in one browser, Microsoft Edge, and I'm guessing we'll see implementations in more of the major javascript engines by the end of 2016.

I don't think we need to do anything fancy, like compiling async and await to es6 code à la iced-coffee-script, or by automatically restructuring code to use promises as @Artazor proposed. While I like the idea, and was actually trying to figure out how to do that earlier today before I found this discussion, I think it belongs in a fork of coffeescript. Instead just compile straight to ES7's async function syntax. From a practical perspective, there's already tools out there, i believe, that will regenerate the compiled javascript code.

• I'm a bit iffy about using a new symbol for async functions instead of just async ->, because it seems like it would be confusing to read code with it. But it might actually be pretty easy to get used to, and the payoff of not having to type the extra characters every time might be worth it. I think a good candidate would be a->, since the letter a is much more meaningful than any special character could be.
• The postfix ! for await as @phaux mentioned above seems interesting, but confusing, and I don't know that there's very many use cases where you'll be chaining method calls on await expressions without saving the result to a local variable, like in the above example. Plus, an exclamation point isn't nearly as clear as the actual english word "await".
• What definitely wouldn't be a good idea, in my opinion, is implicit async functions, which are magically changed from normal functions by the existence of an await expression within the function. That just seems like a new source of confusion for anyone reading the code.

@drewhamlett thanks for the update and the link.

Food for though: since coffeescript's classes are themselves functions, is there any useful concept of async classes?

[davibe]

about your third point: We already have "implicit generators" when the function contains yield tho

[dustinrjo]

I agree with @phaux

Please just make it so that async -> (or whatever syntax for async functions we choose) compiles to async function(){}. The await keyword works fine as it is.

The only problem with the current behaviour is that async -> compiles incorrectly to async(function(){}).

Using more symbols for different kinds of functions in CS when a keyword is used in JS will just get confusing for people to learn CS and ultimately make it harder to read. For generators this was different because the * is used in JS as well, for better or worse.

Staying as close to JS as possible is really important to sell people on using CS for projects. I would only trade words for symbols if any bum off the street could look at the change and say "now that is super helpful and I can remember that". Otherwise we are going backwards. There is a reason || became or and && became and and async should be async. Let's not lose that.

[mpseidel]

+1 for any solution that allows nice async control flow with coffee. I would love to use coffee for all my tests insteaf of typescript but I need a clean way to await stuff for the tests to stay clean and easy to read.

[taoeffect]

+1 for ! postfix operator. It fits nicely with CS's existing succinctness and has precedent with the succinctifying nature of the ? postfix operator.

[AJeezy9]

+1-ing in hopes that CS can somehow be resuscitated to implement new ES6/7 features.

Increasingly prevalent use of async/await & decorators are starting to force my hand to make the switch, but I much prefer CS.

ES6 does more to obfuscate than improve javascript, and its grammatical conventions remain far more tedious than Coffeescript.

[DylanPiercey]

@AJeezy9 totally agree, was sad to move away from CoffeeScript. Personally I would love something like "babel-lite" where a huge amount of features are forcibly removed. The only thing that I always really miss from CS is the "one-page" docs (much less syntax than js) and the "everything is an expression" which works so well with JSX and functional programming.

@Artazor regarding your "promise unwraping" proposal It has been partially implemented here (in babel) and works pretty well, i'd definitely +1 that.

[phaux]

@dustinrjo

Staying as close to JS as possible is really important to sell people on using CS for projects.

I don't necessarily agree on that. I'd prefer Coffee to stay one step ahead of JS and make things prettier and more concise. ^-> and *-> look cool to me. async -> might be confused with a function taking single async argument.

@DylanPiercey

-1 for postfix. Hard to skim for promises.

Then how about using a different sigil? (~^ as an example)

download fetch('/gallery.html')~^.text()~^.match(/<img src="(.*?)"/)[1]

[kirly-af]

Concerning the existence of a postfix syntax in general, it's not that I'm against the idea but I don't feel the need for this. Though, I didn't find any of the suggested operators appealing.

I personally feel ! as a terrible operator choice because of its usual meaning in most programming languages (ie: not). Considering await is a new feature and potentially not widely used for a while, it would bring even more confusion for people discovering ES* features or to CoffeeScript new comers (not even mentioning people in both cases). It might look to be a detail but I'm sure I'm not the only one who faced this kind of situation.

Besides, to me all the pseudo-arrow usage result in ugly syntax. I would hate to write something like: fetch('/gallery.html')~^.text() (very ugly) or whatever similar to fetch('/gallery.html')*->.text() (quite ugly and too close to classic arrows).

I find way more elegant the idea of an operator such as <- or << (suggested by @barretron), but in the end I think the simple use of the await keyword is completely fine.

I'm new to this discussion, and I'm a bit afraid by the time it's taking to add a feature that is said to be ready (forgive me if I missed a post saying the opposite, the thread got quite long now, so please be indulgent if it is the case).

I love CoffeeScript for its elegant syntax like most people on this thread, but what I primarily like a language for is the features it provides. Wouldn't it be more logical to add the feature with the simple await keyword so we can start playing with it, while keep thinking about syntactic sugar that could be added later on ? Many people share this point of view but after more than a year it looks like we're still not getting anywhere. Apart from that, has @jashkenas ever shown any interest for this feature ?

[DomVinyard]

@mikeyhew said:

What definitely wouldn't be a good idea, in my opinion, is implicit async functions, which are magically changed from normal functions by the existence of an await expression within the function. That just seems like a new source of confusion for anyone reading the code.

I disagree, where would the confusion arise? Implicit async definitely seems like the way to go - if it's a function, it's a function. If you see 'await' then you know that line of code is 'awaiting'... It seems like the only elegant solution proposed.

[GabrielRatener]

@DomVinyard having both implicit generator functions with yield, and implicit async functions is challenging. For generators without yields, there are yield return statements, for implicit async functions there may be an await return analog. When async generator functions are implemented, what would be its analog?

Even Python 3.5 went with explicit syntax for async functions.

[mikeyhew]

@DomVinyard I've mostly changed my mind since I made that comment. At this point, I don't think it would be the end of the world if a function returns a promise because it contains an await expression. Sure, it might make things a bit more confusing at first, but it would be better to be consistent with generators, and there's something to be said for using the same -> for all types of functions and not having to write async -> or invent another syntax for async functions. And the improvement in readability that arises when you switch from explicit promises to using await really trumps any regressions that this could cause.

[DomVinyard]

@GabrielRatener No mixing yield with async. Compile time error. For some edge use cases it's problematic and would require coding around, granted. But, ultimately, async/await is too powerful to not bake into 'default coffeescript', it's too natural an evolutionary language feature.

[mikeyhew]

@GabrielRatener said:

Even Python 3.5 went with explicit syntax for async functions.

yes, python 3.5 uses async def for coroutines but just def for generators. But why is that? I thought it was just because they added coroutines later on.

[DomVinyard]

@mikeyhew And it would reduce the friction for people just starting out with async, they can throw an await call into an existing production function without the mental overhead of what some would consider "refactoring the whole function". @jashkenas Count me +1

[GabrielRatener]

@mikeyhew The reasons for adding explicit syntax in Python are described here. I think both explicit and implicit syntax would be better than nothing, but explicit syntax really indicates that the functions are fundamentally different.

@DomVinyard Async generators will be in JS eventually, so shouldn't accommodations be made for the future syntax? Also, implicit syntax can cause friction and confusion for people who understand that JS is single threaded. They see rogue await statements that magically appear to make functions block. Both syntaxes can be confusing to different people.

[maxtaco]

FWIW, @zapu and I recently rewrote IcedCoffeeScript to use ES6 generators. The compiler and transpiled code are significantly simpler, but the semantics remain the same. Code is available in our iced3 branch or via npm at iced-coffee-script-3.

[DomVinyard]

@GabrielRatener

They see rogue await statements that magically appear to make functions block. Both syntaxes can be confusing to different people.

I totally agree, I'm making my case based on the idea that coffeescript is appealing to the very people who prefer the magic over the explicit, fans of abstraction. Although I do take your point that it would be alienating to some, I think it's absolutely worth the conceit.

In the same way that - if an explicit form won out - I think there are a group of people that would use the async function form by default 'just in case'. Like many do with the fat arrow.

[kirly-af]

@DomVinyard

if an explicit form won out - I think there are a group of people that would use the async function form by default 'just in case'. Like many do with the fat arrow.

Such cases are easily handled with a linter. I don't see this point as a valid reason.

I have a preference for the implicit syntax (without the async keyword). As many said, it keeps the logic of implicit generators. However I have no idea if it could be a problem to later bring async generators to coffeescript as some were discussing.

[DomVinyard]

@mikeyhew

The link to the Python discussion was interesting, thanks for posting up

While it is possible to just implement await expression and treat all functions with at least one await as coroutines, this approach makes APIs design, code refactoring and its long time support harder. Let's pretend that Python only has await keyword: If a function is refactored and someone removes all await expressions from it, it would become a regular python function, and all code that depends on it, [...] would be broken. To mitigate this issue a decorator similar to @asyncio.coroutine has to be introduced.

Am I right in saying that this limitation doesn't apply to ES7 because

var three = await 3;
console.log(three === 3) // true

[GeoffreyBooth]

Closed in favor of #3757.