Extending async state machine API

[YairHalberstadt]

Introduction

C# already has support for working with monads using linq query expressions. However the syntax is relatively longwinded and only works with a subset of the language. It's also tends to very allocation heavy.

For asynchronous workflows, the language team decided that wasn't good enough. They introduced async await, which allows you to lift values from the Task monad and work with them almost as if this was normal code. The majority of C# language features are available within an async method, and it's compiled down to a very efficient, low allocation state machine.

This proved such a popular model it's spread way beyond C#, to javascript, rust, kotlin, scala and beyond.

What about doing exactly the same thing for other monads?

Task like Types

The C# language already allows you to define your own types that can be awaited or returned from an async method. See https://github.com/dotnet/roslyn/blob/master/docs/features/task-types.md for more details.

This can be hacked to allow using async await for types which have nothing to do with Tasks.

Awaitables Experiment

I've created an experimental library, Awaitables to explore how well we can work with some common monads using the existing state machinery. So far I've implemented the following awaitable monads:

• Option<T>
• Result<T>
• AwaitableEnumerable<T>

Please feel free to play with them, and tell me what you think!

Option bails out of the method and returns None if it ever awaits a None option.
Result bails out of the method, and returns Failure if it ever awaits a failed Result. It also returns Failure if anywhere in the method throws.
AwaitableEnumerable runs the rest of the method for every single item in any collection you await. Essentially it acts like every await is a SelectMany, but allows you to write imperative style code, as well as to use loops etc.

It's actually very impressive how well this works. The async await model is very intuitive to work with, and the implementations are allocation free for Option and Result and only minimum allocations for AwaitableEnumerable. The implementations are actually quite simple to follow and understand. This experiment has convinced me that there's huge potential in applying async await to other domains, but there are currently a few easy to fix critical limitations that prevent it from being at all usable. I hope to highlight in this issue some of the extensions to the API that would be necessary to make the Awaitables library production ready.

Examples

https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Option.Example/Program.cs
https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Result.Example/Program.cs
https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Enumerable.Example/Program.cs

Option

using System;
using System.Collections.Generic;

namespace Awaitables.Example
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var program = GetSelectedProgram();
            if (program.HasValue)
            {
                program.Value();
            }
            else
            {
                Console.WriteLine("Invalid argument");
            }
            async Option<Action> GetSelectedProgram()
            {
                var arg = await args.ElementAt(0);
                var asInt = await arg.TryParseInt();
                return await _programs.TryGetValue(asInt);
            }
        }

        public static Dictionary<int, Action> _programs = GeneratePrograms();

        private static Dictionary<int, Action> GeneratePrograms() => new Dictionary<int, Action>
        {
            { 1, () => Console.WriteLine("Play chess") },
            { 2, () => Console.WriteLine("Solve World Hunger") },
            { 3, () => Console.WriteLine("Win a noble prize") },
        };
    }

    public static class Extensions
    {
        public static Option<T> ElementAt<T>(this IReadOnlyList<T> list, int index)
            => list.Count > index ? Option.Some(list[index]) : Option.None<T>();
        public static Option<TValue> TryGetValue<TKey, TValue>(this IReadOnlyDictionary<TKey, TValue> dictionary, TKey key)
            => dictionary.TryGetValue(key, out var value) ? Option.Some(value) : Option.None<TValue>();
        public static Option<int> TryParseInt(this string str)
            => int.TryParse(str, out var value) ? Option.Some(value) : Option.None<int>();
    }
}

Result

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace Awaitables.Example
{
    class Program
    {
        static void Main(string[] args)
        {
            var result = UpdateDbWithRetry();

            if (result.IsSuccessful)
            {
                Console.WriteLine($"Wrote {result.Value} items to db");
            }
            else
            {
                Console.WriteLine(result.Exception);
            }

            Result<int> UpdateDbWithRetry()
            {
                return UpdateDb() switch
                {
                    { IsSuccessful: true } result => result,
                    { Exception: TimeoutException _ } => UpdateDbWithRetry(),
                    var result => result,
                };
            }

            async Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = await Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return await Db.Write(connectionString, squares);
            }
        }
    }

    public static class Db
    {
        private static readonly Random _random = new Random();
        public static async Result<IEnumerable<TResult>> Query<TResult>(string connectionString, string query) => _random.Next() % 2 == 0
            ? throw new TimeoutException("could not connect")
            : Enumerable.Range(0, _random.Next() % 10).Cast<TResult>();

        public static async Result<int> Write<T>(string connectionString, IEnumerable<T> data) => data.Count() > 5
            ? throw new InvalidDataException("Too many items!!!")
            : data.Count();
    }
}

Enumerable

using System;

namespace Awaitables.Example
{
    class Program
    {
        static void Main(string[] args)
        {
            var exception = new AggregateException(
                new AggregateException(
                    new Exception("Bad stuff")), 
                new InvalidCastException("Bad cast!!"), 
                new AggregateException(
                    new InvalidOperationException("Bad Operation"), 
                    new AggregateException(
                        new ArgumentNullException("name"), 
                        new NullReferenceException("another null"))));

            FlattenAndLog(exception);
        }

        static async AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            // Note that this is far more efficient than the naive recursive solution,
            // where you Flatten each nested exception recursively, loop over the
            // returned results, and yield them.
            // It also will not blow up the stack on very deeply nested exceptions.
            while(exception is AggregateException aggregateException)
            {
                exception = await aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
    }
}

Suggestions

I've divided these suggestions into two categories

1. extensions to the existing API.
2. how would we design a new API that was not focused primarily on Tasks.

Suggested extensions

1. ability to force finally blocks to run.

The single most critical limitation that Result<T> and Option<T> share is that finally blocks do not run if they exit early. Option exits early if it awaits a None option, and Result if it awaits a Failure result. This is the only critical flaw that would prevent me using these in production.

The current async state machinery is designed so that if MoveNext exits after calling AwaitOnCompleted finally blocks are not run, to ensure that the finally block is only ever run once.

Since Option and Result never call the continuation in AwaitOnCompleted and just exit, this means the finally block will never be run. Some Api needs to be provided to indicate that this has occured and run the blocks.

I would suggest the following:

If the AsyncMethodBuilder has the following member:

public bool Completed { get; set; }

(possibly implementing some magic interface to prevent changing behavior of existing code),
then the finally blocks will check if Completed is true, and if so run the finally block even if AwaitOnCompleted has been called. This would be a zero cost abstraction.

2. specify that certain types can only be awaited in methods returning specific types and vice versa

Currently the following two methods compile fine:

public async Task<int> M1() => await Option.Some(42);
public async Option<int> M2() => await Task.FromResult(42);

Both obviously fail disastrously at runtime.

An analyzer could be used to prevent this, but it feels like this deserves something inbuilt.

In order for Result<T> to be able to access certain properties from the Awaiter, without allocating, we add the IHasException constraint to AwaitOnCompleted:

        public void AwaitOnCompleted<TAwaiter, TStateMachine>(
            ref TAwaiter awaiter, ref TStateMachine stateMachine)
            where TAwaiter : INotifyCompletion, IHasException
            where TStateMachine : IAsyncStateMachine
        {
            Task = Result.Failure<T>(awaiter.Exception);
        }

The compiler errors if this method is called with an invalid awaiter.

We could make this more formal. An AsyncMethodBuilder can have a [ValidAwaiterAttribute(System.Type)] attribute applied. Only an awaiter assignable to the type used in the attribute can be awaited in a method using this MethodBuilder. AwaitOnCompleted and AwaitUnsafeOnCompleted can constrain the awaiter to this type.

Meanwhile we add a [ValidAsyncMethodBuilder(System.Type)] attribute which can be applied to an Awaiter. An Awaiter with one or more of these attributes can only be awaited in an method using one of these builders.

3. Allow AsyncMethodBuilder to have multiple generic parameters

Currently an AsyncMethodBuilder cannot have multiple type parameters. This makes it impossible to use Result<TError, TResult> as the return type of an async method. There are many other common monads that also require this - Either, Reader and State for example.

We should allow AsyncMethodBuilders to have the same number of type parameters as the awaitable. SetResult(T) must use the same type parameter as GetAwaiter().GetResult() returns.

4. Allow extension GetAwaiter and AsyncMethodBuilders

I had to define an AwaitableEnumerable type because you can't define AsyncMethodBuilder for a type if you don't own the type.

If the BCL ever defines their own Option or Result types, it is unlikely they would make it awaitable. If we want to avoid ecosystem splits, it would be useful if you could define AsyncMethodBuilder without owning the type.

We could allow using an [AsyncMethodbuilderOf(System.Type)] attribute to specify a method builder for a type you don't own.

5. Add Clone method to IAsyncStateMachine

For AwaitableEnumerable the implementation requires cloning the async state machine. In release, the state machine is a struct and copying is trivial, but in debug it's a class, and I had to use reflection to call MemberwiseClone. See https://github.com/YairHalberstadt/awaitables/blob/master/Awaitable.Common/StateMachineCopier.cs.

This would be less hacky if IAsyncStateMachine defined a clone method, that just called straight into MemberwiseClone when the state machine is a class, or returned this when it's a struct.

6. Add way to prevent awaiting in try block.

There is no sensible way to work with try/catch/finally/using blocks with AwaitableEnumerable. Sometimes you'll want the finally blocks to run exactly once, sometimes for every item that was in the enumerable that was awaited, and sometimes there's no valid semantics.

The only option is to ban awaiting in a try block. This could be done via analyzers, but it would be safer if this was a language feature.

Designing a new API that is not focused primarily on Tasks

The current API is focused very much on asynchronous programming. Whilst you get used to awaiting an option, it hardly makes any sense.

Perhaps it might be worth creating a new API that reused much of the existing compiler machinery, but used more generic names, both for the operators (Haskell's <- perhaps?) and for the AsyncMethodBuilder and GetAwaiter APIs?

EDIT

After discussion with @CyrusNajmabadi on gitter, he felt that whilst this idea is interesting the chief thing holding it back is syntax. The C# team will not be keen on encouraging people to await things that are nothing to do with asynchronous programming.

Based on that I've been working on ideas for new syntax. The actual code for generating the state machine will mostly be reused from await, plus the additions requested above. The new required APIs for the equivalent of AsyncMethodBuilder and GetAwaiter will presumably be given new more generic names. However since only the most advanced users will interact directly with these APIs this is less critical.

There are two options for the syntax:

1. Use an extremely generic operator that is suitable everywhere.
2. Allow the api to define it's own operator.

Use an extremely generic operator that is suitable everywhere.

Haskell uses a left arrow <- in place of await. This fits all the examples we have so far as being indicative of "extracting a possible value or values out of a monadic context". Whilst there are monads which aren't like this, they tend to be much rarer.

<- currently has a meaning in C#. eg if(value<-10) Console.WriteLine("Less than minus 10");

There are two options here:

a) Use a modifier on the method signature (like async) to tell the parser to parse the method differently, and treat <- as the bind operator. Less than minus will have to formatted < - as it usually is anyway.
b) Find a different flavor of arrow that isn't ambiguous. <-- is, as is <~ and <=. <--- and <== are not.

Allow the api to define it's own operator.

We could allow the AsyncMethodBuilder, or GetAwaiter, to define the operator used in place of await via an attribute: e.g. [BindOperator(String)]

That way the user could define an appropriate operator for each monad. E.g.

Option: value
Result: try
Enumerable: element

In order to make parsing simple, we would have to use make sure that the operator was used in a way which is currently illegal.

For example, currently an expression can't be followed by !! or preceded by a ., so we could require the operator to be used in such a fashion: try!! or .try.

Examples with new syntax.

In order to give a taste of how these suggested syntaxes may look, I've copied over sections of the examples given above with some of my suggestions. When a method signature modifier was required I've used the strawman modifier controlled.

Option

            controlled Option<Action> GetSelectedProgram()
            {
                var arg = <- args.ElementAt(0);
                var asInt = <- arg.TryParseInt();
                return <- _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = <--- args.ElementAt(0);
                var asInt = <--- arg.TryParseInt();
                return <--- _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = value!! args.ElementAt(0);
                var asInt = value!! arg.TryParseInt();
                return value!! _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = .value args.ElementAt(0);
                var asInt = .value arg.TryParseInt();
                return .value _programs.TryGetValue(asInt);
            }

Result

            controlled Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = <- Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return <- Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = <--- Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return <--- Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = try!! Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return try!! Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = .try Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return .try Db.Write(connectionString, squares);
            }

Enumerable

        static controlled AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = <- aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = <--- aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = element!! aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = .element aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }

Please comment on which syntax you prefer, and suggest new ones if you have.

My personal opinion is leaning towards a generic syntax based on the arrow operator. I think that makes discoverability much easier, as you don't need to try and work out what the correct operator should be. It also is easier on the tooling. Also it really does work very well for almost all monads, and is consistent with other languages such as haskell and scala.

I'm not sure whether to prefer a modifier or not. On the one hand I prefer <- to <---, and adding a modifier allows you to lower the method to a state machine without awaiting inside the method (this is useful for Result as doing so will catch any exceptions in a method and convert them to a Result). On the other hand I'm struggling to come up with a good name for the modifier. If you have any ideas, please suggest them!

[orthoxerox]

• A blog post that explores the same problem space: https://habr.com/ru/post/458692/
• Is it worth pursuing improvements to AwaitableEnumerable at all? We already have a nice imperative method of working with them that is arguably more flexible

[YairHalberstadt]

@orthoxerox
I read that blog post but I'm afraid I have no idea why his implementation is so complicated. Mine is much simpler (compare Mine to His) and is allocation free, whereas his is very heavy on allocations.

[YairHalberstadt]

Is it worth pursuing improvements to AwaitableEnumerable at all

Agreed, but I think it's indicative of the sort of problems we might face when working with other Monads. I would love for somebody to contribute more examples to my repo so we can get a clearer picture of what might be required!

[Joe4evr]

(TL;DR: This implementation is far too magical.)

So I've been very on the fence about all this monad talk and what-not, but I think I just realized exactly why I just don't like introducing "high-level magic" syntactic sugar around these concepts: The introduction of implicit control flow.

Just taking the first of these:

Option bails out of the method and returns None if it ever awaits a None option.

So writing this

var value = await option;

gets lowered to something alike

if (!option.HasValue)
    return Option.None;
var value = option.Value;

This is completely opaque to the reader that control can leave the method there based only on what is actually written, without knowing in advance that such is its behavior.

This introduces a conceptual bar that I feel is much too high for C# to take on. I believe that one of the main reasons C# is easy to pick up is because control flow switches are explicit, so reading the flow is always clear regardless of how much experience you have.

The reason the await pattern works well with Tasks is that control flow always comes back to the call-site, and it completely erased the conceptual overhead associated with Callback Hell that asynchronous code had to be before then.

If the LDT wants C# to remain an accessible language to novice coders, then I strongly suspect this isn't the way forward.

I can see the framework adding a set of blessed types for Option/Result/etc. at some point so that everyone can interop with those patterns, and maybe there's room for some syntax sugar to work with them nicely, but this implementation loses too much clarity for its purpose IMO.

[orthoxerox]

Pinging @louthy for his opinion on the syntax vs LINQ.

[HaloFour]

In languages like Scala and Haskell such magic is usually relegated to specific language features like "for comprehensions" and "do notation" respectively. These function more like LINQ query syntax except for more than collections. It's neat that the behavior can be emulated with await but I don't think that expanding await is the right place to implement such features.

[orthoxerox]

Idris has a terse generic syntax similar to this: http://docs.idris-lang.org/en/latest/tutorial/interfaces.html#notation

For Scala there's Effectful which is similarly terse.

[YairHalberstadt]

The Result<T> type acts almost identically to rusts Result type, and await is equivalent to their ? operator.

It also has the advantage of allowing you to return T directly, instead of having to Ok wrap as you have to in Rust.

It's also very similar to Midori's error model, which forced you to prepend try when calling any method that could throw. Here you prepend await instead.

Given how popular both these error models are, I think the ability to do the same in C# would be a huge boon.

[louthy]

Thanks for the ping @orthoxerox

Generally, I'm in favour of this idea. I have done similar experiments myself for language-ext, but never brought them into the library because of the confusion around the keywords async and await, and so some syntactic improvements here would make this into a very viable solution.

why I just don't like introducing "high-level magic" syntactic sugar around these concepts

It's far too easy to dismiss an idea you don't like by calling it magic. Many languages have monadic control-flow and do just fine. It already exists with the async/await and LINQ functionality, but in the case of LINQ it's expensive due to the lambda nesting and, with async/await, the general feature has grammar that's too specific.

As a slight aside: There could be an argument that the compiler could do some deeper analysis of the Select, SelectMany, Where methods and unwrap the methods into a state-machine like async/await, which would give some benefits too. Not sure of the viability of that though, and I assume there's little appetite from the Roslyn devs, as LINQ doesn't seem to be on the radar any more.

I like the idea of using <- like Haskell or PureScript, this feels elegant to me:

    Option<int> Add(Option<int> mx, Option<int> my)
    {
        var x <- mx;
        var y <- my;
        return x + y;
    }

What's nice about a system like this is that it allows a more imperative style for those that prefer it. LINQ is great, but it comes with quite a bit of baggage for some users, and many struggle to see how to build expression oriented code. And so, this would really enable monadic programming without it being cognitively too much of a stretch. It can massively declutter code and properly declare the intent - which is exactly what is seen with async / await

[YairHalberstadt]

Edited the original post to added the following:

Examples

https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Option.Example/Program.cs
https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Result.Example/Program.cs
https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Enumerable.Example/Program.cs

Option

using System;
using System.Collections.Generic;

namespace Awaitables.Example
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var program = GetSelectedProgram();
            if (program.HasValue)
            {
                program.Value();
            }
            else
            {
                Console.WriteLine("Invalid argument");
            }
            async Option<Action> GetSelectedProgram()
            {
                var arg = await args.ElementAt(0);
                var asInt = await arg.TryParseInt();
                return await _programs.TryGetValue(asInt);
            }
        }

        public static Dictionary<int, Action> _programs = GeneratePrograms();

        private static Dictionary<int, Action> GeneratePrograms() => new Dictionary<int, Action>
        {
            { 1, () => Console.WriteLine("Play chess") },
            { 2, () => Console.WriteLine("Solve World Hunger") },
            { 3, () => Console.WriteLine("Win a noble prize") },
        };
    }

    public static class Extensions
    {
        public static Option<T> ElementAt<T>(this IReadOnlyList<T> list, int index)
            => list.Count > index ? Option.Some(list[index]) : Option.None<T>();
        public static Option<TValue> TryGetValue<TKey, TValue>(this IReadOnlyDictionary<TKey, TValue> dictionary, TKey key)
            => dictionary.TryGetValue(key, out var value) ? Option.Some(value) : Option.None<TValue>();
        public static Option<int> TryParseInt(this string str)
            => int.TryParse(str, out var value) ? Option.Some(value) : Option.None<int>();
    }
}

Result

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace Awaitables.Example
{
    class Program
    {
        static void Main(string[] args)
        {
            var result = UpdateDbWithRetry();

            if (result.IsSuccessful)
            {
                Console.WriteLine($"Wrote {result.Value} items to db");
            }
            else
            {
                Console.WriteLine(result.Exception);
            }

            Result<int> UpdateDbWithRetry()
            {
                return UpdateDb() switch
                {
                    { IsSuccessful: true } result => result,
                    { Exception: TimeoutException _ } => UpdateDbWithRetry(),
                    var result => result,
                };
            }

            async Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = await Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return await Db.Write(connectionString, squares);
            }
        }
    }

    public static class Db
    {
        private static readonly Random _random = new Random();
        public static async Result<IEnumerable<TResult>> Query<TResult>(string connectionString, string query) => _random.Next() % 2 == 0
            ? throw new TimeoutException("could not connect")
            : Enumerable.Range(0, _random.Next() % 10).Cast<TResult>();

        public static async Result<int> Write<T>(string connectionString, IEnumerable<T> data) => data.Count() > 5
            ? throw new InvalidDataException("Too many items!!!")
            : data.Count();
    }
}

Enumerable

using System;

namespace Awaitables.Example
{
    class Program
    {
        static void Main(string[] args)
        {
            var exception = new AggregateException(
                new AggregateException(
                    new Exception("Bad stuff")), 
                new InvalidCastException("Bad cast!!"), 
                new AggregateException(
                    new InvalidOperationException("Bad Operation"), 
                    new AggregateException(
                        new ArgumentNullException("name"), 
                        new NullReferenceException("another null"))));

            FlattenAndLog(exception);
        }

        static async AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            // Note that this is far more efficient than the naive recursive solution,
            // where you Flatten each nested exception recursively, loop over the
            // returned results, and yield them.
            // It also will not blow up the stack on very deeply nested exceptions.
            while(exception is AggregateException aggregateException)
            {
                exception = await aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
    }
}

...

After discussion with @CyrusNajmabadi on gitter, he felt that whilst this idea is interesting the chief thing holding it back is syntax. The C# team will not be keen on encouraging people to await things that are nothing to do with asynchronous programming.

Based on that I've been working on ideas for new syntax. The actual code for generating the state machine will mostly be reused from await, plus the additions requested above. The new required APIs for the equivalent of AsyncMethodBuilder and GetAwaiter will presumably be given new more generic names. However since only the most advanced users will interact directly with these APIs this is less critical.

There are two options for the syntax:

1. Use an extremely generic operator that is suitable everywhere.
2. Allow the api to define it's own operator.

Use an extremely generic operator that is suitable everywhere.

Haskell uses a left arrow <- in place of await. This fits all the examples we have so far as being indicative of "extracting a possible value or values out of a monadic context". Whilst there are monads which aren't like this, they tend to be much rarer.

<- currently has a meaning in C#. eg if(value<-10) Console.WriteLine("Less than minus 10");

There are two options here:

a) Use a modifier on the method signature (like async) to tell the parser to parse the method differently, and treat <- as the bind operator. Less than minus will have to formatted < - as it usually is anyway.
b) Find a different flavor of arrow that isn't ambiguous. <-- is, as is <~ and <=. <--- and <== are not.

Allow the api to define it's own operator.

We could allow the AsyncMethodBuilder, or GetAwaiter, to define the operator used in place of await via an attribute: e.g. [BindOperator(String)]

That way the user could define an appropriate operator for each monad. E.g.

Option: value
Result: try
Enumerable: element

In order to make parsing simple, we would have to use make sure that the operator was used in a way which is currently illegal.

For example, currently an expression can't be followed by !! or preceded by a ., so we could require the operator to be used in such a fashion: try!! or .try.

Examples with new syntax.

In order to give a taste of how these suggested syntaxes may look, I've copied over sections of the examples given above with some of my suggestions. When a method signature modifier was required I've used the strawman modifier controlled.

Option

            controlled Option<Action> GetSelectedProgram()
            {
                var arg = <- args.ElementAt(0);
                var asInt = <- arg.TryParseInt();
                return <- _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = <--- args.ElementAt(0);
                var asInt = <--- arg.TryParseInt();
                return <--- _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = value!! args.ElementAt(0);
                var asInt = value!! arg.TryParseInt();
                return value!! _programs.TryGetValue(asInt);
            }
            Option<Action> GetSelectedProgram()
            {
                var arg = .value args.ElementAt(0);
                var asInt = .value arg.TryParseInt();
                return .value _programs.TryGetValue(asInt);
            }

Result

            controlled Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = <- Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return <- Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = <--- Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return <--- Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = try!! Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return try!! Db.Write(connectionString, squares);
            }
            Result<int> UpdateDb()
            {
                var connectionString = "MyConnectionString";
                var integers = .try Db.Query<int>(connectionString, "Select * from Integers");
                var squares = integers.Select(x => x ^ 2);
                return .try Db.Write(connectionString, squares);
            }

Enumerable

        static controlled AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = <- aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = <--- aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = element!! aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }
        static AwaitableEnumerable<Exception> FlattenAndLog(Exception exception)
        {
            while(exception is AggregateException aggregateException)
            {
                exception = .element aggregateException.InnerExceptions.ToAwaitable();
            }
            Console.WriteLine(exception.Message);
            return exception;
        }

Please comment on which syntax you prefer, and suggest new ones if you have.

My personal opinion is leaning towards a generic syntax based on the arrow operator. I think that makes discoverability much easier, as you don't need to try and work out what the correct operator should be. It also is easier on the tooling. Also it really does work very well for almost all monads, and is consistent with other languages such as haskell and scala.

I'm not sure whether to prefer a modifier or not. On the one hand I prefer <- to <---, and adding a modifier allows you to lower the method to a state machine without awaiting inside the method (this is useful for Result as doing so will catch any exceptions in a method and convert them to a Result). On the other hand I'm struggling to come up with a good name for the modifier. If you have any ideas, please suggest them!

[louthy]

<= is less than or equal, so that can't be used if we're trying to maintain backward compatibility (which we are)

<~ feels pretty good to me. ~> is something used in PureScript for natural transformations and Signals, it's used in Haskell for Pipes and many other libraries which represent a value in a context; so the idea that a value is travelling has precedence.

The problem I've found with ~> is typing the ~ character, it's less commonly used, and therefore a touch more awkward to remember where it is on the keyboard. It's a minor thing.

The longer arrows <-- and <== feel too weighty. Hard to describe, but it starts to make the code feel a bit ugly to me.

One thought might be to repurpose the in operator from LINQ. The meaning behaviour would be the same for both LINQ and this state-machine approach. It also removes the problem of introducing a new operator, as it already exists, just it will work in a new context.

Option<Action> GetSelectedProgram()
{
    var arg in args.ElementAt(0);
    var num in arg.TryParseInt();
    var res in programs.TryGetValue(num);
    return res;
}

Obviously, without the proper syntax highlighting, this might not immediately look right. But, if the keyword was red like the return then it could start to look quite pretty. It also reads well:

I want the value in the expression

It should look nicer in a context where there isn't a left-hand-side: i.e.

    Option<int> Foo(Option<int> value)
    {
        return Bar(in value);
    }

Rather than:

    Option<int> Foo(Option<int> value)
    {
        return Bar(<- value);
    }

[YairHalberstadt]

I thought about in

It would lead to the slightly odd:

foreach(var a in in List<List<int>>)

Of course if we use a new method signature modifier we don't need to worry as much about backwards compatibility, since we can parse differently based on the modifier (as C# already does for await). Changing a commonly used operator like <= seems like a bad idea though anyway. <- is much rarer.

[louthy]

I updated my previous comment to include an example where an operator approach might look wrong:

    Option<int> Foo(Option<int> value)
    {
        return Bar(<- value);
    }

That feels inelegant to me. The arrow signifies a direction, which makes sense in the do notation of Haskell, because there always has to be a value on the left-hand-side of the <-; we know where the value is going with a <- ma and where it's coming from. But, when there's nothing on the left, it feels incomplete.

I certainly have much more async code that looks like this:

    async Task<int> Foo(Task<int> value)
    {
        return Bar(await value);
    }

Than I do nested foreach, so I think it probably still feels right for in. Perhaps a requirement for parens:

    foreach(var a in (in listOfLists))

So, this feels good:

    Option<int> Foo(Option<int> value)
    {
        return Bar(in value);
    }

These are just preferences, I am happy with any syntax here. It's the feature that's super, super valuable. It would give C# super powers over many other languages - and would allow for incredibly declarative code, that's all I'd like to see :)

[louthy]

Of course if we use a new method signature modifier we don't need to worry as much about backwards compatibility

I suspect this will be something that the csharplang team worry about, maybe less so from the PoV that a method modifier could change the parser, but more so from the PoV that they'd probably want to use the same code/parser as the async/await feature.

[orthoxerox]

This snippet can be rewritten with fewer temp variables (and that final await is superfluous):

var arg = await args.ElementAt(0);
var asInt = await arg.TryParseInt();
return _programs.TryGetValue(asInt);

return _programs.TryGetValue(await (await args.ElementAt(0)).TryParseInt());

However, now it looks really busy and hard to follow. I think postfix notation makes it flow better (I've deliberately used ‽ to indicate it's not a concrete suggestion:

return _programs.TryGetValue(args.ElementAt(0)‽.TryParseInt()‽);

It could always be made styleable separately from other operators so you could make it black-on-yellow if you wanted it to stand out more.

[orthoxerox]

A few more notes:

1. Should the methods in https://github.com/YairHalberstadt/awaitables/blob/master/Awaitables.Result.Example/Program.cs throw? I think they should return an exception, unless your implementation has grown more magical.

2. The DB methods made me think that a lot of methods can also be actually async as well:

public static async Task<Result<IEnumerable<TResult>>> QueryAsync<TResult>(string connectionString, string query)

The proposed solution should work with them as well. (One of the reasons why Rust went with postfix .await is because they got postfix operators first and (await foo())? everywhere looked horrendous).

[YairHalberstadt]

@orthoxerox

My implementation of Result transforms all exceptions thrown in a method to a result. That's an API design decision which I'm flexible on, but the examples are meant to show the limits of what can be done, not necessarily how I would choose to do it in practice.

The DB methods made me think that a lot of methods can also be actually async as well:

That's an interesting question. I think designing a solution that would work with two different async method builders in the same method would be extremely complicated. I'm not sure there's any good general solution to this. In some cases it may be possible to await the task in the top level method, and then process the result in a local function. In the particular example I gave however that will only work if you have a method to turn a Result<Task<T>> into a Task<Result<T>>. This is easy to do, but there's no generalized method to do this for all monads. Each pair must be coded by hand.

[svick]

@YairHalberstadt

Find a different flavor of arrow that isn't ambiguous.

What about <|? Both < and | are binary operators, so I think combining them into a new operator shouldn't be ambiguous. (Somewhat similar to how combining < and = into <= works.)

---

@louthy

That feels inelegant to me. The arrow signifies a direction, which makes sense in the do notation of Haskell, because there always has to be a value on the left-hand-side of the <-; we know where the value is going with a <- ma and where it's coming from. But, when there's nothing on the left, it feels incomplete.

I agree. Even the = <- syntax fells wrong to me, since you have two operators that kind of mean the same thing ("get the value from the right side and assign it to the left side").

---

return Bar(in value);

That's already valid syntax, so I don't think it could be reused for a completely different purpose.

[louthy]

That's already valid syntax, so I don't think it could be reused for a completely different purpose.

Gah, forgot about that! I'll get my thinking cap on.

think designing a solution that would work with two different async method builders in the same method would be extremely complicated

Unless, the awaiters are automatically composable, I think this could be hard to achieve. It would be interesting to see if it were possible to build transformer awaiters that can contain other awaiters (or an identity awaiter).

[orthoxerox]

That's an interesting question. I think designing a solution that would work with two different async method builders in the same method would be extremely complicated.

Yet necessary. No one will use Result if its sugar won't work in an async method.

[YairHalberstadt]

The DB methods made me think that a lot of methods can also be actually async as well:

public static async Task<Result<IEnumerable<TResult>>> QueryAsync<TResult>(string connectionString, string query)

Warning: technical gibberish ahead.

In general monads don't compose:

http://web.cecs.pdx.edu/~mpj/pubs/RR-1004.pdf

Unfortunately, our real goal, composition of monads, is rather more difficult. .. In fact, we can actually prove that, in a certain sense, there is no way to construct a join function with the type above using only the operations of the two monads (see the appendix for an outline of the proof). It follows that the only way that we might hope to form a composition is if there are some additional constructions linking the two components.

Given two monads A<T>, and B<T>, there is not necessarily any way of converting A<B<T>> to B<A<T>>.

Neither is there any guarantee that A<B<T>> is a monad over T (it's by definition a monad over B<T>. It is however guaranteed to be an applicative over T.

What this means in practice, is that you can always safely await A<B<T>> to extract B<T>, and you can always await B<T> to extract T, as A<B<T>> is an applicative, and therefore a functor. However you can't do so twice in the same method except for specific monads (see the paper above), as that effectively acts as a bind/SelectMany and A<B<T>> is not necessarily a monad.

Now there may be a way to tell the language that a specific monad does compose: For example M<Option<T>> does compose:

M<Option<T1>> SelectMany<T, T1>(this M<Option<T>> mop, Func<T, M<Option<T1>>> func)
{
     mop.SelectMany( x => x.HasValue ? func(x.Value) : M.Unit(Option.None<T1>());
}

However I don't have enough knowledge to design how to do that.

[michaelolof]

Really excited to have stumbled on this issue.
I was also in your shoes a couple of days ago @YairHalberstadt

My implementation of the Result Monad was a bit different though.

To address this:

After discussion with @CyrusNajmabadi on gitter, he felt that whilst this idea is interesting the chief thing holding it back is syntax. The C# team will not be keen on encouraging people to await things that are nothing to do with asynchronous programming.

Essentially I had created a Result<V,E> which was a basic struct. Didn't need to be awaited or anything which worked perfectly
But I quickly realized most of my code was going look like this Task<Result<GoodStuff, BadStuff>> so I tried creating a new monad which will essentially be both a Result and a Task ResultTask<V,E> which I could async/await and directly call my Select and SelectMany methods on it. Awaiting a ResultTask<V,E> should ideally yield a Result<V,E>

But I soon ran into the limitations you mentioned earlier particularly because tasklike-types can only have one generic parameter. Interestingly I also tried using ResultTask where T is a tuple, but the implementation was just too hacky and unusable.

I ended up creating extension methods for Task<Result<V,E>>

(Sidenote: Could it ever be possible to do generics like this?)

// Result is taking 1 tuple of two destructured generic types
struct Result<(V, E)> {
  V val;
  E err;
}

[AlgorithmsAreCool]

I was surprised i didn't see F# Computation Expressions referenced here, they do basically this exact thing and it is awesome.

Instead of the arrow syntax, has any thought been given to borrowing do or do! from F#?

I can see the ! being problematic with all of the nullable stuff however...

[nillkitty]

Happy I stumbled onto this. I've been using custom awaiters to unravel control-flow hell in instant messaging bots to make conversational logic more simple:

   int min = await Prompt.Ask<int>("What's the minimum number you'd like to filter by?");
   int max = await Prompt.Ask<int>("OK good, now what's the maximum number?");
   Reply("Here you go:\n\n" + FilterResults(min, max));

In which control flow bails (or throws a ConversationAbortedException) if the user never responds, or starts a new conversation via a different command. I've run into the same types of Task-oriented issues around this, specifically the finally issue you mentioned.

[timcassell]

Looking at your Awaitables experimental repo, "However all three of them almost work.", I think it would work completely if async functions were directly cancelable (not using throw). #4565 Then this could just be implemented in libraries like yours instead of baked into the language. (Also the AwaitableEnumerable is counter-intuitive, I would expect it to behave like IAsyncEnumerable, which would require new builder machinery #3629, but I get that's not the intention behind it.)