Compile inflights into orchestrated workflows

[Chriscbr]

Community Note

Please vote by adding a 👍 reaction to the issue to help us prioritize.
If you are interested to work on this issue, please leave a comment.

Feature Spec

We are excited to announce Wing SDK now contains support for the cloud.Workflow resource. The workflow resource converts a piece of inflight code into a cloud-specific distributed workflow where individual steps can be automatically retried, run in parallel, and so on, without requiring the programmer to define state machines or configure resource permissions.

When a user compiles a Wing program with a cloud.Workflow resource to the tf-aws target, Wing generates an AWS Step Functions state machine, whose definition is automatically configured from the control flow of the inflight.

To achieve this, the Wing compiler internals has been updated so that instead of generating inflight code snippets for each inflight and referencing them in the preflight.js file, an abstract syntax tree for the inflight code is generated and inlined into preflight.js. Correspondingly, the Wing SDK has been updated to accept this data structure wherever inflight closures are expected in constructor/method parameters (like new cloud.Function(closure) and bucket.on_upload(closure), and to automatically convert the ASTs into Amazon States Language or JavaScript based on which resource is using the inflight.

Use Cases

Writing AWS Step Functions definitions by hand or by defining state machine definitions is tricky and as a user I would like to simply have them generated from my code.

Implementation Notes

See https://github.com/Stedi/ts2asl for inspiration

Component

Language Design, Compiler, SDK

[Chriscbr]

Here are some notes and takeaways I took from a discussion we with @ekeren, @iliapolo and others:

• orchestrations vs choreographies (two kinds of doing workflows)
  • orchestrations are where the workflow is managed and overseen by a service like AWS Step Functions, so state is tracked, and orchestration service is
  • choreographies are where the workflow is implicit from event-driven programming, e.g. by calling invoke on several lambda functions in parallel and awaiting their results before joining them
• orchestrations are useful because they are more reliable and provide better visibility
• "for parallel" - keyword/control structure we might want to iterate but do work over it in parallel
• "retry" - keyword/control structure we might want for defining exception handling semantics
• even though we want to support AWS Step Functions, we want some intermediate format so it's possible for the SDK to convert this format into a cloud function, or a Step Function workflow, or an Apache Airflow workflow, etc.

Cool observations:

• When you are coding on a single machine, there are situations where you need to e.g. rewrite your code from recursion to iteration in order to avoid a stackoverflow error. Likewise, in the cloud, there are cases where you need to convert code from iteration to recursion to avoid service limits. (Maybe in the future, Wing could automatically transform users' code for these situations...)

Open questions:

• How to handle different kinds of errors? e.g. if I call bucket.put in a lambda I may get one type of error from AWS SDK, but if I call S3PutObject in an AWS Step Functions workflow, I may get a different type of error
• What kinds of features & control structures are supported in AWS Step Functions and other workflow engines but not in our language?
• Are there ways Wing can help with more operational concerns of creating and maintaining workflows (like setting up dead letter queues, etc.)

Workflow we tried to implement:

Some pseudocode:

//native wing  
bring cloud; 
let bucket = new cloud.Bucket();
let schedule = new cloud.Schedule();
let processing_queue = new cloud.Queue();
// TODO - add a retry mechanism for reingest 
let reingest = new cloud.Function(inflight (f: str) => {
  // do a bunch of things that take less 5 minutes
  processing_queue.publish(f);
}); 
let start_phase = new cloud.Workflow(inflight (continuation_token: str?) => {
  let page = bucket.list_page(continuation_token);
  if page.has_more {
    // give room for on demand work
    retry() {
      std.sleep(5s);
    }
    
    // continue as new 
    start_phase.start(page.next_token);
  }
  for parallel f in page.objects {  
    if f.ends_with("/metadata.json") {      
      try {
        reingest.invoke(f);
      } retry(TooManyRequests) {
        filter: 
        backoff_rate: 1.1,
        interval: 1m
        max_attempts: 30
      }
    }
  }
});
let start_phase_with_while = new cloud.Workflow(inflight () => {
  readwrite token =  null; 
  
  do {
    let page = bucket.list_page(token);
    if page.has_next() {
      // give room for on demand work
      std.sleep(5s);
    }
    for parallel f in page.objects {  
     if f.ends_with("/metadata.json") {
        reingest(f);
      }
    } // parallel await all promises  
    token = page.next_token()
  }
  while (token)
  
  
});
schedule.on_cron("every 15 min", start_phase);

[eladb]

See https://argoproj.github.io/workflows/

[eladb]

https://www.airplane.dev/blog/introducing-powerful-multi-step-workflows-in-airplane

[eladb]

https://github.com/mbonig/state-machine

[skorfmann]

Just FYI: I like what temporal is doing. Since they own the entire stack of their workflows, they can probably do a bit more out of the box. Having said that, I like that their workflows are looking like plain code, but can easily defer / sleep for months anywhere in the script. Here's an example https://github.com/temporalio/samples-typescript/blob/main/food-delivery/packages/workflows/order.ts

[ekeren]

Yes, temporal code is extremely straightforward. I remember looking at some code example that said sleep(1year)

[skorfmann]

Another thing I stumbled upon: https://www.inngest.com - here with a more detailed example https://www.inngest.com/patterns/event-coordination-for-lost-customers - it's a bit similar to temporal.

I really like the way this works, while the DX is pretty straightforward it allows building pretty sophisticated workflows which are hard to build otherwise. I believe a similar approach could be worthwhile exploring for Wing as well.

[Chriscbr]

Converting this to a discussion