(aws-stepfunctions): L3 Construct for API Gateway HTTP/REST Api Backed by Step Functions

[diegotry]

As a CDK user, I'd like to have a simpler way to create an API Gateway (either REST or HTTP) with Step Functions Synchronous Workflows. The new construct will be responsible for creating the API Gateway resources, connecting it to a Step Functions State Machine with the necessary permissions and predefined features.

Use Case

I want to build APIs that are handled by Step Functions as my API orchestrator. The integration with Synchronous workflows was recently launched:

https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-api-gateway-rest-apis-integrates-with-step-funtions-synchronous-express-workflows/ and I want to leverage the infrastructure to CDK.

Proposed Solution

1. StepFunctionsHttpApi

• Backed by API Gateway HTTP API

2. StepFunctionsRestApi

• Backed by API Gateway REST API

Convention over configuration
Convention over configuration (also known as coding by convention) is a software design paradigm used by software frameworks that attempts to decrease the number of decisions that a developer using the framework is required to make without necessarily losing flexibility.

Convention #1: By default, the Output transformation should return only the Execution Output. If possible, I'd like to have:

HTTP 200: Execution succeeded
HTTP 400: Execution completed, but it failed for some execution-level reason. Example: a validation step. These are for known errors.
HTTP 500: Step Functions failure (server-side, not execution level)

Convention #2: The request body is mapped to the execution input.
Convention #3: All logs from Step Functions are enabled by default
Convention #4: X-Ray is enabled on Step Functions State Machine

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[BenChaimberg]

Thank you for your feature request! This seems like a great pattern for customers, but would not really belong in this library as it is fairly abstract (an L3, as you point out). Check out awslabs/aws-solutions-constructs#145 for a better place, maybe you can help the design and implementation effort over there!

[BenChaimberg]

Actually, on a further look, this may be a good fit for aws-apigatewayv2-integrations. What do you think, @nija-at?

[diegotry]

I believe we may offer two types of integrations - or maybe more. My intention is to use Synchronous Workflows from AWS Step Functions and not the standard workflows, which is asynchronous.

HTTP APIs -> Synchronous Workflows (cheaper and potentially faster), but without all the features from REST APIs.
REST APIs -> Synchronous Workflows (has input/output transformation, but it is more expensive at scale).

[nija-at]

From a basic read of this issue, @BenChaimberg is right. This does not have to be an L3. It just needs to an aws service integration pattern.

We are tracking this already here - #11947.

Let me know if there are other reasons why being L3 would be advantageous.

[BenChaimberg]

Closing in favour of the already open issue, any further discussion of the integration should be located there. Discussion of a possible L3 should occur in the solutions constructs issue mentioned above.

[diegotry]

Correct me if I'm wrong.

Integration pattern works for API-GW V2 (which is HTTP APIs product). The Step Functions integration with HTTP APIs needs a similar version of https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-apigatewayv2-integrations.LambdaProxyIntegration.html which is for Lambda.

For API-GW V1 (a.k.a. REST APIs) the integration is a bit different. This is the corresponding for Lambda: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-apigateway.LambdaRestApi.html

For AWS Services, there is no corresponding one. We can make use of:

https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-apigateway.AwsIntegration.html

For HTTP APIs with Step Functions, it returns all the fields from the API. It includes ARNs (state machine ARN and execution ARN), therefore the Account Id is also returned. If I am offering this API to end customers, I don't want them to have access to those informations. Currently, with HTTP APIs is not possible to apply output transformation.

Rest APIs has more features than HTTP APIs, but it costs more. It is possible to remove all the metadata information and return only the execution output to the caller.

I believe we can have another layer of abstraction that makes it even easier for users. For Step Functions, let's say I want to build an API that is backed by Step Functions. Let's call it Serverless/Workflow APIs. In the end, what I want is a public endpoint for my state machine where I can make HTTP requests and start new executions.

The way I understand (For Step Functions):

L1: Raw CloudFormation resources.
L2: Makes it eiaser to create resources by taking care of IAM permissions and so on.

The next abstraction level/use case that I'd like to have is the API-GW that connects to a State Machine. I don't want to manage in my code an integration proxy (what does it even mean? Will I ever have to change this after it is connected?). I just want an endpoint for my state machine.

The convention over configuration approach that I mentioned is related to the default values. For expert users, they can modify the props of each resource to achieve what they want (lower level of logs, different log group, disable xray, manipulate input/output transformation, etc). It can be as simple as providing a state machine definition, a name and the execution role.

My proposal is:

Add L2 constructs to match what API-Gateway has in the console (Lambda, AWS Service, ..). We can have AwsServiceIntegrationRestApi. Step Functions can build on top of that.

Another option is to build constructs for Step Functions with API Gateway resources and the link between them. I think this makes sense to belong on aws-solutions-constructs.

[ibrahimcesar]

I was working in integrating API Gateway HTTP with Synchronous Express Workflows and this proposal is exactly "the missing link". I think these abstractions baked-in would be a huge leap for a better understanding (and use) of the potential Step Functions have. A perfect fit for checkout-like workflows to anyone using CDK.

[mariobittencourt]

Hi there, I am a bit confused about the current state of this issue. Will we have a v2 integration - like the lambdaproxy - for the API gateway so it enables us to startExecution/startSyncExecution of step functions without an in-between lambda?

[nija-at]

@diegotry - I'm not able to follow all of your arguments or statements in your comment. Let me try and respond to the ones I did.
I'm also not very familiar with all of the various integrations available, so bear with me if I'm missing the obvious.

Add L2 constructs to match what API-Gateway has in the console (Lambda, AWS Service, ..). We can have AwsServiceIntegrationRestApi. Step Functions can build on top of that.

We should aim to also build the specific integrations in the AWS CDK. We could have a generic interface for AWS services that are not yet implemented in the AWS CDK.
We have precedence for this. stepfunctions-tasks contains integrates step functions with specific AWS services and apigatewayv2-integrations contains integrates HTTP API with Lambda, ALB, NLB, etc.

In the end, what I want is a public endpoint for my state machine where I can make HTTP requests and start new executions.

This one is on the line. It could be similar to the LambdaRestApi construct and land in the AWS CDK.
However, I don't have a good home for it in the AWS CDK at the moment, so it's best, to keep it with the solutions construct (if they'll accept it).

For HTTP APIs with Step Functions, it returns all the fields from the API. It includes ARNs (state machine ARN and execution ARN), therefore the Account Id is also returned. If I am offering this API to end customers, I don't want them to have access to those informations

Not clear what you're saying here. What returns all the fields and why is it bad?

---

If you're interested in the implementation, before you work on a pull request, I will say that this work needs to start with some initial design. It would be very helpful by providing the CDK user experience of integrating HTTP API or Rest API with Step Functions.
That way, we can iron out the interfaces and APIs before you get too deep into the implementation.

I would also start with small changes, adding the layers required one by one, rather than a large pull request attempting to make all of these changes.

[diegotry]

What returns all the fields and why is it bad?

Let's say I have a website that sends a request to API-GW HTTP APIs. Behind the API I have a Step Functions workflow. As a user, all I want from the workflow is the output, not all the fields that are returned in the StartSyncExecution API.

HTTP APIs returns all the response from the service it integrates. Rest APIs has a feature to manipulate the API output, so in this case, I can return only the workflow output.

From my understanding, apigatewayv2-integrations has HTTP APIs integrations. The integrations are already defined and it will probably fail during creation if I try to use something that doesn't exist. If I go to the AWS console, this is what I see when I try to create an HTTP API with Step Functions integration:

On the other hand, REST APIs has an open text field that I believe it will succeed during creation and fail during the request (Unknown Operation or something). This is what I see in the console (please see that I selected AWS Service):

I believe we need to make it easier for the AWS Service Integration for Rest API. AwsServiceIntegrationRestApi can be the pair of LambdaRestApi. Please note they are in the apigateway-v1 and not apigateway-v2.

My suggestion is that we start with AwsServiceIntegrationRestApi that allows any integration from Rest API to AWS Services. This is probably possible through AwsIntegration but this class does not give me a well-contructed endpoint in the end, it looks more like the glue between the services.

After we have the AwsServiceIntegrationRestApi available, we can build StepFunctionsRestApi that uses AwsServiceIntegrationRestApi.

Later, we can build something for apigatewayv2-integrations which integrates with HTTP APIs.

[BenChaimberg]

Thank you for your clarifications – I think we've come to a better understanding of what you're proposing! This seems to be an API Gateway feature that allows proxy integration with an AWS service (maybe AwsRestApi). Then, we will have a StepFunctionsRestApi subclass that adds some stricter typing and default transformations to pare down to just a response body. My concern here is that the number of services that can actually handle a proxy-style integration is quite small, as the whole point of AwsRestApi is for every route and method to be handled by the service (otherwise, AwsIntegration is the way to go). Do you have an idea for other services that would fall into this pattern?

[diegotry]

My concern here is that the number of services that can actually handle a proxy-style integration is quite small

Not sure I understood your point here. API-GW RestApi offers integration with 100+ services.

Whoever is creating the API should specify what HTTP Method they want and what AWS Operation they want to match. (ex: HTTP POST -> .).

I think the current structure for creating the AWS Service Integration can be simpler and less error-prone if we have well defined structures. Let me spend some time thinking about what the new structure would look like and I will get back to this request.

I agree those should be part of the apigateway module.

[BenChaimberg]

Unassigning myself for now, please @mention me when you have made updates

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.