Version Publishing for AWS Lambda

[brian-polly]

New user here, totally in love with Terraform. I'm trying to use it in a few API Gateway/Lambda projects, and it's lifechanging.

With that said, one of the cooler new features of Lambda is the ability to publish versions: it gives us a clear, easy, guaranteed rollback. While I'm totally willing to give that up in order to use Terraform, it feels funny that it's missing.

The AWS API documentation for the feature I'm after is here:

Ideally, we'd be able to publish only when the lambda function is updated, but honestly, publishing every time i applied a configuration would also be an improvement over the status quo.

Thanks, everybody.

[apparentlymart]

Hi @brian-polly! Thanks for this feature request.

Terraform's model tends to be a bit weird for externally-versioned stuff since Terraform resources are built around a "Create, Read, Update, Delete" model, but I think one way we could support what you're after here is with a new resource like this:

resource "aws_lambda_function_version" "foo" {
    function_name = "${aws_lambda_function.foo.function_name}"
    source_code_hash = "${aws_lambda_function.foo.source_code_hash}"
}
output "lambda_function_version" {
    // Get the version number assigned by AWS
    value = "${aws_lambda_function_version.foo.version}"
}

I'm imagining its lifecycle as follows:

• Create: Call PublishVersion and write the version number into the exported version attribute.
• Update: Same as Create. This would leave the existing version alone and create a new one.
• Read: Do nothing, assuming that a version is immutable once it's created.
• Delete: Do nothing on AWS, but discard the data cached in Terraform's state

However, I think there might be some tricky interaction here because the source_code_hash of aws_lambda_function is "computed" and so Terraform is likely to be overly-cautious about generating update diffs for the version resource, since it won't be able to tell automatically whether the value is going to change during apply. It may thus do the sub-optimal behavior you mentioned where it creates a new version on every run. I'm not 100% sure, so we'd need to build a prototype and try it to see exactly how Terraform behaves in this case.

[brian-polly]

It's not super clear to me how the source_code_hash works, but is there any reason that we couldn't feed it the same input that we feed the AWS Lambda Function resource: "${base64sha256(file("lambda_function_payload.zip"))}"?

I don't know if there's some reason that it only works in the context of the lambda resource, but if not, it seems like there shouldn't be any problem using that for both resources. As far as I can tell, I have to have access to the zipped payload in order to version the lambda function in the first place, no?

[radeksimko]

This feature was recently discussed somewhere in a different thread about Lambda and I agree it would be nice to support it, but the lack of it is not burning me personally.

The good news is that there seems to be quite a simple solution - the Create API method accepts Publish parameter (bool) which seems to be doing what you described above.

https://github.com/aws/aws-sdk-go/blob/master/service/lambda/api.go#L1139-L1141

I personally don't plan to send a PR, but I'd be happy to review (and eventually merge) one, as long as it also has a test. 😉

[apparentlymart]

@brian-polly, @radeksimko's solution seems much simpler than what I proposed, and so I think better to implement that way, but to answer your question:

Providing the same expression in both places should work fine, but I usually try to avoid duplicating expressions in multiple places in my config, instead preferring to write them out at the "top" of the dependency tree and then refer up the tree. This means if I change the original expression I only need to change it in one place, and it also implicitly creates a dependency between the two resources (in this case the function_name interpolation also creates that dependency, so not strictly necessary here).

This is more a matter of style than anything else. Either way would work, though you'd need to make sure that Terraform is able to know the dependency between these somehow, or else it might try to create the version before it's updated the function.

[brian-polly]

@apparentlymart That makes a lot of sense -- I'm definitely still acclimating myself to what "good style" means here.

[radeksimko]

I was thinking about this again when I noticed couple days ago that AWS introduced AWS::Lambda::Version type for CloudFormation.

I think Terraform could have a similar resource (aws_lambda_function_version) which would have similar set of fields

• source_code_hash
• description
• function_name

and (as opposed to CloudFormation) possibly these too

• filename
• s3_bucket
• s3_key
• s3_object_version

It would be up to the user to decide about the workflow for deployments:

1. Update s3_* or filename & source_code_hash in aws_lambda_function and then publish new version via aws_lambda_function_version which would have source_code_hash = "${aws_lambda_function.x.source_code_hash}"
2. Have aws_lambda_function with filename or s3_* pointing to some kind of "placeholder" code to start with and then use aws_lambda_function_version's filename or s3_* to do both deployment of the code and version publishing. This would also imply having lifecycle { ignore_updates = ["aws_lambda_function.s3_..."] } etc. to prevent collisions.

In terms of CRUD operations on such resource I could imagine it to work like this:

• Create: lambda:PublishVersion
• Read: either lambda:ListVersionsByFunction or lambda:GetAlias
• Delete: lambda:DeleteAlias

Overall I'm not sure how far are versions and aliases interchangeable though. More likely Delete would end up being just noop as it may not be possible to delete old versions.

[apparentlymart]

@radeksimko, I don't really understand why we'd need to duplicate all the code-location-specification attributes from aws_lambda_function on a version. What am I missing? 😀@radeksimko,

Looks like your workflow 1 is basically my original proposal, which also happens to (by lucky coincidence) exactly match the CloudFormation model:

resource "aws_lambda_function" "foo" {
    // ...
}
resource "aws_lambda_function_version" "foo" {
    function_name = "${aws_lambda_function.foo.function_name}"
    source_code_hash = "${aws_lambda_function.foo.source_code_hash}"
}

It seems like workflow two could be written as you originally suggested, without involving and complicating the aws_lambda_function_version resource:

resource "aws_lambda_function" "foo" {
    publish_version = true
}

Seems like the decider between these two approaches is that workflow 1 allows you to potentially create versions for only selected updates, though it'd definitely require some Terraform acrobatics to do that. For example, you could create the function and the version in two separate Terraform modules where the latter uses terraform_remote_state to obtain the latest code hash from the former.

Given how rare that is, I expect the second workflow would be used in 99% of cases and so my instinct is to start with supporting only workflow 2 as you originally proposed, and wait for real use-cases to emerge for workflow 1 before supporting it in Terraform.

[radeksimko]

I don't really understand why we'd need to duplicate all the code-location-specification attributes from aws_lambda_function on a version. What am I missing?

I'm thinking about how to make this compatible with the "Otto-like" deployment model (otto infra + otto deploy) - i.e. imagine you have two Terraform contexts/directories (connected via terraform_remote_state):

• 1st for resources that typically don't change per version and you typically don't want to touch nor change by accident (app-specific S3 bucket, DNS, IAM Roles+policies, ELB, CloudWatch Log Group, ASG, ...)
• 2nd only for resources that are likely to change per version (Lambda Function Version, ECS task definition, ECS service, K8S pod, ...)

You could say that the whole aws_lambda_function should just be always in the 2nd context, but that implicitly forces me to also move things I'd normally have in the 1st context into 2nd too:

• Lambda Permission
• any IAM resources that need to reference ARN of the Lambda function (API Gateway)
• Lambda Event Mappings
• CloudWatch Event Rule Targets

I think that changing IAM per version should be avoidable. AWS API key for common deployment should not need any iam:__ permissions, otherwise the IAM-based model is becoming less predictable/secure if every single deployment can theoretically modify permissions of the app.

I know of organisations using Terraform that have separate contexts/directories for all IAM resources (for good reasons) and I'm probably going to do that too very soon.

Yes, IAM permissions based on on Lambda versions still don't fit into this theory well, but I think it's ok to scope permissions per functions, not versions.

Given how rare that is, I expect the second workflow would be used in 99% of cases

I would personally want to use the 2nd workflow most of the time, because of the risks & reasons mentioned above.

---

I'm aware that not many people may be using Terraform for real-life app deployments yet (as opposed to single-context infra deployments), but I'd say it can be more common in the future if we make it possible.

[apparentlymart]

Of course where exactly to draw the line between "infrastructure" and "app" will always be very subjective 😀 but thinking specifically about your IAM example it seems like this is already solved by having aws_iam_role as a separate resource which the aws_lambda_function can reference. Consequently, I'd be inclined to think of the lambda function as being part of "the app", with it obtaining the IAM role from "the infrastructure".

However, I'd agree that the permissions present a particular problem: the function must be created before the permissions can be assigned. This seems to be a limitation of Lambda's design where the permissions refer to their function, rather than the other way around as with IAM roles/policies.

I certainly agree that being able to do what you describe would be ideal, but I worry about adding too much extra Terraform-specific behavior/workflow on top of the basic Lambda API operations.

---

Like you said, having aws_lambda_function_version also be able to do deployments creates the tricky situation of two resources trying to update the same data. I feel that forcing a user to set ignore_updates suggests a design flaw, and I don't think this should be required for any primary use-case.

One answer would be to remove the possibility of uploading code from the main aws_lambda_function resource altogether, and require that aws_lambda_function_version always be used to upload code:

resource "aws_lambda_function" "example" {
    function_name = "example"
    role = "${aws_iam_role.iam_for_lambda.arn}"

    // filename, handler, source_code_hash, s3_bucket, etc are not supported on this resource at all,
    // and this function just creates a function with an empty placeholder implementation, ignoring
    // the implementation on all future Refresh/Diff.
}

resource "aws_lambda_function_permission" "example" {
    function_name = "${aws_lambda_function.example.function_name}"
    // ...
}

resource "aws_lambda_function_version" "example" {
    function_name = "${aws_lambda_function.example.function_name}"
    filename = "lambda_function_payload.zip"
    handler = "doSomething"
    source_code_hash = "${base64sha256(file("lambda_function_payload.zip"))}"
}

In this model "infrastructure-ish" things would depend on the aws_lambda_function and "app-ish" things would depend on the aws_lambda_function_version. Users could then decide whether (for example) an aws_api_gateway_integration is part of "the app", in which case it could refer to the version directly, or if it's part of "the infrastructure", in which case I expect it'd refer to a version alias that is managed by "the app". That decision would presumably depend on how tightly coupled the API and the implementation functions are, and whether they are expected to evolve separately.

This would of course be a breaking change at this point. Perhaps Lambda support in Terraform and Lambda usage in the real world are early enough that we could tolerate such a thing? I'm not sure... and I still feel a little weird making Terraform subvert the workflow intended by the Lambda product team.

[cristobal23]

Is anyone here pinning their lambda function versions to version aliases? http://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html Perhaps we could use a new resources aws_lambda_function_alias and aws_lambda_function_alias_version which seems to be more CRUD like.

[kmamykin]

I have been experimenting with both CloudFormation and Terraform to setup an decent automatic CI deploy process to aws. Ideally the CI process would build/zip/upload the ${commit}-build.zip to a bucket and configure a new version of lambda function(s) with that code (keeping old versions and their config immutable).

TLDR: From what I've learned, I would vote in favor of publish_version property on aws_lambda_function resource as a solution instead of an aws_lambda_function_version resource.

In fact AWS docs argue the same on http://docs.aws.amazon.com/lambda/latest/dg/versioning-intro.html

We recommend that you publish a version at the same time that you create your Lambda function or update your Lambda function code, especially when multiple developers contribute to the same Lambda function development.

Currently, CloudFormation does not support versioning of Lambda function well. Spend half a day banging my head. AWS::Lambda::Function resource does not have "Publish" property yet (but the latest API docs do http://docs.aws.amazon.com/lambda/latest/dg/API_CreateFunction.html). I believe CF is just behind in the release cycle and eventually "Publish" will be supported.

CF AWS::Lambda::Version resource is useless in it's current form. Function's version is created on create-stack operation, but update-stack either errors out (Updates are not supported) when updating sha/description or has no effect (as version resource already exists).

All other lambda resources support versioned Arn or qualified/versioned function name: AddPermission, CreateEventSourceMapping, CreateAlias. I think Terraform would need to adopt semantic to always create Permission/EventSourceMapping for the newly published version and update Alias to point to the version.

To clarify here is my take on the desired terraform resources:

resource "aws_lambda_function" "example" {
    function_name = "example"
    role = "${aws_iam_role.iam_for_lambda.arn}"
    filename = "lambda_function_payload.zip"
    handler = "doSomething"
    source_code_hash = "${base64sha256(file("lambda_function_payload.zip"))}"
    publish = true // this bit is important
}

resource "aws_lambda_function_permission" "example" {
    function_name = "${aws_lambda_function.example.function_name}" 
    qualifier = "${aws_lambda_function.example.version}"
}

resource "aws_lambda_event_source_mapping" "event_source_mapping" {
    function_name = "${aws_lambda_function.example.arn}" // 
}

resource "aws_lambda_alias" "test_alias" {
        name = "testalias"
        function_name = "${aws_lambda_function.example.function_name}"
        function_version = "${aws_lambda_function.example.version}"
}

[kmamykin]

Here is my basic support: #6233
Looking for someone to review if I am moving in the right direction

[rschweter]

Any updates on this? I'd love to see version support added in.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.