Lambda Layers - Terraform deploys new version and removes the old one

[solveretur]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11

Affected Resource(s)

• aws_lambda_layer_version

Terraform Configuration Files

provider "aws" {
  region = "eu-central-1"
}

# If archive file is changed, updates zip to S3 repository
resource "aws_s3_bucket_object" "object_lambda_common_layer" {
  bucket = "dev-dwh-listings-dw-stst"
  key = "lambda/layers/lambda_common_layer/lambda_common_layer.zip"
  source = "${data.archive_file.layer_zip_lambda_common_layer.output_path}"
  depends_on = [
    "data.archive_file.layer_zip_lambda_common_layer"]
}

data "archive_file" "layer_zip_lambda_common_layer" {
  type = "zip"
  source_dir = "../../src"
  output_path = "../../lambda_common_layer.zip"
}

resource "aws_lambda_layer_version" "lambda_common_layer" {
  layer_name = "lambda_common_layer"
  s3_bucket = "${aws_s3_bucket_object.object_lambda_common_layer.bucket}"
  s3_key = "${aws_s3_bucket_object.object_lambda_common_layer.key}"
  s3_object_version = "${aws_s3_bucket_object.object_lambda_common_layer.version_id}"
  description = "Common layer providing logging"
  compatible_runtimes = ["python3.6"]
}

terraform {
  backend "s3" {
    bucket         = "tfstate-dev"
    region         = "eu-central-1"
    key            = "service/lambda/layers/lambda_common_layer.tfenv"
    dynamodb_table = "terraform_locks"
  }
}

Debug Output

Panic Output

Expected Behavior

If something changes in the code It should just add lambda layer new version and keep the old one

Actual Behavior

If something changes in the code, It adds lambda layer new version but everytime it also deletes the old one.

Steps to Reproduce

1. terraform apply

Important Factoids

• S3 bucket where I store layers zip has versioning enabled
• If nothing has changed in the code, it does not do anything which is correct

References

• https://stackoverflow.com/questions/54648704/terraform-deletes-old-aws-lambda-layer-version-instead-of-creating-new/

[amancevice]

+1 for this. It creates a really undesirable situation where Lambdas with old versions of layers will break when the layer is updated.

[asimb123]

+1

[gchamon]

Just arrived at a similar issue.

In my case I have the code deployment stage separated from the infrastructure deployment stage, so when creating the lambda infrastructure I don't have access to the actual code.

In my case, deploying a version and then updating it with ansible results in a layer version that terraform is unaware of.

My solution, which maybe good for your usecase too, is to manage layer versions outside terraform, and import them using Data Sources, like this:

• Compile, pack and deploy lambda code

aws s3 cp {{ lambda_function_release_file }} s3://{{ lambda_s3_bucket }}/{{ s3_key }}

• Create new lambda layer version

aws lambda publish-layer-version \
          --layer-name 'my-layer' \
          --content S3Bucket={{ lambda_s3_bucket }},S3Key={{ s3_key }} \
          --compatible-runtimes {{ lambda_function.compatible_runtimes }}

• Access the version in terraform:

data "aws_lambda_layer_version" "existing" {
  layer_name = "my-layer"
}

There is a tradeoff in it though. You won't be managing a part of your infrastructure with it though, at least not directly, but at least the older layer versions won't be in jeopardy when updating lambda layer code.

A workaround for this would be to trigger code when deleting a function, so that its layers are also deleted, like so:

resource "aws_lambda_function" "example" {
  # ... other configuration ...
  layers = ["${aws_lambda_layer_version.example.arn}"]
}

resource "null_resource" "delete-layers" {
  depends_on = [aws_lambda_function.example]

  provisioner "local-exec" {
    when = "destroy"
    command = "aws lambda delete-layer-version --layer-name my-layer --version-number 1"
  }
}

From the documentation, however, it is stated that "when you delete a layer version, you can no longer configure functions to use it. However, any function that already uses the version continues to have access to it."[1] So it should not interfere with functions already configured with older layer versions when terraform deletes layer versions in order to update code.

[1] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-manage

[vfilimonov]

From the documentation, however, it is stated that "when you delete a layer version, you can no longer configure functions to use it. However, any function that already uses the version continues to have access to it."[1] So it should not interfere with functions already configured with older layer versions when terraform deletes layer versions in order to update code.

That is indeed true. However it will prevent you to redeploy existing lambda. E.g. if you have some CI/CD routines that deploy some lambdas automatically, you would actually have to go and modify the layer requirements for all of them, otherwise all deploys will crash.

Furthermore, if the new version of layer brings some incompatibility issues (e.g. the lambdas working with layer-v1 would be broken after being updated to layer-v2) there would be no easy way to resolve the problem.

@CGreenburg I saw you've pushed a PR 👍👍
Are there any news on it?

[abhilashnarayanan]

Hi Team, when this fix will be released?
@CGreenburg

[dkryptr]

It's waiting on integration tests I think. I haven't added them in that merge request which is why it can't move forward. If anyone is willing to add them, feel free to fork and submit a new pull request. Thanks!

[abhilashnarayanan]

Thanks for the information! the fix is much needed for our Organization. Will you be able to work on this or can you help me where/what the impacted component is and the necessary changes (on a high level) -as i haven't worked on this earlier.

[sankalpkotewar]

For now one can make use of terraform command state rm, more details here

[Geartrixy]

ETA on the MR :) ?

[clemlesne]

Hi, we encounter the issue and this is damageable for us. Is someone is working on this?

[schollii]

The only viable workaround for us so far is to call terraform state rm aws_lambda_layer_version.NAME before calling terraform apply.

However, in general you will want to do this only if there is a modification planned so you would want to do terraform plan and check to see if aws_lambda_layer_version.NAME has a planned change (rather than a creation), if it does then rm it from the state.

Unfortunately, there is then no longer a way of automating the removal of all versions of a layer upon terraform destroy.

I think the real problem here is that aws_lambda_layer_version is the wrong concept to model lambda layer provisioning. It should be aws_lambda_layer and the behavior should be that if the zip given to that object has changed (this can be done by getting the latest layer version, which aws cli does, finding the sha of the code, and comparing it to the sha of the new zip), then create a new layer version, otherwise don't do anything; and on destroy, list layer versions and remove all versions.

@apparentlymart is there anything that can be done to solve this issue? I'm not familiar enough with the regression testing system to help or I would.

[jammymalina]

Any update on this?

[github-actions]

This functionality has been released in v3.66.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.