Bug: Lambda function (with alias) doesn't create a new version after its layer is updated, even if AutoPublishAliasAllProperties is set to true

[zhanzhenzhen]

Description:

If we use alias for a lambda function (for provisioned concurrency, etc), it won't create a new version after its layer is updated, even if AutoPublishAliasAllProperties is set to true. So the alias still points to the old layer version. But the $LATEST version is synced with the new layer version. Am I missing something, or is it a bug?

Steps to reproduce:

1. Create those 4 files, then sam build, then sam deploy. Visit the output URL.
2. After initial deployment, replace the string "old layer" with "new layer" in the layer/layer.mjs file. Build and deploy it again. Visit the output URL.

template.yaml:

AWSTemplateFormatVersion: "2010-09-09"

Transform:
  - AWS::Serverless-2016-10-31

Globals:
  Api:
    Cors:
      AllowMethods: "'*'"
      AllowHeaders: "'*'"
      AllowOrigin: "'*'"
  Function:
    Runtime: nodejs20.x
    Architectures:
      - arm64
    MemorySize: 128
    Timeout: 8

Resources:
  Layer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: !Sub ${AWS::StackName}-Layer
      CompatibleArchitectures:
        - arm64
      CompatibleRuntimes:
        - nodejs20.x
      ContentUri: layer
      RetentionPolicy: Delete
    Metadata:
      BuildMethod: nodejs20.x
      BuildArchitecture: arm64

  AppFunction:
    Type: AWS::Serverless::Function
    Properties:
      AutoPublishAlias: Main
      AutoPublishAliasAllProperties: true
      CodeUri: lambda
      Handler: app.handler
      Layers:
        - !Ref Layer
      Events:
        Api:
          Type: Api
          Properties:
            Path: /app
            Method: GET

Outputs:
  URL:
    Description: "Test URL"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/app"

lambda/app.mjs:

import layer from "/opt/nodejs/layer.mjs";

export const handler = async (event, context) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify({fnVersion: process.env.AWS_LAMBDA_FUNCTION_VERSION, layer: layer})
    };
    return response;
};

layer/layer.mjs:

export default "old layer";

samconfig.toml:

version = 0.1

[default]
[default.global.parameters]
stack_name = "TestApp"

[default.build.parameters]
cached = true
parallel = true

[default.validate.parameters]
lint = true

[default.deploy.parameters]
capabilities = "CAPABILITY_NAMED_IAM"
confirm_changeset = true
resolve_s3 = true
s3_prefix = "TestApp"
image_repositories = []

[default.package.parameters]
resolve_s3 = true

[default.sync.parameters]
watch = true

[default.local_start_api.parameters]
warm_containers = "EAGER"

[default.local_start_lambda.parameters]
warm_containers = "EAGER"

Observed result:

In step 1, the result is: {"fnVersion":"1","layer":"old layer"}.

In step 2, the result remains the same.

Expected result:

I expect the result in step 2 to be {"fnVersion":"2","layer":"new layer"}.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux-6.1.90-99.173.amzn2023.x86_64-x86_64-with-glibc2.34
2. sam --version: 1.114.0
3. AWS region: us-east-1

[hnnasit]

Hi @zhanzhenzhen, thanks for reporting the issue. After chatting with the SAM team, this seems like a known issue. I am transferring the ticket to their repo for the SAM team to provide more context and any workaround.

[paulhcsun]

Hi @zhanzhenzhen,
As @hnnasit mentioned we are currently aware of this issue. Here is some context and a workaround:

Context + Root cause

The reason the AutoPublishAliasAllProperties property is not picking up the changes to the Layers property is because of the specific ordering that we have when transforming the resources listed in a SAM template. The current implementation will translate all AWS::Serverless::Functions before translating the AWS::Serverless::LayerVersion. Because the Serverless::Function is translated first, the !Ref to the Serverless::LayerVersion will not have changed yet - that changes after we transform the LayerVersion but by then it won't go back to transform the Serverless::Function a second time.

Because SAM doesn't resolve intrinsics, we wouldn't necessarily consider this a bug, but we can definitely understand that it is not a great experience for our customers.

To address this issue with a fix, we would have to change the order in which resources are transformed which could result in unintended consequences. We have broken existing templates in the past so we try to tread carefully when making changes like this where it is difficult to estimate the full impact it would have on other customers.

Workaround

As a workaround, you can pass a combined shasum of the .zip used for the LayerVersion's ContentUri + the Lambda function code to the AutoPublishCodeSha256 property. This means that whenever the lambda or the referenced layer is changed then a new lambda version will be created.

shasum ~/.../layer.zip
26babb64f2c18e1a845cbdb684e6ef2f99319773  /.../Downloads/layer.zip

The combined shasum can be passed in as a global paramater:

Parameters:
  AutoDeployVersion:
    Type: String

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      ...
      AutoPublishAlias: foo
      AutoPublishAliasAllProperties: True
      AutoPublishCodeSha256: !Ref AutoDeployVersion
      Layers:
        - !Ref TestEnvLayer

and this can be set with the --parameters AutoDeployVersion="sha value" flag through the CLI deployment command you are using.

Please give this a try and let me know if you have any other questions!

[zhanzhenzhen]

@paulhcsun Thank you for providing the workaround. Could you explain this in more detail:

a combined shasum of the .zip used for the LayerVersion's ContentUri + the Lambda function code

I usually only use sam build and sam deploy commands. Could you tell me how to generate the layer.zip file? I looked into sam deploy but cannot find --parameters. Do you mean --parameter-overrides?

[zhanzhenzhen]

Because the Serverless::Function is translated first, the !Ref to the Serverless::LayerVersion will not have changed yet - that changes after we transform the LayerVersion but by then it won't go back to transform the Serverless::Function a second time.

But why it updates the layer version when the function is not using alias?

[paulhcsun]

Sorry yes, --parameter-overrides is the correct flag. And ah my bad, the layer.zip is from a previous issue that I used this workaround for. In your case you could just the shasum command for your layer/layer.mjs file.

If you are just manually deploying and want a simpler workaround, you could just add a Description property to your Function and manually change the value there when you want to deploy a new Lambda Version which should be picked up by the AutoPublishAliasAllProperties.

But why it updates the layer version when the function is not using alias?

I'm not quite sure I follow your question here, could you elaborate what you mean?

[zhanzhenzhen]

I'm not quite sure I follow your question here, could you elaborate what you mean?

It seems that if a lambda function isn't deployed with AutoPublishAlias or AutoPublishAliasAllProperties, it's synced with new layer versions. I don't know why.

[paulhcsun]

Oh that's interesting, I'll try to investigate why that happens but at the moment I'm also not sure why it would properly sync with new layer versions if those properties are not specified.

I'll move this into Discussion to collect more feedback from anyone facing this issue in the meantime.

[miguel-aws]

Facing the same issue.

Also when implementing the "Description workaround", it seems that SAM creates new Lambda version only if the hardcoded property value changes.

For example, a new Lambda version is created if you add or change Description value:

Description: abcde

But it won't work if you parameterize the Description value:

Description: !Ref DescriptionValueParam

It's cumbersome to manually change the Description value everytime we need to update something to the layer, so we thought of using a parameter, but that doesn't work either.

paulhcsun, can you confirm that SAM doesn't evaluate referenced values, but looks at the hardcoded property values to determine if a new Lambda version should be created?