[Bug]: Provider produced inconsistent final plan with aws_lambda_function skip_destroy

[xxfatumxx]

Terraform Core Version

1.3.6

AWS Provider Version

4.57

Affected Resource(s)

aws_lambda_function.lambda_function detected so far, might be more

Expected Behavior

lambda should be deployed successfully.

Actual Behavior

Returns error "Provider produced inconsistent final plan" for all lambda functions' skip_destroy attribute

Relevant Error/Panic Output Snippet

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ module.blabla.aws_lambda_function.lambda_function to include new
│ values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .skip_destroy: was cty.False, but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.57"
    }
  }
}

Steps to Reproduce

• Deploy lambda with provider version 4.55
• Try to re-deploy newer version of lambda with provider version 4.57

Rollback to 4.55 rectifies the issue.

Debug Output

No response

Panic Output

No response

Important Factoids

Caught it on dev env so for security reasons cannot provide more detailed code and outputs, also haven't tested it on dummy code yet. If it won't be reproduced with dummy lambda, I will try to investigate further on my side. Thank you in advance.

References

Relates #29646.

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[dheeg]

Same for us. Provider 4.56 works fine.

[jmacmullin]

Same here :-(

[n00borama]

We've pinned provider 4.56.0, as you guys have, that works but bit of a faff when we commit the provider to pipelines to keep them 'static'.

[justinretzolk]

Hi all 👋 Thank you for taking the time to raise this, and for the additional reports. We've attempted to reproduce this internally, but so far have not been able to.

@xxfatumxx, I know you mentioned that you're unable to provide an exact configuration. Are you able to provide any information around what attributes are being changed during the apply when you experience the issue?

If anyone else who has experienced this is able to provide a working reproduction, or any additional details, or even debug logging (redacted as needed) we'd be grateful for that as well.

[meetvasu15]

Got same issue today. I use a module for the lambda function "terraform-aws-modules/lambda/aws". FWIW, I can share the config of the module here that was originally working.

module "my-lambda-func" {
  source                                  = "terraform-aws-modules/lambda/aws"
  version                                 = "4.1.1"
  role_name                               = "${random_pet.this.id}_${var.func_name}_role"
  function_name                           = "${random_pet.this.id}_${var.func_name}"
  description                             = "This function listen cloudwatch logs event"
  handler                                 = "${var.py_file_name}.lambda_handler"
  runtime                                 = "python3.8"
  publish                                 = true
  source_path                             = "${var.dir_path}${var.py_file_name}.py"
  create_package                          = true
  ignore_source_code_hash                 = false
  store_on_s3                             = false
  create_current_version_allowed_triggers = false
  environment_variables                   = { "LAMBDA_FUNCTION_ARN" : module.another-lambda-func.lambda_function_arn }
  allowed_triggers = {
    OneRule = {
      principal  = "events.amazonaws.com"
      source_arn = aws_cloudwatch_event_rule.new_log_group_created.arn
    }
  }
  attach_policy_json = true
  # allows lambda to access cloudwatch, logs
  policy_json = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*",
                "logs:*"
            ],
            "Resource": ["*"]
        }
    ]
}
EOF
  depends_on  = [module.another-lambda-func]
}

[justinretzolk]

@meetvasu15 👋 Thank you for supplying that! Unfortunately we're still not able to reproduce the issue. Were any changes made to the configuration when the issue started to present itself?

[meetvasu15]

@justinretzolk Just the tags were updated. The issue I saw is on my automated CICD setup, I tried reproducing it on my local with the same lambda tf but couldn't. I would suspect this has to do with some combination of tf, aws version.
The CICD env where I see the issue is an older version of tf. The following is the version output

+ terraform version
Terraform v1.2.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.57.0
+ provider registry.terraform.io/hashicorp/external v2.2.3
+ provider registry.terraform.io/hashicorp/local v2.3.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/vault v3.13.0

And on my local, which doesnt have the issue has the following version info

Terraform v1.3.6
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v4.50.0
+ provider registry.terraform.io/hashicorp/external v2.2.3
+ provider registry.terraform.io/hashicorp/local v2.3.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/vault v3.12.0

[n00borama]

@justinretzolk, here is our plan that prompts the error:

aws_lambda_function.router will be updated in-place
  ~ resource "aws_lambda_function" "router" {
        id                             = "product-us-east-2-router"
      ~ last_modified                  = "2023-02-24T16:12:46.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:accountID:function:product-us-east-2-router:8" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:accountID:function:product-us-east-2-router:8/invocations" -> (known after apply)
      + skip_destroy                   = false
      ~ source_code_hash               = "hash" -> (known after apply)
        tags                           = {}
      ~ version                        = "8" -> (known after apply)
        # (16 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }

Slightly redacted some values, but beyond code hash changes it's mostly wanting to add skip_destroy which prompts the issue.

Then the eventual error:

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_lambda_function.router to include new
│ values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .skip_destroy: was cty.False, but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Versions:

Provider: hashicorp/aws v4.57.0
TERRAFORM='1.3.5'
TERRAGRUNT='v0.40.2'

[xxfatumxx]

Tested on localstack with dummy lambda and it works fine on 4.57.0 :( Also tried: forcefully use skip_destroy attribute true|false, changing lambda code, cleaning up cache before apply - no luck. Once my env will be less busy I will try to reproduce it once again.

[ericrichtert]

I'm having the same issue.

λ terraform version
Terraform v1.3.9
on windows_amd64

• provider registry.terraform.io/hashicorp/archive v2.3.0
• provider registry.terraform.io/hashicorp/aws v4.57.0
• provider registry.terraform.io/hashicorp/external v2.2.3
  29777.log.gpg.gz

[einarjohnson]

Same here. Reverting back to 4.56.0 fixes the problem.

[ericrichtert]

The 4.57.1 version fixed (at least) my issues

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.