terraform fails to modify lambda function, but performing same action succeeds in AWS console

[f0rk]

Background

I created the lambda function and other resources. I then wanted to update the lambda function to run inside the VPC. I added the policy attachment above as well as the vpc_config. But I got the above error when I ran terraform apply. So I went into the AWS console and set the subnets and security group ids from the lambda function's manage page. I then ran terraform apply again, and no changes were made.

Terraform Version

Terraform v0.11.1
+ provider.aws v1.6.0

Affected Resource(s)

Affected resources:

• aws_lambda_function

Terraform Configuration Files

resource "aws_iam_role" "lambda_exec_role" {
  name = "lambda_exec_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
  EOF
}

resource "aws_iam_role_policy_attachment" "lamba_exec_role_eni" {
  role = "${aws_iam_role.lambda_exec_role.name}"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

resource "aws_lambda_function" "api" {
  depends_on = ["aws_iam_role_policy_attachment.lamba_exec_role_eni"]

  s3_bucket = "example-lambda-packages-bucket"
  s3_key = "api.zip"
  function_name = "api"
  description = "API"
  role = "${aws_iam_role.lambda_exec_role.arn}"
  runtime = "python3.6"
  source_code_hash = "..."
  handler = "index.handler"
  timeout = 10

  vpc_config = {
    subnet_ids = ["..."]
    security_group_ids = ["..."]
  }
}

Debug Output

I don't have debug output, as when I undo the change from the console, terraform can successfully re-apply the change. It's possible this error only occurs when the role policy attachment and the lambda function modification happen in the same run.

Expected Behavior

Apply complete!

Actual Behavior

Error: Error applying plan:

1 error(s) occurred:

* aws_lambda_function.api: 1 error(s) occurred:

* aws_lambda_function.api: Error modifying Lambda Function Configuration api: InvalidParameterValueException: The provided execution role does not have permissions to call CreateNetworkInterface on EC2
        status code: 400, request id: 23474581-efe2-11e7-b0b3-afe4f8d056bc

Steps to Reproduce

Steps:

1. Create the above configuration without vpc_config
2. Run terraform apply and verify that the function is created.
3. Add vpc_config
4. Receive the above error.

[f0rk]

I actually destroyed and recreated the function, it only occurs when both the policy attachment and vpc configuration happen at the same time. However, it works correctly when all the resources are created at once. I have had issues in the past with AWS reporting an object as created but being unable to use it. Maybe there is a timing issue at play here as well.

[jen20]

Hi! Thanks for opening an issue about this. It looks like we have an eventual consistency issue to me - could you try adding a null resource with a local provisioner of sleep 5 into the dependency tree to isolate whether this is the case?

[f0rk]

Yes, it took a sleep 15, but that allowed the apply to proceed without error.

[AndreasFoetschl]

Ensure to set permissions in the AWS role:
AWSLambdaVPCAccessExecutionRole
AmazonVPCFullAccess

[bflad]

This particular IAM eventual consistency issue (Error modifying Lambda Function Configuration XXXXX: InvalidParameterValueException: The provided execution role does not have permissions) was previously fixed in #3116, released in version 1.8.0 of the AWS provider, and has been available in all releases since. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For anyone who thinks they've come across this in the future, please open a new issue with all the issue template details and we'll investigate further, thanks!

[avoidik]

I've faced the same issue as OP with the:

* provider.aws: version = "~> 2.3"

I have been using a custom IAM policy (not the built-in AWSLambdaBasicExecutionRole), and on a first attempt when I had started from scratch it has always failed with

* aws_lambda_function.lambda_function: Error modifying Lambda Function Configuration api: InvalidParameterValueException: The provided execution role does not have permissions to call CreateNetworkInterface on EC2

The following change has helped me:

resource "aws_lambda_function" "lambda_function" {
  ...
  depends_on = ["aws_iam_role_policy_attachment.lambda"]
}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!