IAM instance profile not created fast enough to modify EC2 instance

[hashibot]

This issue was originally opened by @b-dean as hashicorp/terraform#15199. It was migrated here as part of the provider split. The original body of the issue is below.

---

When creating an IAM instance profile and modifying an EC2 instance to use it, I get the error: Invalid IAM Instance Profile name

This seems to be the same sort of thing that was reported in hashicorp/terraform#1885 and fixed by hashicorp/terraform#2037. The difference being this is when an EC2 instance is modified, not when it is created.

Terraform Version

0.9.8

Affected Resource(s)

• aws_instance

Terraform Configuration Files

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type = "Service"
      identifiers = [
        "ec2.amazonaws.com",
      ]
    }
  }
}

data "aws_ami" "amazon" {
  most_recent = true
  filter {
    name   = "owner-alias"
    values = ["amazon"]
  }

  filter {
    name   = "name"
    values = ["amzn-ami-hvm-2017.03.*-x86_64-ebs"]
  }
}

/*
resource "aws_iam_role" "foo" {
  name = "foo"
  assume_role_policy = "${data.aws_iam_policy_document.assume_role.json}"
}

resource "aws_iam_instance_profile" "foo" {
  name = "foo"
  role = "${aws_iam_role.foo.name}"
}
*/

resource "aws_instance" "foo" {
  ami = "${data.aws_ami.amazon.id}"
  instance_type = "t2.nano"
  // iam_instance_profile = "${aws_iam_instance_profile.foo.name}"
}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply to create the EC2 instance
2. Edit the tf file to uncomment the aws_iam_role.foo, aws_iam_instance_profile.foo, and the iam_instance_profile argument on the aws_instance.foo resource
3. terraform apply to create the EC2 instance

Expected Behavior

The aws_iam_role and aws_iam_instance_profile should be created and the aws_instance modified to use the instance profile.

Actual Behavior

The aws_iam_role and aws_iam_instance_profile are created fine, but when the aws_instance is being modified the following error occurs:

Error applying plan:

1 error(s) occurred:

* aws_instance.foo: 1 error(s) occurred:

* aws_instance.foo: InvalidParameterValue: Value (foo) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
	status code: 400, request id: 9e5d217e-6b5d-41dc-a856-f2d5b31747a6

References

• Better check during planning of security group rules when large number of rules #1885
• Feature: Elastic Search now supports VPC, Subnets and Security Groups #2037

[thomasbiddle]

This is still an issue; thankfully was able to just re-run terraform apply and things worked on the second run.

[b-dean]

A similar problem was fixed for instance create in hashicorp/terraform#2037.

They really just need something similar to that solution, but for updating instances, somewhere in here.

[kaii-zen]

I'm suffering from this as well. I don't know why but it looks like it started happening more often than it used to.

[ctlajoie]

+1

[realflash]

Workaround:

  name  = "example"
  role = "${aws_iam_role.example_role.name}"
  provisioner "local-exec" {
      command = "sleep 40" # wait for instance profile to appear due to https://github.com/terraform-providers/terraform-provider-aws/issues/838
  }
}

[abinashmeher999]

Leaving it here in case if it works for someone else too

I think this issue occurs because when terraform tries to find aws_iam_instance_profile.example_profile.name it is already available to it locally from the config that we provide and it need not wait for the whole resource to actually appear before if tries to spawn ec2 with that profile name. This can be avoided if we explicitly use a computed value of aws_iam_instance_profile.example_profile like arn. That way terraform needs to fetch it before it can use it and it will only be available to terraform once creation is complete.
Adding this part in my aws_instance worked for me. Not sure if this will work for everybody.

provisioner "local-exec" {
  command = "echo ${aws_iam_instance_profile.example_profile.arn}"
}

Let me know if this works for anyone else too. Better than waiting for some random amount of time.

[yokzy88]

Does anybody else successfully use this trick?
I'm still failed and have to use the sleep 40.

resource "aws_iam_instance_profile" "example_profile" {
  name  = "example_profile"
  role = "${aws_iam_role.example_role.name}"
  provisioner "local-exec" {
     "echo ${aws_iam_instance_profile.example_profile.arn}"
  }
}

[yokzy88]

I even try it further by using depends_on + data source + null resource + printing all computed data.
Still have no luck, even though the log clearly displaying the null_resource echo before start modifying the instance.

data "aws_iam_instance_profile" "example_profile" {
  name = "example_profile"
}

resource "null_resource" "dummy" {
  provisioner "local-exec" {
    command = "echo ${data.aws_iam_instance_profile.example_profile.role_id}-${data.aws_iam_instance_profile.example_profile.arn}-${data.aws_iam_instance_profile.example_profile.create_date}-${data.aws_iam_instance_profile.example_profile.path}"
  }
}

resource "aws_instance" "example_instance {
 iam_instance_profile =  "${data.aws_iam_instance_profile.example_profile.name}"
 depends_on = ["data.aws_iam_instance_profile.example_profile","null_resource.dummy"]
}

[afeld]

Launch Configurations were having the same issue: hashicorp/terraform#5862

[afeld]

Official documentation about it: https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency

[milankubin]

Don't want to sound like a broken record as this has been pointed out in others threads, but 40s sleep time seems quite random, as AWS says there is a built in 2min delay.

https://forums.aws.amazon.com/thread.jspa?messageID=593651

[typicalaimster]

So far the only thing I've found to resolve this issue recently was to add a sleep as pointed out in this bug..

hashicorp/terraform#2349

Adding a sleep of about 10-15 seconds resolves the issue.

[bflad]

Hi folks! Sorry you have been having trouble with this. Usually these issues stem from the nature that many AWS services (especially IAM) are eventually consistent as noted above. We should be able to improve the provider code here to automatically retry up to two minutes here instead of just returning the error immediately.

Briefly looking at the aws_instance resource code, it looks like currently 30 seconds of retries are implemented during instance creation for this issue, but no retries are implemented during an update. I don't see an open PR right now, but can submit one shortly to fix this since its a small change.

[bflad]

This fix with an associated failing acceptance test has landed in master and will be released in v1.7.1 (most likely shipping tomorrow). Cheers! 🎉

[bflad]

This has been released in terraform-provider-aws version 1.7.1. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[josh-padnick]

I'm using the latest Terraform AWS provider (1.17.0) and I'm still getting this issue. Unfortunately, I'm using name-prefixes so my IAM Role has a dynamically generated name, which means I'm re-creating a different IAM Role name each time. I happen to be connecting from Phoenix, AZ to the eu-west-1 (Dublin) region so I suspect the higher latency is increasing the likelihood of this occurring for me.

I actually see my Launch Configuration wait for 2 minutes before throwing the error. Here's the exact error message.

* module.kafka.module.kafka_brokers.module.kafka_brokers.aws_launch_configuration.server_group: 1 error(s) occurred:

* aws_launch_configuration.server_group: Error creating launch configuration: ValidationError: Invalid IamInstanceProfile: server-group-20180504030238066800000001
	status code: 400, request id: 5e02b577-xxxx-xxxx-xxxxx-xxxxxxxxxxxx

[xelalexv]

I can confirm this problem with AWS provider version 1.14.1, trying to create launch configurations and instances. I'm in Munich and using eu-central-1, i.e. Frankfurt, so even with low latency this problem can come up.

[josh-padnick]

Just to give an update, we run automated tests on a module that threw this error. Those automated tests repeatedly passed, and then, while I was getting this error, they failed with the error in question. Now they pass again. So while this may well be an AWS provider issue, it seems to be triggered by some kind of transient AWS issue.

[bflad]

I would suggest opening a new issue with all the template details so we can further troubleshoot rather than commenting on a closed issue. The aws_launch_configuration resource already has 90 seconds of retries for ValidationError: Invalid IamInstanceProfile

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!