Terraform not using ECS Task IAM Role when retrieving modules from S3

[tomgoren]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.11.7

Affected Resource(s)

• https://www.terraform.io/docs/modules/sources.html#s3-bucket

Terraform Configuration Files

The failure occurs regardless of the bucket address syntax used:

module "consul" {
  source = "s3::https://s3-eu-east-1.amazonaws.com/example-bucket-1/example.zip"
}

module "consul" {
  source = "example-bucket-1.s3-eu-east-1.amazonaws.com/example.zip"
}

Debug Output

https://gist.github.com/tomgoren/3025fddff5e79574c92dc6ff1a9221db

Expected Behavior

Load the modules

Actual Behavior

Error loading modules: AccessDenied: Access Denied
	status code: 403, request id: 99D2E7EF11B32287, host id: vkAZbkzG7w+ir15EsMXrHZpOFzr4I6CUcvDzRixZyBVSRvmW3IfOiouRFy99wf3qZfCLYo+bizA=

Steps to Reproduce

1. terraform get (or init, or plan, or anything that requires pulling the modules from S3)

Important Factoids

We experience the problem when running in ECS in the context of a container that should use the IAM role associated with the Task, which has permissions to pull the file from the bucket.

Using awscli there are no issues doing the AssumeRole operation, that is:

bash-4.3# aws s3 ls s3://example-bucket-1/example.zip
2018-07-27 13:24:32     365196 example.zip

No issue.

Running terraform get will produce the error posted above (see attached Gist for full details).

However, if I manually retrieve the temporary IAM credentials produced by ECS, a la:

export AWS_ACCESS_KEY_ID=$(curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.SecretAccessKey')
export AWS_SESSION_TOKEN=$(curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.Token')

Now terraform get works as expected.

Thanks! I hope that this is enough information, please let me know if it's just me or if this is an actual bug!

[bflad]

Hi @tomgoren 👋 Sorry you're running into this trouble -- there is slightly awkward relationship between the AWS provider and Terraform core at the moment when it comes to the AWS SDK and the provider authentication chain. This issue might be fixed by upgrading those upstream dependencies (e.g. hashicorp/terraform#17901) and cutting a new Terraform core release, but I'm unsure if we'll be cutting a 0.11.8 release before 0.12.

If you can confirm that working with the provider in general outside your module fetching case above (e.g. AWS provider resources/data sources) works as expected from where you're running Terraform, then this should hopefully be fixed in Terraform 0.11.8 or 0.12.

[tomgoren]

Ah nice @bflad - thanks for the information! It was a bit crazy making initially, because this is a very sneaky bug, especially when hidden deep into a CI process (which makes for lengthy iterations).

Given that we have a workaround, we could probably wait until 0.12. That said - I think a lot of people are going to have to take their time in the transition (us included), due to the various upcoming backwards-incompatible changes. For that reason I believe a 0.11.8 would be prudent, especially if there are known solved outstanding, and potentially disruptive, bugs.

I appreciate your time! A lot of people are excited about 0.12!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!