Support for AWS access via IAM - AssumeRole

[ncraike]

I get it's probably a lot of complexity to add, but it'd be great if Terraform supported using IAM roles in its AWS provider authentication.

This adds a few steps to how API credentials are used – you use your normal AWS access key and secret key to call sts assume-role in the API, and this responds with a temporary access key, secret key and session token which you use for the rest of the session. An example with AWS's own CLI is here.

The big use case for this is cross-account access. We have lots of AWS accounts to isolate billing to individual projects, and we link these to one parent account with AWS's consolidated billing. The cross-account roles let us have one set of IAM users in the parent account (eg one IAM user for each actual developer using AWS), and then just a single IAM role in each sub-account, which the developers can assume when needed. This saves having to create a new set of IAM users in each sub-account.

I imagine this would involve adding extra AWS provider config fields to describe the role to assume (using an AWS ARN like arn:aws:iam::account-of-role-to-assume:role/name-of-role), and would also involve changing the AWS API access layer to use the temporary role access/secret keys and a session token.

This should be linked to the AWS meta-issue. I felt the explanation was long enough to warrant a full issue.

[radeksimko]

I'm currently working on #1049 which is providing IAM support via the aws-sdk-go library, but it's not covering the assume-role use-case (yet?).

The only use-case for the IAM authorisation at the moment is to call the metadata API from inside a running EC2 instance: https://github.com/awslabs/aws-sdk-go/blob/440f9da54cf714e7c27b5b4ffa789793a5fbd658/aws/auth.go#L203-228

To fully understand the suggested solution, would you expect the provider config to have following options?

provider "aws" {
  credentials_provider = "iam"
  iam_role_arn = "arn:aws:iam::account-of-role-to-assume:role/name-of-role"
}

How do you expect to provide the initial pair of access_key & secret_key to call the STS API which will give you the token for further API requests? Statically? File config? ENV vars?

Related: http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-usingrole-switchapi.html

[radeksimko]

Also if you use MFA for the AssumeRole, it would require some other changes in the whole TF CLI workflow as well, e.g. Terraform interactively asking for the TOTP and providing it in that API request to STS.

[radeksimko]

btw. there's a lot more use-cases = workflows for this...

http://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-web-identity.html
http://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-saml.html

Ain't saying it should all be covered in the first implementation, just thinking loudly about it...

[ncraike]

Adding an iam_role_arn config is about as much as I think would be required, though maybe iam_role_to_assume or similar might be more meaningful and readable.

How do you expect to provide the initial pair of access_key & secret_key to call the STS API which will give you the token for further API requests? Statically? File config? ENV vars?

I would expect to provide them the same way access_key and secret_key are currently provided, as fields within the provider "aws" { block, as in the current docs.

I'm not sure if the credentials_provider = "iam" setting is needed, unless we're going to add a lot more providers.

[ncraike]

So, to clarify, I'd expect this to be enough for the cross-account assume-role use case:

provider "aws" {
    access_key = "${var.aws_access_key}"
    secret_key = "${var.aws_secret_key}"
    iam_role_to_assume = "arn:aws:iam::account-number:role/name-of-role"

    region = "us-east-1"
}

[ncraike]

I admit the other role workflows and MFA would add a lot of scope.

I'm primarily interested in this for the cross-account use case I described, because it seems to be the only method AWS supports for accessing a consolidated billing sub-account from the parent account, other than creating a new set of IAM users for each sub-account.

[ncraike]

Ah, looking at PR #1049, it does look like you've added support for more complex AWS credential provider configurations?

Sorry, I don't actually know the Terraform codebase or goamz, aws-go-sdk, etc very well. Currently I don't know Go (though Terraform may motivate me to learn), so my issues and comments are largely from the point-of-view of someone using Terraform and using AWS's own tools and API.

[willmcg]

Another use case for IAM roles is when you are running Terraform from an EC2 instance with an IAM role in its instance profile. In this case the temporary access credentials (access, secret & token) for the account can be obtained directly from the EC2 instance metadata. This relieves you from the need to manage these credentials manually if you are willing to run terraform in an EC2 instance and would make things very easy for the use case of deploying into the same account.

For the cross-account access you would only need to specify the IAM role to assume in the other account and Terraform would call AssumeRole using the credentials from the instance profile to get a new set of access credentials for the other account.

These temporary credentials from the instance profile can expire so some logic would need to be implemented to renew from the instance metadata when necessary, although it is unlikely that a terraform invocation would run longer than the minimum lifetime a set of credentials obtained from the instance metadata.

[radeksimko]

@willmcg

Another use case for IAM roles is when you are running Terraform from an EC2 instance with an IAM role in its instance profile.

Yes, that's already included in the linked #1049

For the cross-account access you would only need to specify the IAM role to assume in the other account and Terraform would call AssumeRole using the credentials from the instance profile to get a new set of access credentials for the other account.

Good point with cross-account access in general. I'd personally say that it's actually a reason to implement the whole assume-role logic separately in another PR as it's increasing the scope of the original PR quite significantly.

These temporary credentials from the instance profile can expire so some logic would need to be implemented to renew from the instance metadata when necessary.

The renewing logic is already part of the go library we use:
https://github.com/awslabs/aws-sdk-go/blob/440f9da54cf714e7c27b5b4ffa789793a5fbd658/aws/auth.go#L212-217

The expiration period though is unfortunately currently hardcoded. I will have a look if there's any way we can find the expiration period automatically, if not, I will probably expose it as another variable, that user will need to set or overwrite.

[willmcg]

Ah... thanks for the code pointers. Agreed on splitting to another PR... significant scope creep :-)

The expiration time is available in the EC2 metadata along with the temporary credentials:

http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE/Expiration

[radeksimko]

The expiration time is available in the EC2 metadata along with the temporary credentials:

👓 read the code more carefully, before you post, Radek... 😑

My bad, the hard-coded time period actually only applies to token provided from a config file:
https://github.com/awslabs/aws-sdk-go/blob/master/aws/auth.go#L130
there might be a way on how to track down the IAM policy which has the time period value, but it may not be that easy as the user calling the API may not have access to the actual IAM resource (can't see any details of policies which are affecting him).
More importantly even if we track it down, then we've got another much bigger problem ahead. These temporary credentials inside files are usually generated using another set of credentials (which only have privilege to generate tokens and not access any other APIs) and the generation process often includes MFA/SAML authorisation.

The IAM provider already reads the expiration period from the API:
https://github.com/awslabs/aws-sdk-go/blob/master/aws/auth.go#L250-259

All of the above means that for the basic implementation we're IMO good to go with #1049 😃

[radeksimko]

@ncraike would you mind modifying the issue title to "Support for AWS access via IAM - AssumeRole" or something along those lines?

[ncraike]

Done.

I feel I've opened a can of worms here, though.

[clstokes]

+1 as cross-account access is a desire of ours too. 😍

[ghost]

Some news about this feature?

[mbainter]

It's not as clean as a proper solution in terraform would be, but in the short term if you're using the fish shell you can use the credentials file and this function to handle setting up environment variables to authenticate with. It doesn't have support for using AssumeRole (yet) but you can change up the sts arguments pretty easily to manage that.

[mlrobinson]

While this is a little late for this ticket, here is the solution I worked out to work with AssumeRole for the time being: https://gist.github.com/mlrobinson/944fd0e2ad4926ba71c9.

It works with the latest version of terraform, and also with the aws cli tools as well as anything that respects the environment variables used here.

[joekhoobyar]

Cross-account deployments via IAM is definitely preferred over using API keys for our use cases. Any more news about this feature?

[MarsuperMammal]

So much this.. +1

[gjohnson]

+1

[outclassed]

+1

[SpencerBrown]

+1 yes right now it is not possible to set up AWS instances with Terraform and variabilize the account that they use to do the setup in any automated way. I tried for a day to make this work.

[apparentlymart]

Given the description of how this mechanism works from @ncraike in the initial issue description, I wonder if this could instead be implemented via a separate tool outside Terraform that would be able to wrap any command that uses the standard AWS environment variables. For example:

aws-with-assume-role {any-neccessary-arguments} terraform plan

Here's how I'd expect that to work:

• aws-with-assume-role looks in the environment or in standard AWS config files for a set of AWS credentials.
• It calls the AssumeRole API with those credentials and any necessary arguments to specify the role.
• If that call is successful, it gets back a new set of credentials.
• It launches the given command (terraform plan in this case) with the environment updated so that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set to the temporary credentials, rather than those that were originally provided.

Splitting the problem up into these two steps means that aws-with-assume-role would be usable with any tool that supports the de-facto standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, including terraform and packer and presumably the AWS CLI.

That might be a good thing to include in the AWS CLI. 😀

Not to say that Terraform couldn't have explicit support for this to make things simpler, but if such a tool were to exist it could be a reasonable alternative.

Edit: reading the thread again, I see that this is actually pretty similar to the script that @mlrobinson shared, except that script outputs environment variables to help you update your shell environment for future commands rather than just running one command as I'd suggested. Maybe that script could be adapted to support what I described above as an extra utility, but as-is it basically addresses this use-case.

[ghost]

It seems possible to variabilize aws instances with user_data field from terraform resources.

-- 
Julien Lavergne
+33670273883
http://julienlavergne.com
Sent with Airmail

On October 16, 2015 at 12:51:55 PM, Spencer Brown (notifications@github.com) wrote:

+1 yes right now it is not possible to set up AWS instances with Terraform and variabilize the account that they use to do the setup in any automated way. I tried for a day to make this work.

—
Reply to this email directly or view it on GitHub.

[akurilin]

As a temporary workaround, is there anything preventing us from wrapping terraform calls in a script that will use AWS cli to obtain temporary assumed role credentials, set those in the environment and then call terraform itself?

[ghost]

IMHO it’s just a trick. Terraform must be isolated from 3rd parties but custom providers.

-- 
Julien Lavergne
+33670273883
http://julienlavergne.com
Sent with Airmail

On October 31, 2015 at 4:58:45 AM, Alexandr Kurilin (notifications@github.com) wrote:

As a temporary workaround, is there anything preventing us from wrapping terraform calls in a script that will use AWS cli to obtain temporary assumed role credentials, set those in the environment and then call terraform itself?

—
Reply to this email directly or view it on GitHub.

[jfelixetcetera]

For now, I'm using https://github.com/jbuck/assume-aws-role to solve this problem, but it would really be great to be able to specify within a provider block.

With assume-aws-role, AWS_PROFILE={profile} assume-aws-role {role} opens a shell in which you can just run terraform ... without specifying any more credentialss. This does require your actual AWS credentials to come from the AWS CLI environment variables, and not Terraform, but is a good workaround until this issue is resolved.

[imduffy15]

The wrapper idea is a neat workaround, but how are people handling expiring API tokens mid terraform run?

[ozbillwang]

@imduffy15

Normally the STS token will be expired in 1 hour, should be enough for a terraform run.

[ghost]

That's a good question ...

Le 7 juil. 2016 à 02:27, Ian Duffy notifications@github.com a écrit :

The wrapper idea is a neat workaround, but how are people handling with expiring API tokens mid terraform run?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

Any implementation to come?

Le 7 juil. 2016 à 05:32, Bill Wang notifications@github.com a écrit :

@imduffy15

Normally the STS token will be expired in 1 hour, should be enough for a terraform run.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[netors]

+1

[ng-aaron]

Would love to have this as well.

[ghost]

Very soon?

Julien Lavergne
+33670273883
http://julienlavergne.com

Le 14 juil. 2016 à 18:07, Marduk notifications@github.com a écrit :

+1

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[ShwethaBallariMallappa]

Any updates on this issue ?

[ghost]

This issue is so important

Good implementation is needed

Le 21 juil. 2016 à 23:48, ShwethaBallariMallappa notifications@github.com a écrit :

Any updates on this issue ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[agarstang]

A solution to this is needed.

Writing a wrapper to populate environment variables (which would be the most elegant solution) is broken on instances with EC2 roles. See #7768.

[dbrandt]

Here's the wrapper I use it anyone find it useful: https://gist.github.com/dbrandt/b440f3be67897222cdc4650db510e262

[zoltrain]

I ported the aws-profile python script to pure bash if anyones interested. It caches the creds in the same file your CLI uses, so you only have to MFA once if it's enabled. It requires the python aws cli tools. https://github.com/zoltrain/aws-profile

[agarstang]

These wrappers will not work when the instance is using an EC2 IAM role (standard procedure when inside AWS) as it will always use the token from the EC2 IAM role (see #7768).

[drewsonne]

I think this change made upstream would address this problem. It looks like the upstream aws-go-sdk library now supports this as of version 1.3.0.

If I'm right that this addresses that problem, hopefully we'll see this being brought into terraform soon-ish then.

[BerndWessels]

Not sure if this is related:

I want to use the delegation set in Account A when creating a hosted zone in Account B.

Account A:

resource "aws_route53_delegation_set" "Platform" {
  reference_name = "${var.environment}"
}

Account B:

resource "aws_route53_zone" "Product" {
  name = "${var.product_domain}"
  delegation_set_id = "the_id_of_the_delegationset_in_account_a"
}

Would any of the suggestions in this issue allow me to achieve this?

At the moment I get the following error when trying to apply the terraform script for Account B:

Error applying plan:
1 error(s) occurred:
* aws_route53_zone.Product: AccessDenied: User: arn:aws:iam::123456789012:user/terraform is not authorized to access this resource
        status code: 403, request id: 6cc22c52-64bd-11e6-bf31-a1ca9bc0fefa

[johnrengelman]

That error looks like the user you are running terraform with for account B doesn't have access to create route53 zones.

[imduffy15]

Added a potential solution for this over in the PR at #8506

It adds support for specifying a roleArn in the provider block.

It will use any valid credentials it finds in the credentials chain to execute the assume role.

I'm a Golang newbie so would love people to jump in to get it across the line and merged.

[ghost]

When will this issue be resolved?

Julien Lavergne
+33670273883
https://julienlavergne.com

On 31 Aug 2016, at 14:37, Ian Duffy notifications@github.com wrote:

Added a potential solution for this over in the PR at #8506

It adds support for specifying a roleArn in the provider block.

It will use any valid credentials it finds in the credentials chain to execute the assume role.

I'm a Golang newbie so would love people to jump in to get it across the line and merged.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[ketzacoatl]

@julienlavergne, it'll be resolved when there's resolution.

[ghost]

There is always a solution. Any AWS engineer wants to help us?

Julien Lavergne
+33670273883
https://julienlavergne.com

On 31 Aug 2016, at 15:04, ketzacoatl notifications@github.com wrote:

@julienlavergne, it'll be resolved when there's resolution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[iroller]

Hey, @jen20 and @imduffy15. Is it supposed to work in latest master?
Thanks for the work!

 provider "aws" {
   region  = "${var.aws_region}"
   profile = "${var.aws_profile}"
+
+  assume_role {
+    role_arn     = "arn:aws:iam::<123>:role/<name>"
+  }
 }

$ terraform plan
2016/09/03 15:50:59 [INFO] Terraform version: 0.7.3 dev b02041062f605002c6605d9656f7dc58ce8a1ae1+CHANGES

Errors:
  * provider.aws: : invalid or unknown key: assume_role

[jen20]

Hi @iroller! I just verified with a build of master (09d6d2c) and verified all is well. To build, I just ran make dev on master, and made sure that there were no terraform-* binaries on PATH (from provider overrides).

[iroller]

I did @jen20. Not sure why is it complaining. With TF_LOG=debug its displaying the proper (master) terraform version at the beginning of the run.

Quick question - support for mfa wasn't implemented in that PR, right?

[grahamjenson]

We just released a tool assume-role (https://github.com/coinbase/assume-role) that can export the credentials without using a profile. This means we don't have to use assume_role on the AWS provider. Although this issue is closed, this is how we manage dozens of accounts and roles that have MFA and temporary credentials.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.