terraform and vault: provider.aws: InvalidClientTokenId: The security token included in the request is invalid.

[FlorinAndrei]

Terraform Version

0.9.11

Affected Resource(s)

• aws_provider

Terraform Configuration Files

provider "vault" {
  address = "http://111.222.333.444:8200"
  skip_tls_verify = "true"
}

data "vault_generic_secret" "aws_iam_keys" {
  path = "aws/creds/admin"
}

provider "aws" {
  region = "${var.region}"
  access_key = "${data.vault_generic_secret.aws_iam_keys.data["access_key"]}"
  secret_key = "${data.vault_generic_secret.aws_iam_keys.data["secret_key"]}"

Debug Output

I am watching the traffic between Terraform and Vault with tcpdump, since it's plain text.
I see the AWS keys, generated by Vault, being returned to Terraform in JSON format.

GET /v1/aws/creds/admin HTTP/1.1
Host: 111.222.333.444:8200
User-Agent: Go-http-client/1.1
X-Vault-Token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Sat, 08 Jul 2017 00:48:22 GMT
Content-Length: 325
Connection: close

{"request_id":"3ea862c3-f46b-81e5-d954-e8aefa1a9a66","lease_id":"aws/creds/admin/23abf456-2cc6-1d7d-8eb9-f145fb7a9995","renewable":true,"lease_duration":60,"data":{"access_key":"AKIAJQT5JHHFVO6KZOLQ","secret_key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","security_token":null},"wrap_info":null,"warnings":null,"auth":null}

I know the AWS key generator in Vault works fine because I've tested it with the command line, and the keys it returns are working just fine. The keys are generated with admin privileges, for testing, and so they should be able to do anything.

Expected Behavior

terraform plan should succeed.

Actual Behavior

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.vault_generic_secret.aws_iam_keys: Refreshing state...
Releasing state lock. This may take a few moments...
Error refreshing state: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: f477482f-6379-11e7-8ecb-85e000cae410

Steps to Reproduce

1. terraform plan

[Yuxael]

Could this be the cause: hashicorp/terraform#2972 (comment) ?

[FlorinAndrei]

I strongly suspect now this is due to IAM keys being eventually consistent, but Vault doesn't wait or check. So probably not a Terraform bug. hashicorp/vault#3115

[lattwood]

This is actually due to the last credentials fetched from vault being used for interpolation.

[brainstorm]

I'm not using vault and I'm hitting this same problem. My access-key and secret are, for now, just hardcoded in vars.tf in the following example:

https://github.com/llevar/butler/blob/master/examples/deployment/aws/large-cluster/vars.tf

The actual error after substituting the access key and secret is:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------
Error running plan: 1 error(s) occurred:

* provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: 0623f5c1-a841-11e7-b989-edc4dee41040

The only "unusual characters" that my aws_secret_access_key contains are '+' signs and those credentials work fine with the regular, official, aws-cli client.

Furthermore, I just tried substituting those credentials for dummy test strings instead and the same error yields.

My current terraform version is:

$ terraform --version
Terraform v0.10.7

And the aws provider was downloaded via the usual terraform init command:

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.0.0)...
- Downloading plugin for provider "null" (1.0.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.0"
* provider.null: version = "~> 1.0"

Terraform has been successfully initialized!

[Yuxael]

@brainstorm This seems rather unrelated to this issue unless you actually used fresh new IAM keys.

[AnthonyWC]

I ran into this issue and resolved it with -var-file option that pointed to my tfvars file with credential. This issue does not trigger if one is using credentials with environment parameter.

hashicorp/terraform#2659
hashicorp/terraform#15894

[rberlind]

Hi @AnthonyWC : There are many ways to get AWS keys into a Terraform plan. The particular issue raised by @FlorinAndrei focuses on the issue that if you dynamically generated short-lived AWS credentials with Vault using Terraform's Vault Provider and vault_generic_secret data source to read from the AWS secrets backend in Vault, the AWS keys that were just created are not yet available from all AWS API endpoints. He provided a workaround in hashicorp/terraform#2972 (comment).

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!