[version 1.42] No valid credential sources found for AWS Provider.

[shifres]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.10
+ provider.aws v1.42.0
+ provider.template v1.0.0

Affected Resource(s)

• provider.aws

Terraform Configuration Files

provider "aws" {
  region = "${var.region}"
}

Output

terraform plan
.
.
.
module.bla.bla.bla.provider.aws: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

Expected Behavior

Shouldn't get No valid credential sources found for AWS Provider. error

Steps to Reproduce

run terraform init --reconfigure to download the latest provider.aws version: provider.aws: version = "~> 1.42"

1. terraform plan

Important Factoids

I'm running terraform locally and using a ~/.aws/credentials file.
When I lock the aws.provider version to 1.41 everything runs properly.

provider "aws" {
  version = "1.41"
  region = "${var.region}"
}

[bflad]

Hi @shifres 👋 Sorry you're running into trouble here.

It might be very helpful to turn on Terraform debug logging in your scenario (e.g. TF_LOG=debug terraform plan) to see if the underlying error(s) cannot be found instead of our less helpful error here.

There wasn't any intended changes to the provider and its authentication in this previous release so we may want to check the upstream AWS Go SDK changes to see if something occurred there as well. This one looks suspicious, but would also be very environment specific (two // in credentials): aws/aws-sdk-go#2239

[shifres]

Thanks for the quick response @bflad
This is the relevant output from the debug:

2018/11/01 13:09:33 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
.
.
.
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] <GetUserResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <GetUserResult>
    <User>
      <Path>/</Path>
      <UserName>XXXXXXX</UserName>
      <Arn>arn:aws:iam::<ACCOUNT_NUMBER>:user/XXXXXXX</Arn>
      <UserId>XXXXXXXXXXXX</UserId>
      <CreateDate>2018-10-23T21:33:39Z</CreateDate>
    </User>
  </GetUserResult>
  <ResponseMetadata>
    <RequestId>e71e4ac9-ddf8-11e8-89ee-df548ea8d37e</RequestId>
  </ResponseMetadata>
</GetUserResponse>
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAccountAttributes Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.14.31 (go1.11.1; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10
Content-Length: 87
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXX/20181101/us-east-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=e4b3b0bdd2d97fa99385488ac9dc06e730ad3e31e32a1cd5fa60551fa5f7d06e
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20181101T170933Z
Accept-Encoding: gzip

Action=DescribeAccountAttributes&AttributeName.1=supported-platforms&Version=2016-11-15
-----------------------------------------------------
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 403
Content-Type: text/xml
Date: Thu, 01 Nov 2018 17:09:32 GMT
X-Amzn-Requestid: e7280ea6-ddf8-11e8-b3fc-3ba677e6daf4
.
.
.
Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2018-11-01T13:09:33.005-0400 [DEBUG] plugin.terraform-provider-aws_v1.42.0_x4: 2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalConfigProvider, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalOpFilter, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.certificates_alternative_names: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018-11-01T13:09:33.021-0400 [DEBUG] plugin.terraform-provider-aws_v1.42.0_x4: 2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalConfigProvider, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalOpFilter, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [ERROR] root.spill.elb: eval: *terraform.EvalSequence, err: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider
2018/11/01 13:09:33 [TRACE] [walkRefresh] Exiting eval tree: module.spill.module.elb.provider.aws
2018/11/01 13:09:33 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/11/01 13:09:33 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2018/11/01 13:09:33 [INFO] Initializing DeviceFarm SDK connection
2018/11/01 13:09:33 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:

Let me know if there's anything further I can do in order to assist.

[tdewitt]

I can reproduce with 1.42.0 and it works as expected using 1.41.0. I'm running Linux. Coworkers using Apple laptops do not experience the same problem.

[shifres]

@tdewitt , can you reproduce the problem or you don't experience any problems with version 1.42 ?
I am using Apple btw.

[tdewitt]

• On my Linux laptop, using version 1.42.0 of the aws provider plugin, I can reproduce.
• On my Linux laptop, using version 1.41.0 of the aws provider plugin, I cannot reproduce and things work as expected.
• A coworker, on his Apple laptop, using version 1.42.0 of the aws provider plugin, he cannot reproduce.

For reference, this is how the provider block is defined:

terraform {                                                                     
  required_version = ">= 0.11.0"

  backend "s3" {
    encrypt = "true"
    acl     = "bucket-owner-full-control"
  }
}

/**
 * Provider
 */

provider "aws" {
  version = "~> 1.3"
  region  = "${var.aws_region}"

  assume_role {
    role_arn = "${lookup(var.assume_roles, var.aws_account_alias)}"
  }
}

• I changed the version to <= 1.41.0 to get working again
• We use role assumption for everything

~/.aws/config:

[default]                                                                       
output=json

[s3]
calling_format = boto.s3.connection.OrdinaryCallingFormat

~/.aws/credentials:

[name]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED # does have a / in the string

[name::dev]
source_profile = name
role_arn = arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME

[bflad]

It'll be super useful for this issue to know how credentials are being sourced (e.g. environment variables, ~/.aws/config, `~/.aws/credentials) -- thanks for sharing that information, @tdewitt!

Out of curiosity, does it help to wrap your secret access key in quotes?

[name]
aws_access_key_id = REDACTED
aws_secret_access_key = "REDACTED" # does have a / in the string

[tdewitt]

This is my workflow:

$ awsenv name
AWS environment is name
AWS_PROFILE=name
AWS_DEFAULT_REGION=us-west-2

# change to template dir
$ terraform init -backend-config=role_arn=arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME -backend-config region=us-west-2 -backend-config bucket=company-us-west-2-remote-state-develop -backend-config key=base/terraform.tfstate -backend-config dynamodb_table=us-west-2-remote-state-develop -lock=false

$ terraform plan -var-file ../../vars/variables.tfvars -var-file ../../vars/develop/shared.tfvars -var-file ../../vars/develop/default/shared.tfvars -var-file ../../vars/develop/default/us-west-2.tfvars -out out.us-west-2.tfstate

$ terraform apply out.us-west-2.tfstate

I just verified with the coworker who isn't having a problem (using an Apple) that he also has a forward slash in his secret key. Quoting the secret key doesn't change the behavior (still fails on my laptop).

[bflad]

Just providing for reference that we have heard about some odd behavior with secret access keys that contain /: #663 (comment)

Antidotally, I've heard a case internally about this as well, but since there was an easy workaround (rotating your credentials) I never personally spent any time diving further into it. 😅

[shifres]

I've changed the AWS_PROFILE env var to credentials that don't contain any special characters (like /, + etc...) and I still encounter the same problem.
I have no problem reading from the TF s3 backend (which requires valid credentials) but still encounter the error: No valid credential sources found for AWS Provider for the AWS provider.
My ~/.aws/config file contains:

[default]
output = json
region = us-east-1

and my ~/.aws/credentials file format is:

[name]
aws_access_key_id=<KEY>
aws_secret_access_key=<SECRET>

[jhumbug]

Getting the error as well for 1.42

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: Error creating AWS session: SharedConfigLoadError: failed to load config file, /Users/****/.aws/credentials
caused by: INIParseError: invalid state with ASTKind {section_stmt {1 STRING 0 [97 119 115 45 97 100 117 108 116 115 119 105 109]} true []} and TokenType {4 NONE 0 [58]}

Manually setting the version to 1.41 works.

[shifres]

I see similar error to @jhumbug when I setup AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

[bflad]

I think TokenType {4 NONE 0 [58]} means its getting stuck on a : character. You could try wrapping that with quotes. Here's another bug reported upstream: aws/aws-sdk-go#2246

[br4nd0n]

I'm running this on a Mac laptop, high sierra 10.13.6, and I can reproduce this with AWS provider 1.42. AWS provider 1.41 works fine for me.

[jritsema]

@bflad in case it's helpful in resolving. We use https://github.com/turnerlabs/samlkeygen which populates our credentials file with temporary keys. For example:

[account1:devops]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED
aws_session_token = REDACTED
aws_security_token = REDACTED
last_updated = 2018-11-02T13:48:32Z
expiration = 2018-11-02T22:48:32Z

[mjreed-wbd]

Based on the comments, the closed upstream bug only resolves the issue for colons in section headers, not in values. But creds file values can contain colons - not only timestamps like the ones above from samlkeygen, but also role ARNs (which show up when you use a credentials profile to assume a role from another profile, as in @tedewitt's post above).

[jritsema]

aws/aws-sdk-go#2247 (pending release) seems to fix our issue. Hopefully that makes it into a v1.42.1 or v1.43.0 release.

[bflad]

Version 1.43.0 of the AWS provider will include AWS Go SDK v1.15.68+, which contains both upstream INI parsing fixes. If there are still other problems with the parser, please continue to file them upstream. 👍

We're still tracking to release on our "normal" cadence middle of next week, but for visibility I'm actually going to keep this issue open until then even though the fixes may already be merged in case other issues emerge.

[jritsema]

Thanks @bflad for turning this around so quickly! And for all the awesome work you guys do!!!

[bflad]

This has been released in version 1.43.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[alienth]

@bflad Still encountering this on 1.43.0. I have to pin to 1.41.0 to get my TF stuff to work. Symptom is the exact same:

* provider.aws: No valid credential sources found for AWS Provider.

My AWS credentials file contains : in values due to expiry time. Nothing else special going on there.

[bflad]

@alienth I would suggest opening a SDK issue upstream (similar to aws/aws-sdk-go#2246) and a new tracking issue here as well. 👍

[bflad]

FYI, just locking this issue to encourage fresh bug reports if there are any more. 😄