add validate_credentials option to provider-aws

[iamatypeofwalrus]

My dev team is planning to use Terraform to manage our AWS resources in all of our environments (Dev, Test, and Prod).

We develop 100% locally (fake_sqs and DynamoDB Local) and do not have access to AWS credentials. We'd like to use Terraform to stand up tables against DynamoDB Local (or any other local AWS service).

I added a validate_credentials option to the provider-aws Schema. It defaults to true. When set to false the credentials are not validated at all against IAM or the white / black list of account ids.

Since DynamoDB Local does not check the validity of the credentials we can create tables against it.

[radeksimko]

Hi @iamatypeofwalrus
What you want to achieve most likely makes sense. You made me think about this problem in wider context, see #6535

I'm wondering what's going to be the user experience for other non-DynamoDB resources which don't have custom endpoints defined (hence Terraform won't be able to create these). Will user see any meaningful error message in such cases?

Does your infrastructure only consist of DynamoDB + SQS + compute? or how do you deal with other services which are not fake-able? Are you somehow communicating to developers which AWS services and resources are "supported" in local environments?

---

I would personally hold off on reviewing & merging this until we actually make the credentials validation work for STS credentials - see #6523 (comment)

I'd like to get that ^ done as soon as possible as it seems to be the only blocker for STS users, then I'm happy to refocus on the use case with alternative AWS service providers and revisit this PR more thoroughly.

[iamatypeofwalrus]

Thanks for the reply!

Will user see any meaningful error message in such cases?

If users disable validate_credentials and they attempt to CRUD resources with fake credentials against real AWS they will see authentication or authorization errors. The errors that I receive when I attempt to create a DynamDB table with fake credentials is:

* InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: ...

I think this error is pretty clear in conjunction with a user having explicitly set validate_credentials to false. That said the provider could always be more helpful in the case that a CRUD operation fails by providing a hint like:

Is validate_credentials set to false?

Does your infrastructure only consist of DynamoDB + SQS + compute?

Yes, the application itself is that simple. However, in order to be considered production ready we need to manage metrics, alarms, and paging. I created an internal terraform provider to CRUD those. We will need real credentials for request signing in order to manage those resources.

Are you somehow communicating to developers which AWS services and resources are "supported" in local environments?

My thoughts for managing those resources is to structure an infrastructure directory like so:

infrastructure/
  README.md
  development.tfvars
  dynamodb/
    dynamodb.tf
  sqs/
    sqs.tf
  monitors/
    monitors.tf

The README would document which resources are needed in which environment. monitors/ are Prod only.

A developer wishing to create or update dynamo tables in their sandbox would do:

terraform plan -var-file=development.tfvars dynamodb/

For managing production resources we would need to grab credentials from our credential management system.

Ultimately, there may be a set of simple shell scripts for grabbing the correct credentials, setting the Terraform vars, and calling terraform.

[iamatypeofwalrus]

@radeksimko, has there been any movement on this PR? Happy to push this design forward and search for an alternative with some more feedback.

[radeksimko]

@iamatypeofwalrus Thanks again for bringing this use case to my attention.

I think I'm most likely going to merge #7874 in favour of this PR as it has more granular options (separate option for metadata API) which in fact may/should be even more granular. I'm keeping it open just until the mentioned PR is actually merged.

I'd be happy to merge a separate PR for the typo in docs ((Optional) Optional)) straight away. 😉

[iamatypeofwalrus]

@radeksimko just happy to see progress on this :-)

can't wait for #7874 to get merged! it solve a real pain point for me.

[radeksimko]

Closed in favour of #7874

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.