Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 backend: Support policy ARNs argument for assume role configuration #22601

Closed
timoguin opened this issue Aug 26, 2019 · 5 comments · Fixed by #25134
Closed

S3 backend: Support policy ARNs argument for assume role configuration #22601

timoguin opened this issue Aug 26, 2019 · 5 comments · Fixed by #25134
Assignees
@bflad
Labels
backend/s3 enhancement
Milestone
v0.13.0

Comments

@timoguin
Copy link

The assume_role configuration block for the S3 backend supports passing in a scoped down policy for the role session via the assume_role_policy argument.

In addition to the Policy parameter that passes a single inline policy, the STS API also supports a PolicyArns parameter that allows passing up to 10 policy ARNs for the role session.

I propose adding an assume_role_policy_arns argument to both the S3 backend and the Terraform remote state data source. This would allow more flexibility in scoping down role sessions.

The existing support for passing a JSON string provides no validation and swallows errors with a generic AccessDenied, so it can be difficult to debug if issues are actually due to permissions or if there is a simple syntax/validation issue with the passed policy. Alternately, specifying policy ARNs allows that policy validation to happen in another context.

References:
@teamterraform
Copy link
Contributor

Thanks for sharing this use-case, @timoguin!

The S3 backend builds on the authentication mechanisms supported by the AWS provider, so if the AWS provider doesn't already offer the feature you're requesting, it'd be better to open a Feature Request in the AWS provider and then once it's implemented over there we can bring that same functionality into the S3 backend.

@timoguin timoguin mentioned this issue Aug 27, 2019
Closed
@timoguin
Copy link
Author

timoguin commented Aug 27, 2019 •
edited
Loading

Thanks. It looks like the S3 backend and the AWS provider actually use the shared aws-sdk-go-base library. I've opened an issue there, along with one for the AWS provider.

Issues:

@timoguin timoguin mentioned this issue Aug 27, 2019
Closed
@timoguin
Copy link
Author

timoguin commented Aug 28, 2019 •
edited
Loading

Unfortunately, this isn't supported yet by the AWS Go SDK. I created an issue: aws/aws-sdk-go#2794.

Edit to the above message: Support in the Go SDK is present. See my comment on a related issue for the aws-go-sdk-base for more detail: hashicorp/aws-sdk-go-base#11 (comment)

@bflad bflad self-assigned this Jun 2, 2020
@bflad bflad added this to the v0.13.0 milestone Jun 2, 2020
bflad added a commit that referenced this issue Jun 4, 2020
@bflad
backend/s3: Updates for Terraform v0.13.0
2c3b0fa 
Reference: #13410
Reference: #18774
Reference: #19482
Reference: #20062
Reference: #20599
Reference: #22103
Reference: #22161
Reference: #22601
Reference: #22992
Reference: #24252
Reference: #24253
Reference: #24480
Reference: #25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
@bflad bflad mentioned this issue Jun 4, 2020
Merged
bflad added a commit that referenced this issue Jun 5, 2020
@bflad
backend/s3: Updates for Terraform v0.13.0 (#25134)
ba081aa 
* deps: Update github.com/hashicorp/aws-sdk-go-base@v0.5.0

Updated via:

```
$ go get github.com/hashicorp/aws-sdk-go-base@v0.5.0
$ go mod tidy
$ go mod vendor
```

* backend/s3: Updates for Terraform v0.13.0

Reference: #13410
Reference: #18774
Reference: #19482
Reference: #20062
Reference: #20599
Reference: #22103
Reference: #22161
Reference: #22601
Reference: #22992
Reference: #24252
Reference: #24253
Reference: #24480
Reference: #25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
@bflad
Copy link
Contributor

bflad commented Jun 5, 2020

Support for this enhancement has been merged for the S3 Backend and will release with version 0.13.0-beta2 of Terraform.

mildwonkey pushed a commit that referenced this issue Jun 12, 2020
@bflad @mildwonkey
backend/s3: Updates for Terraform v0.13.0 (#25134)
8f430b4 
* deps: Update github.com/hashicorp/aws-sdk-go-base@v0.5.0

Updated via:

```
$ go get github.com/hashicorp/aws-sdk-go-base@v0.5.0
$ go mod tidy
$ go mod vendor
```

* backend/s3: Updates for Terraform v0.13.0

Reference: #13410
Reference: #18774
Reference: #19482
Reference: #20062
Reference: #20599
Reference: #22103
Reference: #22161
Reference: #22601
Reference: #22992
Reference: #24252
Reference: #24253
Reference: #24480
Reference: #25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
@ghost
Copy link

ghost commented Jul 6, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Jul 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bflad bflad

Labels
backend/s3 enhancement
Projects
None yet
Milestone
v0.13.0
Development

Successfully merging a pull request may close this issue.

backend/s3: Updates for Terraform v0.13.0 hashicorp/terraform
4 participants
@bflad @timoguin @hashibot @teamterraform