aws_s3_bucket_object update fails with Access Denied

[ansoni]

Using AWS Provider - v2.29.0, aws_s3_bucket_object update fails with the following error:

aws_s3_bucket_object.object-python (destroy): 1 error(s) occurred:

• aws_s3_bucket_object.object-python: error deleting S3 Bucket (...) Object (....zip): AccessDenied: Access Denied

Ran with a DEBUG and found that it was this API call that was giving grief.

2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ REQUEST POST-SIGN ]----------------------------- 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: GET /?prefix=....zip&versions= HTTP/1.1 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Host: ....s3.us-west-2.amazonaws.com 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: User-Agent: aws-sdk-go/1.23.15 (go1.12.6; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Authorization: ... 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Date: 20190921T172949Z 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Accept-Encoding: gzip 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019-09-21T17:29:49.979Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ----------------------------------------------------- 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/09/21 17:29:50 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjectVersions Details: 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ RESPONSE ]-------------------------------------- 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: HTTP/1.1 403 Forbidden 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Connection: close 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Transfer-Encoding: chunked 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: application/xml 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Date: Sat, 21 Sep 2019 17:29:49 GMT 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Server: AmazonS3 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Id-2: ... 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Request-Id: ... 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ----------------------------------------------------- 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/09/21 17:29:50 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?> 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>D77F21A2E6CDFB19</RequestId><HostId>dEcbNzMwmSpf368ACe4n1r5bw8dvs3pDh/DYM4F8VbmdU6UOjx1XGH8RkYQ7CYuaXHgdq5P+GGk=</HostId></Error> 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/09/21 17:29:50 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjectVersions failed, attempt 0/25, error AccessDenied: Access Denied 2019-09-21T17:29:50.131Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: status code: 403, request id: ..., host id: dEcbNzMwmSpf368ACe4n1r5bw8dvs3pDh/DYM4F8VbmdU6UOjx1XGH8RkYQ7CYuaXHgdq5P+GGk= 2019/09/21 17:29:50 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

Locked my provider version to 2.28.1 and the error goes away, object is updated successfully. I'm assuming that we changed some flow here. This terraform aws_s3_bucket_object resource has been deployed and updated for over a year at this point.

[ansoni]

Looking at #9942, it appears you accidentally made this resource only work for versioned buckets:

https://github.com/terraform-providers/terraform-provider-aws/pull/9942/files#diff-d7a339333642c37555f0eee5057d31cbR490

We use to utilized the version_id resource attribute for determining if we needed to do a version delete or just delete the object. Now that statement is gone and my object is not versioned which is most likely why the API call fails (permissions are there).

[nywilken]

@ansoni thanks for catching this issue and calling out the possible culprit. We are working on getting things fixed. Please keep an eye on the thread for updates.

For folks running into this same issue please lock the provider version to 2.28.1 until a fixed is released.

provider "aws" {
  ...
  version = "2.28.1"
}

Cheers

[nywilken]

@ansoni I just wanted to follow up with the issue you are seeing. You are correct that the updated resource now checks if an object has a version; always calling ListObjectVersions regardless of version_id id being set. But in my testing I find the call, assuming the proper permissions are in place, to work for both versioned and non-versioned buckets.

To help rule out a permissions issue. Can you confirm that the credentials you are using for Terraform have the appropriate Read permissions (i.e ListBucketVersions and GetObjectVersion*)?

In creating a restricted user with no perms for obtaining object versions I am able to reproduce the issue you are seeing within our acceptance testing. But I do see a few references to handling an "Access Denied" error message within the new code base so I don't want to rule out other possible culprits.

Thanks for the extra info and help in solving this problem.

[eedwards-sk]

@nywilken

You are correct that the updated resource now checks if an object has a version; always calling ListObjectVersions regardless of version_id id being set

Maybe I'm confused, but that is the regression -- I should NOT have to give a user more permissions than it used to.

[nywilken]

@eedwards-sk nah you are right. I read too much into the comment "Now that statement is gone and my object is not versioned which is most likely why the API call fails (permissions are there)." and went looking into an issue with errors around getting version information. I marked this as a regression and will work on making sure that things work as expected. Thanks for clarifying.

[bflad]

The fix for this has been merged and will release with version 2.31.0 of the Terraform AWS Provider, tomorrow. 👍

[ghost]

This has been released in version 2.31.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!