Add S3 endpoint override ability and expose S3 path style option

[renier]

Add S3 endpoint override ability and expose S3 path style option

• Overriding S3 endpoint - Enable specifying your own S3 api endpoint to override the default one, under endpoints.
• Force S3 path style - Expose this option from the aws-sdk-go configuration to the provider.

Sample provider config:

provider "aws" {
  region = "us-east-1"
  s3_force_path_style = true
  endpoints {
    s3 = "https://1.2.3.4"
  }
}

[radeksimko]

Hi @renier
thank you for the PR.

Would you mind separating this into multiple PRs to make review a bit easier?
I'd appreciate if this was at least in 2 pieces - S3 related and AWS auth related PR.

I'd like to have a separate discussion about the authentication. It looks like #6533 would be more hammer approach/solution to the same problem, what do you think?

Thanks.

[renier]

@radeksimko Sure. It's all in the same PR now because without skipping iam/sts validation, using a third-party implementation of S3 won't work.

I'll create two branches. First the one with the auth options, then the one that adds the S3 options.

About the previous pull request 6533, I think that boolean options should always default to false as a standard (I see them as off switches) and the name of the option should reflect that end of the polarity. So instead of validate = true (default), skip_validation = false (default) feels better. Then you turn on the switches you want with true values.

[renier]

@radeksimko Branch updated. The other PR for the auth stuff is #7874 .

[renier]

@radeksimko This will need a rebase once #8114 is merged, but aside from that, do you have any comments?

[radeksimko]

@renier Aside from rebase / resolving conflicts this looks pretty good.

Do you know how alternative AWS S3 API implementations deal with things like bucket website URLs? Is it possible to expose the bucket as website at all?

At this point it looks like the URL would be always constructed as ...%s.amazonaws.com which is probably not what you'd want if you defined custom S3 endpoint.

Not that I'm saying this is a blocker for the PR, but (1) I'm curious (2) somebody may rise that as an issue in the future.

[renier]

@radeksimko Rebase done.

I don't know for sure if the current implementation I'm using supports bucket websites. Currently, it's necessary for me to force S3 path style to read and access the bucket.

[radeksimko]

👍 Thanks.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.