"default region bug" in amazon s3 api results in broken createBucket api calls to S3 for us-east-1

[edwardotis]

Amazon has an undocumented quirk in their s3 api. (I call it a bug.)
Sending the default region in the s3 createbucket PUT request results in a generic LocationConstraint error.
This is extensively covered in a python aws sdk here:
boto/boto3#125

The Amazon documentation does not correctly document this quirk, so everyone has to find it by trial and error and googling. (I've requested that Amazon fix their docs.)
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html

In the meantime, forcing agorapulse.libs.awssdk.util.AwsClientUtil#DEFAULT_REGION to be added to PUT requests is breaking the calls to createBucket, and possibly to other api calls.
grails.plugin.awssdk.s3.AmazonS3Service#createBucket(java.lang.String, java.lang.String)

region = serviceConfig.region ?: config.region ?: AwsClientUtil.DEFAULT_REGION
The required workaround to the Amazon bug is to send no region when using 'us-east-1'.
The current version of this sdk does not support that option.

If I set the region to 'us-east-2', I can successfully create a bucket, and using a different region is the only programmatic workaround I know of for others using the grails aws sdk. (Of course, depending on your use case, you can simply create your buckets manually in us-east-1 via s3 web console.)

By using a different region, your S3 latency and costs will go up since you will have to now pay for data transfer between the default region and your workaround region.

I have not tested how much of the Amazon S3 API breaks when the default region is sent in the request. It could be a much bigger problem than just creating buckets.

Environment:
org.grails.plugins:aws-sdk-s3:2.1.9
ubuntu 14.04 LTS
java 8
grails 3.1.14

[benorama]

Thank you for the detailed issue.

As a workaround did you try to call the client directly?
amazonS3Service.client.createBucket(bucketName, '')

[edwardotis]

Ah, I was thinking AmazonS3Client client was package private, but that's Java, not Groovy. Thanks!

Actually, amazonS3Service.client.createBucket(bucketName, '') fails with exception:

ERROR StackTrace - Full Stack Trace:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: 8509B9CACA0AA3D2)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1586)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
        at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:998)
        at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:951)
        at com.amazonaws.services.s3.AmazonS3$createBucket$1.call(Unknown Source)

And this fails:
amazonS3Service.client.createBucket(bucketName, "us-east-1")
However, this works fine.
amazonS3Service.client.createBucket(bucketName)
And this works fine.
amazonS3Service.client.createBucket(bucketName, com.amazonaws.services.s3.model.Region.US_Standard)

and this is ok:
amazonS3Service.client.createBucket(bucketName, Region.US_Standard.toString())
but not this:
amazonS3Service.client.createBucket(bucketName, null)

One should note that Amazon put a hack into their Region enum to make this work. US_Standard evaluates to null in String form instead of 'us-east-1'. However the rest of the Regions
evaluate to their expected string representation, 'us-east-2', etc.

com.amazonaws.services.s3.model.Region

public enum Region {

  /**
     * The US Standard Amazon S3 Region. This region
     * uses Amazon S3 servers located in the United
     * States.
     * <p>
     * This is the default Amazon S3 Region. All requests sent to
     * <code>s3.amazonaws.com</code> go
     * to this region unless a location constraint is specified when creating a bucket.
     * The US Standard Region automatically places
     * data in either Amazon's east or west coast data centers depending on
     * which one provides the lowest latency.
     * </p>
     */
    US_Standard((String[])null),

    /**
     * The US-East-2 (Ohio) Region. This region
     * uses Amazon S3 servers located in Ohio.
     * <p>
     * When using buckets in this region, set the client
     * endpoint to <code>s3.us-east-2.amazonaws.com</code> on all requests to these buckets
     * to reduce any latency experienced after the first hour of
     * creating a bucket in this region.
     * </p>
     */
    US_East_2("us-east-2"),

....
}

So, the String, 'us-east-1', is not accepted by the Amazon S3 api.
Therefore, I would recommend removing the DEFAULT_REGION String from AwsClientUtil, and start passing around the
com.amazonaws.services.s3.model.Region enum instead.

class AwsClientUtil {

    static final String DEFAULT_REGION = 'us-east-1'
}

[benorama]

It should be fixed in 2.1.11.
I just checked: AwsClientUtil.DEFAULT_REGION is only used in S3 and the bucket creation was the only impacted by this.
Thanks for the report!