r/aws_s3_bucket: Support S3 Cross-Region Replication filtering based on S3 object tags

[ewbankkit]

Just announced S3 Cross-Region Replication filtering based on S3 object tags.
Add support to aws_s3_bucket resource's replication_configuration attribute.

[ewbankkit]

Requires:

• vendor: aws/aws-sdk-go@v1.15.38 #5936

[bflad]

Exciting to see this added to the API. I will see about getting #3577 merged since it will be in the same code neighborhood and looks okay at first glance.

[ewbankkit]

@bflad Could someone also take a look at #5447? I think it is in the same code neighborhood also.

[ewbankkit]

Proposed syntax:

• Add tags {} to the replication replication_configuration.rules attribute:

resource "aws_s3_bucket" "bucket" {
  bucket   = "tf-test-bucket-12345"

  replication_configuration {
    rules {
      id     = "foobar"
      priority = 123
      prefix = "foo"
      tags {
          ReplicateMe = "Yes"
      }
      status = "Enabled"

      destination {
        bucket        = "${aws_s3_bucket.destination.arn}"
        storage_class = "STANDARD"
      }
    }
  }
}

[ewbankkit]

I if I create 4 rules through the AWS console

• rule1 applies to the whole bucket
• rule2 applies just to the prefix rule2-prefix
• rule3 applies just to objects tagged with Rule3Tag=V
• rule4 applies to the prefix rule4-prefix and objects tagged with Rule4Tag_1=V and Rule4Tag_2=VV

$ aws s3api get-bucket-replication --bucket ewbankkit-us-east-1
{
    "ReplicationConfiguration": {
        "Rules": [
            {
                "Status": "Enabled", 
                "Filter": {
                    "And": {
                        "Prefix": "rule4-prefix", 
                        "Tags": [
                            {
                                "Value": "VV", 
                                "Key": "Rule4Tag_2"
                            }, 
                            {
                                "Value": "V", 
                                "Key": "Rule4Tag_1"
                            }
                        ]
                    }
                }, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 4, 
                "ID": "rule4"
            }, 
            {
                "Status": "Enabled", 
                "Filter": {
                    "Tag": {
                        "Value": "V", 
                        "Key": "Rule3Tag"
                    }
                }, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 3, 
                "ID": "rule3"
            }, 
            {
                "Status": "Enabled", 
                "Filter": {
                    "Prefix": "rule2-prefix"
                }, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 2, 
                "ID": "rule2"
            }, 
            {
                "Status": "Enabled", 
                "Filter": {}, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 1, 
                "ID": "rule1"
            }
        ], 
        "Role": "arn:aws:iam::000000000000:role/service-role/r"
    }
}

If I add the equivalent of rule2 with the current provider:

resource "aws_s3_bucket" "e" {
  bucket        = "ewbankkit-us-east-1"
  region        = "us-east-1"
  acl           = "private"
  force_destroy = false

  replication_configuration {
    role = "arn:aws:iam::000000000000:role/service-role/r"

    rules {
      id     = "rule2"
      status = "Enabled"
      prefix = "rule2-prefix"

      destination {
        bucket = "arn:aws:s3:::ewbankkit-us-west-2"
      }
    }
  }
}

then

$ aws s3api get-bucket-replication --bucket ewbankkit-us-east-1
{
    "ReplicationConfiguration": {
        "Rules": [
            {
                "Status": "Enabled", 
                "Prefix": "rule2-prefix", 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "ID": "rule2"
            }
        ], 
        "Role": "arn:aws:iam::000000000000:role/service-role/r"
    }
}

so it looks like we have to deal with ReplicationConfiguration.Rules[*].Prefix and ReplicationConfiguration.Rules[*].Filter.Prefix being equivalent.

If I apply the TF configuration above and then go to add a new rule via the AWS console:

More details:

• https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-crr.html#enable-crr-add-rule
• https://docs.aws.amazon.com/AmazonS3/latest/dev/crr-add-config.html#crr-backward-compat-considerations

[ewbankkit]

It also looks like we'll need to add support for the Delete Marker Replication functionality:

When you delete an object from your source bucket without specifying an object version ID, Amazon S3 adds a delete marker. If you use V1 of the replication configuration XML, Amazon S3 replicates delete markers that resulted from user actions. In other words, if the user deleted the object, and not if Amazon S3 deleted it because the object expired as part of lifecycle action. In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled.

So we'll have an additional optional delete_marker_replication attribute in the rule.

[ewbankkit]

resource "aws_s3_bucket" "e" {
  bucket        = "ewbankkit-us-east-1"
  region        = "us-east-1"
  acl           = "private"
  force_destroy = false

  replication_configuration {
    role = "arn:aws:iam::000000000000:role/service-role/r"

    rules {
      id     = "rule1"
      status = "Enabled"
      prefix = "rule1-prefix"

      destination {
        bucket = "arn:aws:s3:::ewbankkit-us-west-2"
      }
    }

    rules {
      id     = "rule2"
      status = "Enabled"
      prefix = "rule2-prefix"

      destination {
        bucket = "arn:aws:s3:::ewbankkit-us-west-2"
      }
    }
  }
}

before schema migration via the AWS console

$ aws s3api get-bucket-replication --bucket ewbankkit-us-east-1
{
    "ReplicationConfiguration": {
        "Rules": [
            {
                "Status": "Enabled", 
                "Prefix": "rule1-prefix", 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "ID": "rule1"
            }, 
            {
                "Status": "Enabled", 
                "Prefix": "rule2-prefix", 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "ID": "rule2"
            }
        ], 
        "Role": "arn:aws:iam::000000000000:role/service-role/r"
    }
}

after schema migration via the AWS console:

$ aws s3api get-bucket-replication --bucket ewbankkit-us-east-1
{
    "ReplicationConfiguration": {
        "Rules": [
            {
                "Status": "Enabled", 
                "Filter": {
                    "Prefix": "rule1-prefix"
                }, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 1, 
                "ID": "rule1"
            }, 
            {
                "Status": "Enabled", 
                "Filter": {
                    "Prefix": "rule2-prefix"
                }, 
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }, 
                "Destination": {
                    "Bucket": "arn:aws:s3:::ewbankkit-us-west-2"
                }, 
                "Priority": 2, 
                "ID": "rule2"
            }
        ], 
        "Role": "arn:aws:iam::000000000000:role/service-role/r"
    }
}

[ewbankkit]

My approach is going to be introduce a new resource schema version and in the MigrateState function to upgrade the CRR schema version via the S3 PutBucketReplication method.
The rule's Priority field is required but for backwards compatibility we'll have this as Optional and use the rule's hash value to derive a default value.
The documentation says

Valid values: 0 - INT-MAX.

so we should be OK.

[ewbankkit]

After some work with the approach described above (motivated by trying to make the change look a bit like #899) I have come to the conclusion that in order to maintain backwards compatibility we will need a new filter attribute in the CRR rule.
So the syntax suggested above will become

resource "aws_s3_bucket" "bucket" {
  bucket   = "tf-test-bucket-12345"

  replication_configuration {
    rules {
      id     = "foobar"
      priority = 123
      filter {
        prefix = "foo"
        tags {
          ReplicateMe = "Yes"
        }
      }
      status = "Enabled"

      destination {
        bucket        = "${aws_s3_bucket.destination.arn}"
        storage_class = "STANDARD"
      }
    }
  }
}

• For a specific rule, prefix conflicts with filter
• If any rule has filter specified then they all must
• priority is optional but must be unique between multiple rules

[ewbankkit]

I ended up with a simpler implementation:

• Document the rules listed above rather than trying to implement in code
• Don't expose the Delete Marker Replication functionality as there is only one valid value right now
• Just use 0 as the default priority value

[bflad]

This has been released in version 1.42.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[nekloth]

It also looks like we'll need to add support for the Delete Marker Replication functionality:

When you delete an object from your source bucket without specifying an object version ID, Amazon S3 adds a delete marker. If you use V1 of the replication configuration XML, Amazon S3 replicates delete markers that resulted from user actions. In other words, if the user deleted the object, and not if Amazon S3 deleted it because the object expired as part of lifecycle action. In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled.

So we'll have an additional optional delete_marker_replication attribute in the rule.

Hello

I need to setup the delete marker in the rule. But I do not see this attribute in docs...

Did I miss something?

Nek

[ewbankkit]

@nekloth For schema V2 (filter is not empty) the DeleteMarkerReplication attribute is always set to Disabled: https://github.com/terraform-providers/terraform-provider-aws/blob/ce20280b6b1383ae48c6374b294114779e6f9b0c/aws/resource_aws_s3_bucket.go#L1918-L1920

[nekloth]

@ewaltman Ok, thanks.

However, I had to explicitly mention the value of the DeleteMarkerReplication attribute in the tule to make it work .... but using the s3api.

{"DeleteMarkerReplication": { "Status": "Disabled" }

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!