custom-resource-handlers/aws-s3/auto-delete-objects-handler: Object locked objects prevent cdk destroy

[aaronmills1]

Describe the bug

When an s3 bucket is created with object lock governance and the s3 bucket is set for removal cdk destroy fails. This happens even when the role has permissions to bypassgovernancelock

This is because in the auto-delete-objects-handler the s3 delete objects is called without BypassGovernanceRetention

Current: await s3.deleteObjects({ Bucket: bucketName, Delete: { Objects: records } });

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I expect for the cdk destroy to remove the governed objects provided the role has the necessary permissions

Current Behavior

The cdk destroy fails and the stack reverts back to its previous state

Reproduction Steps

    import * as s3 from 'aws-cdk-lib/aws-s3'
    import * as cdk from 'aws-cdk-lib'
    new s3.Bucket(scope, 'testID', {
          autoDeleteObjects: true,
          removalPolicy: cdk.RemovalPolicy.DESTROY,
          versioned: true,
          objectLockDefaultRetention: s3.ObjectLockRetention.governance(
            cdk.Duration.days(7)
          )
        })

Add some objects, then run cdk destroy

Possible Solution

await s3.deleteObjects({ Bucket: bucketName, Delete: { Objects: records }, BypassGovernanceRetention: true });

The above will work for governed objects however will fail for non object locked objects. For this reason a combination of the two will need to be used

Additional Information/Context

No response

CDK CLI Version

2.160.0 (build 7a8ae02)

Framework Version

No response

Node.js Version

v21.7.1

OS

MacOS 14.7.1

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

Easily reproducible using provided code (assuming that TerminationProtection is not enabled for stack). Running cdk destroy gives below error:

Are you sure you want to delete: CdktestStack (y/n)? y
CdktestStack: destroying... [1/1]
11:41:08 AM | DELETE_FAILED        | AWS::S3::Bucket             | testID9C8A8850
Resource handler returned message: "The bucket you tried to delete is not empty. You must delete all versions in the bucket. (Service: S3, Status Code: 409, Request ID: RX44AEJ09S87FP7H, Extended Request ID: IYaKec09mJdNBH32jFvgsm8IhYk9kTIxAHgUbERHAjtUcVKD3ekZ8w
qr//7sDWQ7UTBVjPOEg0U=)" (RequestToken: 7fd28968-c0f1-d427-d9d7-b1e5c754c293, HandlerErrorCode: GeneralServiceException)


 ❌  CdktestStack: destroy failed Error: The stack named CdktestStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [testID9C8A8850]. ): Resource handler returned message: "The bucket you tried to delete is not empty. You must delete all versions in the bucket. (Service: S3, Status Code: 409, Request ID: RX44AEJ09S87FP7H, Extended Request ID: IYaKec09mJdNBH32jFvgsm8IhYk9kTIxAHgUbERHAjtUcVKD3ekZ8wqr//7sDWQ7UTBVjPOEg0U=)" (RequestToken: 7fd28968-c0f1-d427-d9d7-b1e5c754c293, HandlerErrorCode: GeneralServiceException)
    at destroyStack (/Users/ashdhin/.nvm/versions/node/v18.20.4/lib/node_modules/aws-cdk/lib/index.js:516:2450)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CdkToolkit.destroy (/Users/ashdhin/.nvm/versions/node/v18.20.4/lib/node_modules/aws-cdk/lib/index.js:521:14012)
    at async exec4 (/Users/ashdhin/.nvm/versions/node/v18.20.4/lib/node_modules/aws-cdk/lib/index.js:575:56240)
The stack named CdktestStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [testID9C8A8850]. ): Resource handler returned message: "The bucket you tried to delete is not empty. You must delete all versions in the bucket. (Service: S3, Status Code: 409, Request ID: RX44AEJ09S87FP7H, Extended Request ID: IYaKec09mJdNBH32jFvgsm8IhYk9kTIxAHgUbERHAjtUcVKD3ekZ8wqr//7sDWQ7UTBVjPOEg0U=)" (RequestToken: 7fd28968-c0f1-d427-d9d7-b1e5c754c293, HandlerErrorCode: GeneralServiceException)

Per Retention Modes section at Locking objects with Object Lock:

In governance mode, users can't overwrite or delete an object version or alter its lock settings unless they have special permissions. With governance mode, you protect objects against being deleted by most users, but you can still grant some users permission to alter the retention settings or delete the objects if necessary. You can also use governance mode to test retention-period settings before creating a compliance-mode retention period.

To override or remove governance-mode retention settings, you must have the s3:BypassGovernanceRetention permission and must explicitly include x-amz-bypass-governance-retention:true as a request header with any request that requires overriding governance mode.

Also, the error from custom resource handler is The bucket you tried to delete is not empty. You must delete all versions in the bucket.. Below is the example code from AWS .NET SDK (assuming user has proper permissions). We could perhaps use similar code in TypeScript in custom resource handler, first inspecting if the S3 bucket has Object Lock enabled:

IAmazonS3 s3Client = new AmazonS3Client();
string bucketName = "<<some-bucket>>";

var listVersionsRequest = new ListVersionsRequest
            {
                BucketName = bucketName
            };

            ListVersionsResponse listVersionsResponse;

            // Iterate through the objects in the bucket and delete them.
            do
            {
                // List all the versions of all the objects in the bucket.
                listVersionsResponse = s3Client.ListVersions(listVersionsRequest);

                if (listVersionsResponse.Versions.Count == 0)
                {
                    // If the bucket has no objects break the loop.
                    break;
                }

                var keyVersionList = new List<KeyVersion>(listVersionsResponse.Versions.Count);
                for (int index = 0; index < listVersionsResponse.Versions.Count; index++)
                {
                    keyVersionList.Add(new KeyVersion
                    {
                        Key = listVersionsResponse.Versions[index].Key,
                        VersionId = listVersionsResponse.Versions[index].VersionId
                    });
                }

                try
                {
                    // Delete the current set of objects.
                    var deleteObjectsResponse = s3Client.DeleteObjects(new DeleteObjectsRequest
                    {
                        BucketName = bucketName,
                        Objects = keyVersionList,                        
                        BypassGovernanceRetention = true
                    });
                }
                catch
                {                    
                }

                // Set the markers to get next set of objects from the bucket.
                listVersionsRequest.KeyMarker = listVersionsResponse.NextKeyMarker;
                listVersionsRequest.VersionIdMarker = listVersionsResponse.NextVersionIdMarker;

            }
            // Continue listing objects and deleting them until the bucket is empty.
            while (listVersionsResponse.IsTruncated);