(aws-s3-deployment): bucket deployment fails when toolkit stack uses Customer KMS

[jaecktec]

Describe the bug

When initialising the toolkit stack with --bootstrap-kms-key-id the bucket deployment fails.

Expected Behavior

Bucket deployment copies the ressources to the desired destination

Current Behavior

The deployment fails and the logs read:

download failed: s3://cdk-<qualifier>-assets-<account>-<region>/<asset-hash>.zip to <tmp-folder> An error occurred (AccessDenied) when calling the GetObject operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

Reproduction Steps

Bootstrap with KMS:

npx cdk bootstrap aws://$AWS_ACCOUNT_ID/$AWS_REGION --bootstrap-kms-key-id $KMS_KEY_ID

create a stack with a bucket deployment

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create an S3 bucket
    const bucket = new s3.Bucket(this, 'MyBucket', {
      bucketName: 'my-bucket-name'
    });

    // Deploy a text file to the bucket using BucketDeployment
    new s3deploy.BucketDeployment(this, 'DeployTextFile', {
      sources: [s3deploy.Source.asset('./path/to/text/file.txt')],
      destinationBucket: bucket,
      destinationKeyPrefix: 'path/to/deployed/text/file.txt'
    });
  }
}

Possible Solution

The handlerRole of the BucketDeployment construct needs to have something like this:

    Key.fromKeyArn(this, 'cdk-toolkit-encryption-key', Stack.of(this).formatArn({
      region: Aws.REGION,
      service: 'kms',
      account: Aws.ACCOUNT_ID,
      resource: 'key',
      resourceName: Fn.importValue(`CdkBootstrap-${this.node.tryGetContext('@aws-cdk/core:bootstrapQualifier')}-FileAssetKeyArn`),
    }))
      // @ts-ignore
      .grantEncryptDecrypt(bucketDeployment.handlerRole);

Additional Information/Context

Bootstrap Stack is updated to the most recent version

CDK CLI Version

2.73.0

Framework Version

No response

Node.js Version

16.20.0

OS

macos 13.2.1

Language

Typescript

Language Version

4.9.5

Other information

No response

[pahud]

Thank you for your report. Yeah I believe you will need to grant your handler role to decrypt your CMK in this case. I believe we probably can improve this user experience even better. I am leaving this issue open and welcome all community feedbacks.

[kxue-godaddy]

@pahud , I think we are facing the same problem and this pattern might apply to other enterprise users as well.

My company uses a slightly customized version of the CDK Bootstrap utility – we hard-code ServerSideEncryptionByDefault.KMSMasterKeyID in the CFN template of the CDKToolkit stack. Therefore, the StagingBucket is by default encrypted with a KMS key from a central management account. When using BucketDeployment, we need to give the execution role of the CustomResourceHandler Lambda permission to perform kms:Decrypt on this cross-account KMS key. We know that we can pass in BucketDeploymentProps.role, but this means we need to manually add to this role all the S3 permissions which would otherwise be automatically generated inside BucketDeployment. Currently we are using the "monkey-patching" hack below, but I don't think it's a good coding pattern because it replies on private implementation details.

const lambda = bucketDeployment.node.findChild('CustomResourceHandler') as any;
const role = lambda.role as Role;
role.addToPolicy(
  new PolicyStatement({
  actions: ['kms:Decrypt'],
  effect: Effect.ALLOW,
  resources: ['*'],

I want to propose possibly adding one (or more) of the following new features to BucketDeployment, to make the user-experience better. Could you let me know if you would accept any of them into the code base? If yes, I'd be happy to prepare a PR. I think I have the code change in mind.

1. Expose the Lambda execution role as a public attribute of BucketDeployment. This enables users to easily call addToPolicy on the role without using the hack above. It is also a simple code change I think.
2. I noticed that most of the source buckets seem to be the StagingBucket of the CDKToolkit stack. This stack by default has a CFN export named CdkBootstrap-hnb659fds-FileAssetKeyArn, which is the ARN of the KMS encryption key used. We can add the kms:Decrypt permission to this specific ARN inside BucketDeployment. This probably solves most of the situations (at least ours) without users touching the execution role.
3. If there is a source coming from using s3deploy.Source.bucket(...) on an existing bucket, it seems the only way to get the encryption key ARN of the bucket is to use another Custom Resource. Using this could add all the necessary (and minimal) kms:Decrypt permissions without exposing new attributes of BucketDeployment, but I'm not sure if it's worth the effort.
4. Maybe we give a "wholesale" kms:Decrypt permission on resources: ['*'] to the Lambda execution role by default? This doesn't change the "user interface" of BucketDeployment either. At my company all KMS keys are typically from a management account and have resource-based policies, so I'm not worried about the wildcard permission, but this is possibly not a good practice for other users.

I'd be happy to make a PR on changes that you think are acceptable. Please let me know. Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.