Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws-cdk/aws-s3: Unable to deploy CDK due to asset bucket request missing headers #31926

Closed
1 task done
cedw93 opened this issue Oct 29, 2024 · 3 comments
Closed
1 task done

aws-cdk/aws-s3: Unable to deploy CDK due to asset bucket request missing headers #31926

cedw93 opened this issue Oct 29, 2024 · 3 comments
Assignees
@rix0rrr
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. p1 potential-regression Marking this issue as a potential regression to be checked by team member

Comments

@cedw93
Copy link

cedw93 commented Oct 29, 2024 •
edited
Loading

Describe the bug

Dependabot recently updated our CDK to version 2.164.1 which has broken our deployments due an issue publishing to s3. The Error is (sensitive information removed):

event-handlerCluster: start: Building 
aee30b6515587d0de5c06fce1889cea27c2ae6eaf9ada7885811078e911caf2c:000000000000-eu-west-1
event-handlerCluster: success: Built 
event-handlerCluster: success: Built aee30b6515587d0de5c06fce1889cea27c2ae6eaf9ada7885811078e911caf2c:000000000000-eu-west-1
event-handlerApplication: start: Building 771cc1f4e8115208b9f2549d7b353fe88ccf21f59c2f4cff0062b49a57b37751:000000000000-eu-west-1
event-handlerApplication: success: Built 771cc1f4e8115208b9f2549d7b353fe88ccf21f59c2f4cff0062b49a57b37751:000000000000-eu-west-1
event-handlerCluster: start: Publishing aee30b6515587d0de5c06fce1889cea27c2ae6eaf9ada7885811078e911caf2c:000000000000-eu-west-1
event-handlerCluster: fail: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

Our buckets have Governance Objects lock on them which we cannot remove.

It seems like it might be related to the FIPS configuration recently raised here: #31883

Reverting to a previous CDK version allowed the deployment to complete but this impacts our patch management if we cannot update CDK.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.162.1

Expected Behavior

CDK deploy works as expected

Current Behavior

CDK deployment fails when the asset buckets have an object lock

Reproduction Steps

Run CDK Deploy for our stack as normal

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.164.1

Framework Version

No response

Node.js Version

18.20.4

OS

Linux & Mac OS

Language

TypeScript

Language Version

No response

Other information

No response
@cedw93 cedw93 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 29, 2024
@github-actions github-actions bot added @aws-cdk/aws-s3 Related to Amazon S3 potential-regression Marking this issue as a potential regression to be checked by team member labels Oct 29, 2024
@cedw93 cedw93 changed the title (module name): (short issue description) aws-cdk/aws-s3: Unable to deploy CDK due to asset bucket request missing headers Oct 29, 2024
@mrgrain mrgrain added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Oct 29, 2024
@mrgrain
Copy link
Contributor

mrgrain commented Oct 29, 2024

Investigating. This seems possible with a non-standard bootstrapping setup.

@jiayiwang7 jiayiwang7 assigned mrgrain and rix0rrr and unassigned mrgrain Oct 29, 2024
rix0rrr added a commit to cdklabs/cdk-assets that referenced this issue Oct 30, 2024
@rix0rrr
fix: upload to Object Lock-enabled bucket fails
d7000e7 
We thought disabling a content checksum for S3 clients would work in all
scenarios, so that we can use the same S3 client config for FIPS and
non-FIPS environments. Turns out that yet another scenario requires
content checksums: PutObject to an Object Lock-enabled S3 bucket.

Again, there's no way to make this work with SDKv2. We do the best
we can do: turn off MD5 signing only for FIPS environments. Result:
it will not be possible to use Object Lock in FIPS environments.

But at least we unbreak the scenario for non-FIPS customers.

Relates to aws/aws-cdk#31926
This was referenced Oct 30, 2024
Merged
Merged
Merged
github-merge-queue bot pushed a commit to cdklabs/cdk-assets that referenced this issue Oct 30, 2024
@rix0rrr
fix: force SHA256 checksums (#153)
018a045 
By default, the S3 client will use MD5 checksums. But in FIPS
environments, those MD5 checksums will not work because Node will not
have access to an M5 digest.

Force the usage of SHA256 for these checksums instead. In SDKv3, this is
as simple as configuring the algorithm to use, and the SDK itself will
do the rest.

Relates to aws/aws-cdk#31926
github-merge-queue bot pushed a commit to cdklabs/cdk-assets that referenced this issue Oct 30, 2024
@rix0rrr
fix: upload to Object Lock-enabled bucket fails (#156)
c39fb85 
We thought disabling a content checksum for S3 clients would work in all
scenarios, so that we can use the same S3 client config for FIPS and
non-FIPS environments. Turns out that yet another scenario requires
content checksums: PutObject to an Object Lock-enabled S3 bucket.

Again, there's no way to make this work with SDKv2. We do the best we
can do: turn off MD5 signing only for FIPS environments. Result: it will
not be possible to use Object Lock in FIPS environments.

But at least we unbreak the scenario for non-FIPS customers.

Relates to aws/aws-cdk#31926
github-merge-queue bot pushed a commit to cdklabs/cdk-assets that referenced this issue Oct 30, 2024
@rix0rrr
fix: upload to Object Lock-enabled bucket fails (#156)
0fea04e 
We thought disabling a content checksum for S3 clients would work in all
scenarios, so that we can use the same S3 client config for FIPS and
non-FIPS environments. Turns out that yet another scenario requires
content checksums: PutObject to an Object Lock-enabled S3 bucket.

Again, there's no way to make this work with SDKv2. We do the best we
can do: turn off MD5 signing only for FIPS environments. Result: it will
not be possible to use Object Lock in FIPS environments.

But at least we unbreak the scenario for non-FIPS customers.

Relates to aws/aws-cdk#31926
github-merge-queue bot pushed a commit to cdklabs/cdk-assets that referenced this issue Oct 30, 2024
@rix0rrr
fix: force SHA256 checksums (#153)
20e91ec 
By default, the S3 client will use MD5 checksums. But in FIPS
environments, those MD5 checksums will not work because Node will not
have access to an M5 digest.

Force the usage of SHA256 for these checksums instead. In SDKv3, this is
as simple as configuring the algorithm to use, and the SDK itself will
do the rest.

Relates to aws/aws-cdk#31926
@github-actions github-actions bot mentioned this issue Nov 1, 2024
Closed
mergify bot pushed a commit that referenced this issue Nov 1, 2024
@rix0rrr
fix(cli): asset uploads fail if Object Lock is enabled on access buck…
ab1e91d 
…et (#31937)

Object Lock requires passing an object checksum. By default, SDKv2 only calculates MD5 checksums.

We used to turn off checksums altogether and rely on SigV4 checksums to produce a workable setup for both FIPS and non-FIPS users, but in case of Object Lock this doesn't work: we must definitely have an S3 content checksum, and the the SigV4 checksum alone is not good enough.

Since SDKv2 only supports MD5 checksums, we now only disable checksums for FIPS environments.

The unfortunate result is that Object Lock will not work in a FIPS environment, but there's no way around that for now.

When we migrate to SDKv3, which can be configured to checksum using SHA256, Object Lock + FIPS will work again.

Relates to #31926

(This PR also adds tests for the PluginHost because otherwise the build fails due to coverage requirements)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions github-actions bot mentioned this issue Nov 4, 2024
Closed
@rix0rrr
Copy link
Contributor

rix0rrr commented Nov 6, 2024

Closed by #31937

@rix0rrr rix0rrr closed this as completed Nov 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2024

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@rix0rrr rix0rrr

Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. p1 potential-regression Marking this issue as a potential regression to be checked by team member
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mrgrain @rix0rrr @cedw93