feat(s3): EventBridge bucket notifications #18614
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rix0rrr
added a commit
that referenced
this pull request
Jan 26, 2022
In #18150, a change was merged that blew up the size of the inline Lambda beyond its limit of 4096 characters. This change was not detected because the Lambda constructs being used there didn't use the regular `aws-lambda` module, but escape hatches that bypass the regular validation (released in 1.139.0, 2.8.0). Because this effectively broke S3 notifications, it was rolled back in #18507 (released in 1.140.0, not yet released in 2.x line). In this PR, add validation to make sure an event like this doesn't happen again. This will be relevant for #18614.
|
Be sure to merge only after #18660 |
otaviomacedo
added
the
pr/do-not-merge
mergify bot
pushed a commit
that referenced
this pull request
Jan 26, 2022
In #18150, a change was merged that blew up the size of the inline Lambda beyond its limit of 4096 characters. This change was not detected because the Lambda constructs being used there didn't use the regular `aws-lambda` module, but escape hatches that bypass the regular validation (released in 1.139.0, 2.8.0). Because this effectively broke S3 notifications, it was rolled back in #18507 (released in 1.140.0, not yet released in 2.x line). In this PR, add validation to make sure an event like this doesn't happen again. This will be relevant for #18614. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
Now that #18660 is merged, any chance we can get this one in so it's available in the next release please? |
|
@jaidisido, I will merge #18660 soon. However, as mentioned in the description this PR is still blocked on Lambda SDK update. |
|
|
TikiTDO
pushed a commit
to TikiTDO/aws-cdk
that referenced
this pull request
Feb 21, 2022
In aws#18150, a change was merged that blew up the size of the inline Lambda beyond its limit of 4096 characters. This change was not detected because the Lambda constructs being used there didn't use the regular `aws-lambda` module, but escape hatches that bypass the regular validation (released in 1.139.0, 2.8.0). Because this effectively broke S3 notifications, it was rolled back in aws#18507 (released in 1.140.0, not yet released in 2.x line). In this PR, add validation to make sure an event like this doesn't happen again. This will be relevant for aws#18614. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
2 tasks
rix0rrr
added
feature-request
|
any updates on this? we are looking to use s3 bridge events |
|
@otaviomacedo , I just run integration test in I merged latest master branch. This is now ready to be merged. |
rix0rrr
approved these changes
Apr 1, 2022
rix0rrr
changed the title
|
Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
This was referenced Apr 7, 2022
mergify bot
added a commit
that referenced
this pull request
Apr 7, 2022
See [CHANGELOG](https://github.com/aws/aws-cdk/blob/bump/1.152.0/CHANGELOG.md) For convenience, extracted the relevant CHANGELOG entry: ## [1.152.0](v1.151.0...v1.152.0) (2022-04-06) ### Features * **cfnspec:** cloudformation spec v63.0.0 ([#19679](#19679)) ([dba96a9](dba96a9)) * **cfnspec:** cloudformation spec v65.0.0 ([#19745](#19745)) ([796fc64](796fc64)) * **cli:** add --build option ([#19663](#19663)) ([eb9b8e2](eb9b8e2)), closes [#19667](#19667) * **cli:** preview of `cdk import` ([#17666](#17666)) ([4f12209](4f12209)) * **core:** throw error when stack name exceeds max length ([#19725](#19725)) ([1ffd45e](1ffd45e)) * **eks:** add k8s v1.22 ([#19756](#19756)) ([9a518c5](9a518c5)) * **opensearch:** Add latest Opensearch Version 1.2 ([#19749](#19749)) ([a2ac36e](a2ac36e)) * add new integration test runner ([#19754](#19754)) ([1b4d010](1b4d010)) * **eks:** alb-controller v2.4.1 ([#19653](#19653)) ([1ec08df](1ec08df)) * **lambda:** add support for ephemeral storage ([#19552](#19552)) ([f1d9b6a](f1d9b6a)), closes [#19605](#19605) * **s3:** EventBridge bucket notifications ([#18614](#18614)) ([d8e602b](d8e602b)), closes [#18076](#18076) * **synthetics:** new puppeteer 3.5 runtime ([#19673](#19673)) ([ce2b91b](ce2b91b)), closes [#19634](#19634) ### Bug Fixes * **aws_applicationautoscaling:** Add missing members to PredefinedMetric enum ([#18978](#18978)) ([75a6fa7](75a6fa7)), closes [#18969](#18969) * **cli:** apps with many resources scroll resource output offscreen ([#19742](#19742)) ([053d22c](053d22c)), closes [#19160](#19160) * **cli:** support attributes of DynamoDB Tables for hotswapping ([#19620](#19620)) ([2321ece](2321ece)), closes [#19421](#19421) * **cloudwatch:** automatic metric math label cannot be suppressed ([#17639](#17639)) ([7fa3bf2](7fa3bf2)) * **codedeploy:** add name validation for Application, Deployment Group and Deployment Configuration ([#19473](#19473)) ([9185042](9185042)) * **codedeploy:** the Service Principal is wrong in isolated regions ([#19729](#19729)) ([7e9a43d](7e9a43d)), closes [#19399](#19399) * **core:** `Fn.select` incorrectly short-circuits complex expressions ([#19680](#19680)) ([7f26fad](7f26fad)) * **core:** detect and resolve stringified number tokens ([#19578](#19578)) ([7d9ab2a](7d9ab2a)), closes [#19546](#19546) [#19550](#19550) * **core:** reduce CFN template indent size to save bytes ([#19656](#19656)) ([fd63ca3](fd63ca3)) * **ecs:** 'desiredCount' and 'ephemeralStorageGiB' cannot be tokens ([#19453](#19453)) ([c852239](c852239)), closes [#16648](#16648) * **ecs:** remove unnecessary error when adding volume to external task definition ([#19774](#19774)) ([5446ded](5446ded)), closes [#19259](#19259) * **iam:** policies aren't minimized as far as possible ([#19764](#19764)) ([876ed8a](876ed8a)), closes [#19751](#19751) * **logs:** Faulty Resource Policy Generated ([#19640](#19640)) ([1fdf122](1fdf122)), closes [#17544](#17544)
mergify bot
added a commit
that referenced
this pull request
Apr 7, 2022
See [CHANGELOG](https://github.com/aws/aws-cdk/blob/bump/2.20.0/CHANGELOG.md) For convenience, extracted the relevant CHANGELOG entry: ## [2.20.0](v2.19.0...v2.20.0) (2022-04-07) ### Features * **cfnspec:** cloudformation spec v63.0.0 ([#19679](#19679)) ([dba96a9](dba96a9)) * **cfnspec:** cloudformation spec v65.0.0 ([#19745](#19745)) ([796fc64](796fc64)) * **cli:** add --build option ([#19663](#19663)) ([eb9b8e2](eb9b8e2)), closes [#19667](#19667) * **cli:** preview of `cdk import` ([#17666](#17666)) ([4f12209](4f12209)) * **core:** throw error when stack name exceeds max length ([#19725](#19725)) ([1ffd45e](1ffd45e)) * **eks:** add k8s v1.22 ([#19756](#19756)) ([9a518c5](9a518c5)) * **opensearch:** Add latest Opensearch Version 1.2 ([#19749](#19749)) ([a2ac36e](a2ac36e)) * add new integration test runner ([#19754](#19754)) ([1b4d010](1b4d010)) * **eks:** alb-controller v2.4.1 ([#19653](#19653)) ([1ec08df](1ec08df)) * **lambda:** add support for ephemeral storage ([#19552](#19552)) ([f1d9b6a](f1d9b6a)), closes [#19605](#19605) * **s3:** EventBridge bucket notifications ([#18614](#18614)) ([d8e602b](d8e602b)), closes [#18076](#18076) ### Bug Fixes * **aws_applicationautoscaling:** Add missing members to PredefinedMetric enum ([#18978](#18978)) ([75a6fa7](75a6fa7)), closes [#18969](#18969) * **cli:** apps with many resources scroll resource output offscreen ([#19742](#19742)) ([053d22c](053d22c)), closes [#19160](#19160) * **cli:** support attributes of DynamoDB Tables for hotswapping ([#19620](#19620)) ([2321ece](2321ece)), closes [#19421](#19421) * **cloudwatch:** automatic metric math label cannot be suppressed ([#17639](#17639)) ([7fa3bf2](7fa3bf2)) * **codedeploy:** add name validation for Application, Deployment Group and Deployment Configuration ([#19473](#19473)) ([9185042](9185042)) * **codedeploy:** the Service Principal is wrong in isolated regions ([#19729](#19729)) ([7e9a43d](7e9a43d)), closes [#19399](#19399) * **core:** `Fn.select` incorrectly short-circuits complex expressions ([#19680](#19680)) ([7f26fad](7f26fad)) * **core:** detect and resolve stringified number tokens ([#19578](#19578)) ([7d9ab2a](7d9ab2a)), closes [#19546](#19546) [#19550](#19550) * **core:** reduce CFN template indent size to save bytes ([#19656](#19656)) ([fd63ca3](fd63ca3)) * **ecs:** 'desiredCount' and 'ephemeralStorageGiB' cannot be tokens ([#19453](#19453)) ([c852239](c852239)), closes [#16648](#16648) * **ecs:** remove unnecessary error when adding volume to external task definition ([#19774](#19774)) ([5446ded](5446ded)), closes [#19259](#19259) * **iam:** policies aren't minimized as far as possible ([#19764](#19764)) ([876ed8a](876ed8a)), closes [#19751](#19751) * **logs:** Faulty Resource Policy Generated ([#19640](#19640)) ([1fdf122](1fdf122)), closes [#17544](#17544)
StevePotter
pushed a commit
to StevePotter/aws-cdk
that referenced
this pull request
Apr 27, 2022
## Duplicate of aws#18150 ## ~~Blocked on Lambda runtime SDK update to Botocore >= v1.23.16 (Integration test currently fails as current version (v1.21.55) does not contain EventBridge configuration)~~ ## ~~Check latest version here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html~~ ### **Description** Adds EventBridge bucket notification configuration. See https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/ ### **Implementation** - Added new Bucket property to enable this feature (`eventBridgeEnabled: true`) - Added EventBridge config to `S3BucketNotifications` Custom Resource - Added unit tests - Added integration test (currently fails, see below for more info) - Fixed dependent integration tests Closes aws#18076 ### **FAQ** 1. **Why not simply expose EventBridge Cfn property via S3 BucketProps?** Currently CDK manages `NotificationConfigurations `via CustomResource. If we were to expose that way, then e.g. SNS configuration would override EventBridge config. 2. **Why not create new `IBucketNotificationDestination` class for EventBridge?** We can, but there is no need. Usually we create a subclass to `IBucketNotificationDestination` in order to adjust resource permissions, however in this case there is no need to adjust permissions: [default EventBridge does not require any additional permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ev-permissions.html) unlike SQS/SNS/Lambda destinations. Additionally, enabling this feature via bucket props is much cleaner/simpler API than creating new dummy object of type `IBucketNotificationDestination` for customers. However, if you still think that we need to create new `IBucketNotificationDestination` subclass for EventBridge for consistency, let me know and I will refactor. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Duplicate of #18150
Blocked on Lambda runtime SDK update to Botocore >= v1.23.16 (Integration test currently fails as current version (v1.21.55) does not contain EventBridge configuration)Check latest version here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.htmlDescription
Adds EventBridge bucket notification configuration.
See https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/
Implementation
eventBridgeEnabled: true)S3BucketNotificationsCustom ResourceCloses #18076
FAQ
Currently CDK manages
NotificationConfigurationsvia CustomResource. If we were to expose that way, then e.g. SNS configuration would override EventBridge config.IBucketNotificationDestinationclass for EventBridge?We can, but there is no need. Usually we create a subclass to
IBucketNotificationDestinationin order to adjust resource permissions, however in this case there is no need to adjust permissions: default EventBridge does not require any additional permissions unlike SQS/SNS/Lambda destinations. Additionally, enabling this feature via bucket props is much cleaner/simpler API than creating new dummy object of typeIBucketNotificationDestinationfor customers.However, if you still think that we need to create new
IBucketNotificationDestinationsubclass for EventBridge for consistency, let me know and I will refactor.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license