Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Handle batch failures in FIFO queues correctly #1183

Merged
merged 24 commits into from
Jul 11, 2023

Conversation

scottgerring
Copy link
Contributor

@scottgerring scottgerring commented Jun 8, 2023

Issue #, if available: #1140

Description of changes:

When processing a FIFO-queue batch, once a message for a given messageGroupId fails, no subsequent messages in the group should be processed. This ensures ordering of messages is maintained.

The implementation for powertools python is here, and the documentation here. Note that the python implementation requires the user to specify the FIFO batch processor explicitly. I believe this is unnecessary - we can tell that we are working with a FIFO queue when we see the MessageGroupId attribute on the message.

This implementation is related to #797 which will see us redo our batch processing implementation. IMHO it is worth fixing our existing bug with the existing FIFO implementation as the new version that supersedes this will likely be an opt-in breaking change and is a much more significant piece of work.

Checklist

Breaking change checklist

RFC issue #:

  • Migration process documented
  • Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@scottgerring scottgerring marked this pull request as draft June 8, 2023 05:17
@codecov-commenter
Copy link

codecov-commenter commented Jun 8, 2023

Codecov Report

Patch coverage: 95.65% and project coverage change: +0.13 🎉

Comparison is base (c7aedc4) 78.94% compared to head (2a39a84) 79.08%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1183      +/-   ##
============================================
+ Coverage     78.94%   79.08%   +0.13%     
- Complexity      629      634       +5     
============================================
  Files            72       73       +1     
  Lines          2332     2352      +20     
  Branches        253      256       +3     
============================================
+ Hits           1841     1860      +19     
  Misses          412      412              
- Partials         79       80       +1     
Impacted Files Coverage Δ
...oftware/amazon/lambda/powertools/sqs/SqsUtils.java 79.72% <95.23%> (+4.72%) ⬆️
...ption/SkippedMessageDueToFailedBatchException.java 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@scottgerring scottgerring changed the title [WIP] Handle batch failures in FIFO queues correctly [bug] Handle batch failures in FIFO queues correctly Jun 8, 2023
@pull-request-size pull-request-size bot added size/L and removed size/S labels Jun 8, 2023
@scottgerring scottgerring changed the title [bug] Handle batch failures in FIFO queues correctly [fix] Handle batch failures in FIFO queues correctly Jun 12, 2023
@scottgerring scottgerring changed the title [fix] Handle batch failures in FIFO queues correctly fix: Handle batch failures in FIFO queues correctly Jun 12, 2023
@scottgerring
Copy link
Contributor Author

Hey @mriccia do you have time to look at this? Trying to spread the PR review burden around ;)

@scottgerring scottgerring marked this pull request as ready for review June 21, 2023 13:43
Copy link
Contributor

@jeromevdl jeromevdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are not easy to read, but probably because the code itself is... Few comments, Would be great to have another pair of eyes on it.

@jeromevdl jeromevdl assigned scottgerring and unassigned jeromevdl and mriccia Jul 10, 2023
Copy link
Contributor

@jeromevdl jeromevdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there

scottgerring and others added 3 commits July 11, 2023 09:52
…/sqs/SqsUtils.java

Co-authored-by: Jérôme Van Der Linden <117538+jeromevdl@users.noreply.github.com>
Copy link
Contributor

@jeromevdl jeromevdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just remove the break and we're good

Copy link
Contributor

@jeromevdl jeromevdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !!

@jeromevdl jeromevdl merged commit a19def3 into main Jul 11, 2023
@jeromevdl jeromevdl deleted the fix-fifo-batch-failure branch July 11, 2023 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants