Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Error Processing Files Larger Than 2GB Due to Integer Overflow in Data Prepper #5276

Open
parastoosln opened this issue Dec 19, 2024 · 1 comment
Labels
bug Something isn't working untriaged

Comments

@parastoosln
Copy link

Describe the bug
When processing files larger than 2GB, the system throws an error indicating that the numeric value is out of range for an integer.

To Reproduce
Steps to reproduce the behavior:

  1. Upload a file larger than 2GB to the specified S3 bucket.
  2. Ensure that the S3EventBridgeNotification is configured to trigger when a new file is uploaded.
  3. Once the Data Prepper pipeline is triggered, the error shows up in the logs.

Expected behavior
The system should handle files larger than 2GB without throwing an error.

Additional context

404 [s3-source-sqs-1] ERROR org.opensearch.dataprepper.plugins.source.s3.parser.S3EventBridgeNotificationParser - SQS message with message ID:8947a26b-2500-4859-bfd1-05353e0fc232 has invalid body which cannot be parsed into EventBridgeNotification. Numeric value (2409161669) out of range of int (-2147483648 - 2147483647)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 523] (through reference chain: org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification["detail"]->org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification$Detail["object"]->org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification$Object["size"])
@faizan-planview
Copy link

Issue is here -

Where it is typecasted as int

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged
Projects
Development

No branches or pull requests

2 participants