Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XmlDecodeError when DeleteResult gets SlowDown #873

Closed
ahrens opened this issue Aug 17, 2023 · 6 comments
Closed

XmlDecodeError when DeleteResult gets SlowDown #873

ahrens opened this issue Aug 17, 2023 · 6 comments
Labels
bug This issue is a bug. p2 This is a standard priority issue

Comments

@ahrens
Copy link

ahrens commented Aug 17, 2023

Describe the bug

When the response from a DeleteObjects request is HTTP 200 but with a SlowDown error, the SDK does not correctly decode the result, resulting in an "Unhandled" error.

The Unhandled error says encountered invalid XML root: expected DeleteResult but got StartEl ... This is likely a bug in the SDK. The response is a HTTP 200 with a body of <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message>...

We are using AWS S3 (endpoint: "https://s3.us-west-2.amazonaws.com").

Expected Behavior

Not have an Unhandled error. Beyond that I'm not sure; maybe send().await should return Ok(DeleteObjectsOutput), and DeleteObjectsOutput::errors() should return Some.

Current Behavior

The unwrap in the above code fails with:

thread 'zoa' panicked at 'called `Result::unwrap()` on an `Err` value:
ServiceError( ServiceError {
        source: Unhandled( Unhandled
        {
                source: XmlDecodeError { kind: Custom(
                        "encountered invalid XML root: expected DeleteResult but got StartEl { name: Name { prefix: \"\", local: \"Error\" }, attributes: [], closed: false, depth: 0 }. This is likely a bug in the SDK."
                ) },
                meta: ErrorMetadata { code: None, message: None, extras: None }
        }),
        raw: Response {
                inner: Response
                {
                        status: 200,
                        version: HTTP/1.1,
                        headers: {"x-amz-id-2": "WWoZlnK4pTjKCYn6eNV7GgOurabfqLkjbSyqTvDMGBaI9uwzyNhSaDhOCPs8paFGye7S6b/AB3A=", "x-amz-request-id": "K2H6N7ZGQT6WHCEG", "date": "Wed, 16 Aug 2023 16:35:09 GMT", "transfer-encoding": "chunked", "server": "AmazonS3"},
                        body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>K2H6N7ZGQT6WHCEG</RequestId><HostId>WWoZlnK4pTjKCYn6eNV7GgOurabfqLkjbSyqTvDMGBaI9uwzyNhSaDhOCPs8paFGye7S6b/AB3A=</HostId></Error>")), retryable: true }
                },
        properties: SharedPropertyBag(Mutex {
                data: PropertyBag { contents: [
                        "alloc::vec::Vec<http::version::Version>",
                        "aws_credential_types::credentials_impl::Credentials",
                        "aws_sdk_s3::endpoint::Params", "aws_types::SigningService",
                        "aws_types::region::SigningRegion", "aws_types::region::Region",
                        "aws_smithy_types::endpoint::Endpoint",
                        "aws_sig_auth::signer::OperationSigningConfig",
                        "aws_sig_auth::middleware::Signature",
                        "aws_smithy_http::operation::Metadata",
                        "aws_http::user_agent::AwsUserAgent",
                        "aws_smithy_http::connection::CaptureSmithyConnection",
                        "aws_credential_types::cache::SharedCredentialsCache"
                ] },
                poisoned: false, .. })
        }
})',

Reproduction Steps

Our code is similar to:

                                client
                                    .delete_objects()
                                    .bucket(&self.bucket)
                                    .delete(
                                        chunk
                                            .iter()
                                            .fold(Delete::builder().quiet(true), |builder, key| {
                                                builder.objects(
                                                    ObjectIdentifier::builder().key(key).build(),
                                                )
                                            })
                                            .build(),
                                    )
                                    .send()
                                    .await
                                    .unwrap()

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v0.55.3
│   ├── aws-credential-types v0.55.3
│   │   ├── aws-smithy-async v0.55.3
│   │   ├── aws-smithy-types v0.55.3
│   ├── aws-http v0.55.3
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3
│   │   │   ├── aws-smithy-eventstream v0.55.3
│   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3
│   │   │   ├── aws-credential-types v0.55.3 (*)
│   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   ├── aws-smithy-client v0.55.3
│   │   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.55.3
│   │   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-sdk-sso v0.28.0
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-endpoint v0.55.3
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-types v0.55.3 (*)
│   │   ├── aws-http v0.55.3 (*)
│   │   ├── aws-sig-auth v0.55.3
│   │   │   ├── aws-credential-types v0.55.3 (*)
│   │   │   ├── aws-sigv4 v0.55.3
│   │   │   │   ├── aws-smithy-eventstream v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-eventstream v0.55.3 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-types v0.55.3 (*)
│   │   ├── aws-smithy-async v0.55.3 (*)
│   │   ├── aws-smithy-client v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-http-tower v0.55.3 (*)
│   │   ├── aws-smithy-json v0.55.3
│   │   │   └── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3 (*)
│   ├── aws-sdk-sts v0.28.0
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-endpoint v0.55.3 (*)
│   │   ├── aws-http v0.55.3 (*)
│   │   ├── aws-sig-auth v0.55.3 (*)
│   │   ├── aws-smithy-async v0.55.3 (*)
│   │   ├── aws-smithy-client v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-http-tower v0.55.3 (*)
│   │   ├── aws-smithy-json v0.55.3 (*)
│   │   ├── aws-smithy-query v0.55.3
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-xml v0.55.3
│   │   ├── aws-types v0.55.3 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-types v0.55.3 (*)
├── aws-credential-types v0.55.3 (*)
├── aws-http v0.55.3 (*)
├── aws-sdk-s3 v0.28.0
│   ├── aws-credential-types v0.55.3 (*)
│   ├── aws-endpoint v0.55.3 (*)
│   ├── aws-http v0.55.3 (*)
│   ├── aws-sig-auth v0.55.3 (*)
│   ├── aws-sigv4 v0.55.3 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-checksums v0.55.3
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-eventstream v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-xml v0.55.3 (*)
│   ├── aws-types v0.55.3 (*)
├── aws-smithy-async v0.55.3 (*)
├── aws-smithy-http v0.55.3 (*)
├── aws-smithy-types v0.55.3 (*)
├── aws-smithy-types-convert v0.55.3
│   ├── aws-smithy-types v0.55.3 (*)
├── aws-sdk-s3 v0.28.0 (*)

Environment details (OS name and version, etc.)

Linux

Logs

No response
@ahrens ahrens added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 17, 2023
@jdisanti jdisanti added p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Aug 17, 2023
@jdisanti
Copy link
Contributor

Thanks for reporting this!

@ahrens
Copy link
Author

ahrens commented Aug 17, 2023

Note, if we use delete_object() to delete one object at a time instead of batching, we get an error like the following, which is correctly categorized as a ThrottlingError and retried. The body looks similar to the error from delete_objects(), but the HTTP code is 503.

ServiceError(ServiceError {
        source: Unhandled(Unhandled {
                source: ErrorMetadata {
                        code: Some("SlowDown"),
                        message: Some("Please reduce your request rate."),
                        extras: Some({"aws_request_id": "CD509YQ0PWNZSCZ2", "s3_extended_request_id": "9FSfiFOU/TaOxx+iAKoEjrb2EYLAq8irVucbFPDBvB8Ll2e226hZaTLkjJNJz5r5kEPpi+ncZE4="}) },
                        meta: ErrorMetadata { code: Some("SlowDown"), message: Some("Please reduce your request rate."), extras: Some({"aws_request_id": "CD509YQ0PWNZSCZ2", "s3_extended_request_id": "9FSfiFOU/TaOxx+iAKoEjrb2EYLAq8irVucbFPDBvB8Ll2e226hZaTLkjJNJz5r5kEPpi+ncZE4="}) }
        }),
        raw: Response {
                inner: Response {
                        status: 503,
                        version: HTTP/1.1,
                        headers: {"x-amz-request-id": "CD509YQ0PWNZSCZ2", "x-amz-id-2": "9FSfiFOU/TaOxx+iAKoEjrb2EYLAq8irVucbFPDBvB8Ll2e226hZaTLkjJNJz5r5kEPpi+ncZE4=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Thu, 17 Aug 2023 22:41:47 GMT", "server": "AmazonS3", "connection": "close"},
                        body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>CD509YQ0PWNZSCZ2</RequestId><HostId>9FSfiFOU/TaOxx+iAKoEjrb2EYLAq8irVucbFPDBvB8Ll2e226hZaTLkjJNJz5r5kEPpi+ncZE4=</HostId></Error>")), retryable: true } },
                properties: SharedPropertyBag(Mutex { data: PropertyBag { contents: ["alloc::vec::Vec<http::version::Version>", "aws_credential_types::credentials_impl::Credentials", "aws_smithy_types::endpoint::Endpoint", "aws_sdk_s3::endpoint::Params", "aws_smithy_http::operation::Metadata", "aws_sig_auth::middleware::Signature", "aws_smithy_http::connection::CaptureSmithyConnection", "aws_sig_auth::signer::OperationSigningConfig", "aws_http::user_agent::AwsUserAgent", "aws_types::region::SigningRegion", "aws_types::region::Region", "aws_credential_types::cache::SharedCredentialsCache", "aws_types::SigningService"] }, poisoned: false, .. })
 } })

@jdisanti jdisanti mentioned this issue Aug 21, 2023
Closed
@rcoh
Copy link
Contributor

rcoh commented Aug 29, 2023

yeah the issue here seems to be that we're getting a 200 back so we don't try to hit the error handler. Will see what the best fix is

@aajtodd aajtodd mentioned this issue Aug 29, 2023
Closed
@rcoh rcoh mentioned this issue Aug 29, 2023
Merged
2 tasks
github-merge-queue bot pushed a commit to smithy-lang/smithy-rs that referenced this issue Aug 30, 2023
@rcoh
Allow <Error> to trigger error handling for S3 (#2958)
2db1c34 
## Motivation and Context
- awslabs/aws-sdk-rust#873

## Description
Add a customization for S3 so that if `<Error>` is the root element, we
trigger the error parsing flow

## Testing
- Added an integration test

## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
@rcoh rcoh added the pending-release This issue will be fixed by an approved PR that hasn't been released yet. label Aug 30, 2023
@dayaffe dayaffe mentioned this issue Sep 11, 2023
Closed
2 tasks
@rcoh rcoh closed this as completed Sep 20, 2023
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@abhijat abhijat mentioned this issue Oct 10, 2023
Closed
@rcoh
Copy link
Contributor

rcoh commented Nov 7, 2023

fix verified

@rcoh rcoh removed the pending-release This issue will be fixed by an approved PR that hasn't been released yet. label Nov 7, 2023
@ahrens
Copy link
Author

ahrens commented Nov 7, 2023

Thanks!

@NickDarvey NickDarvey mentioned this issue Jun 25, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. p2 This is a standard priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jdisanti @rcoh @ahrens