-
-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds support for checksums in streamed request trailers #962
Conversation
@thehesiod am not too pleased with |
fdfea39
to
1705f45
Compare
So when I run the S3 tests locally without the changes, I get the same number failed as with the changes, so I don't think they're related :) |
Yeah it looks like moto is unhappy for some reason. I ran the server standalone and pointed the tests at it and they seem to be getting 404'd at least on creating multipart uploads, even using boto3. Update: So if you downgrade Werkzeug to 2.1.2 and Flask to 2.1.3, all the tests pass, is related to getmoto/moto#5341 |
thanks for all the research Terry! hopefully get to this soon, as it is getting that last release out it's 3am and I need to wake up at 7am, yowzers ;) |
looks like they pinned werkzeug + flask and I bumped us to the latest moto, may want to try with what we have and rebase to master |
just read a bit about this feature, very cool! This will solve the whole async file streaming support, since otherwise it didn't make much sense. I'm very interested in getting this working asap so will put further priority on reviewing this |
I could have used this feature several times in the last few years :) |
1705f45
to
7eeb54a
Compare
Yeah the coverage isnt going to be great as we can't test this unless moto runs with TLS. We could make that would though I think that should be done separately. |
hey @terrycain for the delay, been nuts. Hey some good news I got my first sponsor for this project, can I share a few bucks with you? |
Sure :D always up for some free money this is my paypal link else I've finally had the will to setup the github sponsors stuff :). |
7eeb54a
to
c1c5f91
Compare
I dislike being "that guy" but wanted to check in to see what needs to happen to get this in? Looking forward to this fix. |
@thehesiod can we get this merged ignoring codecov? |
hey sorry been nuts, got covid and perhaps RSV too from our kids one of my highest priorities is getting this in, really want to look at this in detail |
ok changes look good, just want to dig into the Transfer-Encoding issue |
ok figured out the chunk thing, and added support for running https tests, unfortunately moto doesn't appear to support the algs yet so still need to xfail for now. |
Codecov Report
@@ Coverage Diff @@
## master #962 +/- ##
==========================================
+ Coverage 84.98% 85.11% +0.12%
==========================================
Files 50 50
Lines 4529 4602 +73
==========================================
+ Hits 3849 3917 +68
- Misses 680 685 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
🎉 |
Confirmed this fixed setting checksum algorithm when using S3FS. Thanks! |
Description of Change
Adds support for S3 streaming uploads to support checksums in appended on the end of the request stream (trailers)
Running a snippet like the following will use streaming uploads and also the new SHA256 checksums
Note - when using Moto or an S3 endpoint not using HTTPS, the checksum will be added to the header, not the trailers as the headers are signed, the trailer is not (hence the requirements for HTTPS)
The problem comes with
botocore.httpchecksum.AwsChunkedWrapper
ending up as the body passed toaiohttp
and it not liking it. We now subclass that wrapper class and add an async iterator implementation.Fixes #954
Fixes terricain/aioboto3#265
Assumptions
That if
"Transfer-Encoding": "Chunked"
is always removed from the header dict before being passed to aiohttp is ok. (aiohttp reallllllly doesnt like it if you send it streaming body, and already set the transfer encoding :/ ). This ends up being injected from multiple places, mainly some of the checksum code as wells somewhere deep in the aws prepare_request() functions,AWS Page - https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/