Only retry multi-part upload as single-part upload on GCS endpoints #2012
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because of GCS compatibility, the multi-part upload retries a multi-part upload as a single
PutObjectwhen it receives an access denied. This introduces a strange error message, because when a multi-part upload isn't allowed (i.e. due to policy), then it retries the operation as a single-part operation.Single part uploads are limited to 5GiB, so if uploading a file larger than 5GiB is uploaded, but not allowed, this will return the following error message:
Your proposed upload size ‘...’ exceeds the maximum allowed object size ‘5368709120’ for single PUT operation.. For AWS S3 and MinIO, there is just a single action for upload operations (s3:PutObject), so it doesn't make sense to retry the upload. This saves a roundtrip and an error message that could be misleading.