Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect multipart threshold #1994

Closed
jasoncwik opened this issue Aug 28, 2024 · 0 comments
Closed

Incorrect multipart threshold #1994

jasoncwik opened this issue Aug 28, 2024 · 0 comments

Comments

@jasoncwik
Copy link
Contributor

Currently, if PutObject is executed with exactly partSize / minPartSize, it will execute a multipart upload with exactly one part. This is wasteful and causes performance issues, especially over the WAN, due to multiple round-trip operations.

Example:

  1. Create object exactly 16MiB
/tmp/minio-examples/examples/s3 $ dd if=/dev/random of=my-testfile bs=1M count=16
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.0712859 s, 235 MB/s
  1. Upload with sample PutObject call
/tmp/minio-examples/examples/s3 $ ls -l my-testfile 
-rw-rw-r-- 1 jcwik jcwik 16777216 Aug 28 09:27 my-testfile

/tmp/minio-examples/examples/s3 $ ./putobject | tee put1.txt
<snip>
2024/08/28 10:29:42 Uploaded my-objectname  of size:  16777216 Successfully in  421.786748ms

You can see there are 3 calls made for the upload, creating a MPU with a single part.

/tmp/minio-examples/examples/s3 $ grep -A 1 START-HT put1.txt
---------START-HTTP---------
GET /testbucket/?location= HTTP/1.1
--
---------START-HTTP---------
POST /testbucket/my-objectname?uploads= HTTP/1.1
--
---------START-HTTP---------
PUT /testbucket/my-objectname?partNumber=1&uploadId=ZGE0YzdiNmQtNzNhYS00MzlhLTgyMzEtZThhMjA2MGM3ZThkLmRjNGUwODUzLTcxY2YtNDhlYy1hZTJkLTAxZTYyNDcyNjYyNngxNzI0ODU5MDE5MjcwOTE2NzI3 HTTP/1.1
--
---------START-HTTP---------
POST /testbucket/my-objectname?uploadId=ZGE0YzdiNmQtNzNhYS00MzlhLTgyMzEtZThhMjA2MGM3ZThkLmRjNGUwODUzLTcxY2YtNDhlYy1hZTJkLTAxZTYyNDcyNjYyNngxNzI0ODU5MDE5MjcwOTE2NzI3 HTTP/1.1

I have a PR, will post in a few minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant