Feature Request: Allow adjustment of the minimum S3 upload part size

[rvrangel]

Feature Description

Allow operators to define the minimum size of the S3 upload part, in case there is a desired to avoid too many roundtrip requests to upload a file, while lowering the number of PUT requests made to S3

Use Case(s)

We have had issues with S3 throttling our backups because of the number of requests we make to S3. Because the way the S3 upload part size is calculated, for small shards it means we are dividing the backup in a huge number of parts unnecessarily.

As an example, a 50GB shard will have around 10k parts of 5MB each. When you have thousands of tablets backing up, this adds up and you can have a huge influx of requests. Each S3 bucket/partition can only handle around 3500 PUT requests, so every now and then we see those being throttled.

It would be nice if we could set a minimum part size instead, so if the normal calculation we do results in a smaller size, it gets overwritten, but if the calculated size is over it (specially on huge shards that are TB sized) we will still use that value to be able to go over the 5GB limit per file/part