Support S3 repositories with multipart upload/download disabled

[eamonryan]

I've noticed that the S3 resource does not function when the target S3 server is something other than Amazon and does not have multipart upload/download enabled.

For uploads, I'm currently seeing a behaviour where those work fine and files appear to be uploaded successfully via the s3-resource so that's good and may be related to #92 though that seems to be for 3.9.0 and we are testing against 3.8.0.

For downloads, the behaviour seen is that the download of a file using the s3-resource is marked as succeeded but has only downloaded 5MB of the file and left it truncated. See this screenshot:

This obviously causes subsequent pipeline steps to fail.

We found that downloads work fine on files smaller than 5MB as you can see here:

Since the default part size is 5MB this sounds suspiciously like it just goes with that default size and since the server does not have multipart enabled and does not send more parts, calls it good and marks it succeeded but you actually end up with a truncated file.

We can download the same files just fine via the s3cli manually, so this does appear to be localised to the s3-resource not handling this server configuration well.

Can we enhance the s3-resource to support this S3 server configuration? At the very least I feel the pipeline should not mark a 5MB truncated file as a successful download and we should note on the s3 resource and any relevant documentation that multipart is a required setting.

However, I've noticed that finding this setting is not always easy with the various S3 software vendors, some requiring patches to add the functionality and so it would be a better solution to just support either configuration if possible.