lower TUS max chunk size from infinite to 0.1 GB

[wkloucek]

Description

We've lowered the TUS max chunk size from infinite to 0.1GB so that chunking actually happens.

Related Issue

• Needs Bugfix: Make TUS capabilities configurable cs3org/reva#2136
• Related to Single big file upload with desktop-client fails if OIDC token expires #1205, fix TUS uploads with transfer token only cs3org/reva#1941

Motivation and Context

How Has This Been Tested?

• locally

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[butonic]

why would this fail? just because the POST request cannot stat the uploaded file when the upload finishes after the token has expired. IMO this is not the correct fix:

1. When the upload finishes the task of the client is done (he transferred all bytes) and then can immediately close the connection. at least this is what I understood is what @dragotin expects.
2. But that means the client cannot expect to get an etag. ... well we could roll a random etag and persist that in the extended attributes. then we could return this preleminary etag in the response... when postprocessing is done without changing the content that etag can be stored in the extended attributes
3. should we return the checksum as the etag? hm ... I would prefer a random string

now taking this one step further:
how should the etag behave when we return files that are currently being uploaded in the propfind listing?

1. we could list the preleminary etag in the propfind response
2. when some postprocessing changes the file the mtime and etag (and the checksums) would change ... allowing the client to sync back the change.

The client should be able to transfer as many bytes as possible. Without chunking. He should be able to resume the upload and discover how many bytes are already on the server. For that the transfer token needs to be valid for a longer period of time. The TUS protocol has an expiration extension that lets clients know how long they can resume an upload: https://tus.io/protocols/resumable-upload.html#expiration The Upload-Expires header would be determined by the transfer token lifetime.

[wkloucek]

Max chunk size default should not be lowered if not needed. The upload needs to be fixed. In the meantime one could work around by lowering the chunk size to mitigate expiring tokens.

[dragotin]

The max chunk size has to be limited when we go in production with the product, maybe to 100 MB. It has to be configurable, but let us have a reasonable default pls to provide a robust product.

Otherwise we end up with support load with every new installation, because in most installations there will be a limit.

[wkloucek]

The max chunk size has to be limited when we go in production with the product, maybe to 100 MB. It has to be configurable, but let us have a reasonable default pls to provide a robust product.

Otherwise we end up with support load with every new installation, because in most installations there will be a limit.

@butonic is right, we should not set a arbitrary limit without reasoning. In tus/tus-resumable-upload-protocol#93 (comment) are some examples. But there is no "hard limit" for a "general usecase". It highly depends on the specific usecase... That's why it is configurable.

[dragotin]

Yes please, ship that!

@TheOneRing agreed.