Allow streaming upload also for unknown length streams

[mifi]

Initial checklist

• I understand this is a feature request and questions should be posted in the Community Forum
• I searched issues and couldn’t find anything (or linked relevant results below)

Problem

Yes companion now streams most files without first storing them when setting COMPANION_STREAMING_UPLOAD=true, however it still downloads the whole file first when encountering a file of unknown length (missing content-length header):

uppy/packages/@uppy/companion/src/server/Uploader.js

Line 288 in f4dd3d5

if (this._needDownloadFirst()) {

This happens because for some of the uploaders we need to know the size of the file before uploading. I know that this happens for the Google Drive plugin when uploading files from Google Docs (export). It can also happen if someone uses the URL plugin to download/upload a file which doesn't have a content-length.

This is problematic for large files (e.g. 100gb) because it requires a lot of storage space.

Solution

When using an uploader that supports it (tus, xhr/multipart, s3?) we can upload directly without knowing the size.

Alternatives

leave it

[Acconut]

Just repeating my comment to add more info for tus:

Companion does not even have to count the bytes on its own. You can pass tus-js-client a readable stream while enabling uploadLengthDeferred and setting chunkSize. Then it will count the bytes on its own and let the server know the size once the readable stream closed.

It's worth noting that this feature requires the tus server to support an optional extension. While most tus server provide this, it is not always the case. So maybe an option to disable this for tus uploads is helpful.

[mifi]

I just tested and it seems to be working! Will create a PR