Implement async progress reporting for upload / assembly

[butonic]

When finishing a large upload the assembly of a 40 GB file might take a long time. Currently, tus has no extension for this and the concatenation extension explicitly states:

The response to a HEAD request for a final upload SHOULD NOT contain the Upload-Offset header unless the concatenation has been successfully finished. After successful concatenation, the Upload-Offset and Upload-Length MUST be set and their values MUST be equal. The value of the Upload-Offset header before concatenation is not defined for a final upload.

transloadit is mentioning the /assembly endpoint https://github.com/tus/tusd/wiki/Uploading-to-Transloadit-using-tus and also that it uses a socket.io server that can be used to track the progress: https://github.com/tus/tusd/wiki/Uploading-to-Transloadit-using-tus#getting-progress-and-more-events-over-web-sockets

The tusd handler has a hook mechanism that can be used to expose progress.

Or we could let clients make HEAD requests against the upload and return a read only progress property in the metadata ... as a quick hack ...

[butonic]

thx @michaelstingl for asking about this

[butonic]

@felixboehm I wonder if we can use websockets for clients and how this integrates with a push server ... maybe related appleboy/gorush#194

[michaelstingl]

I think as a fallback, we'll always need pull/polling-based approach. But it would be cool to have the more fancy options too.

[butonic]

AFAICT an upload has multiple phases:

• transfer (bring bytes to the storage, but not yet accessible to the user)
• assembly (optional, if multiple chunks need to be assembled)
• preprocessing (optional, eg virus scanning, content manipulation)
• finishing (make available to the user)
• postprocessing (content indexing, thumbnail generation ...)

The tus protocol only covers the first and second phase, transfer and assembly.

When no preprocessing happens, we can return the etag and new fileid immediately.
While preprocessing is in progress we want to show the progress when listing the containing folder. This might be a custom PROPFIND request or property? during progress the etag and fileid won't hale changed, but we might return a progress property? Similar to the async / lazy ops header, which returns a location that clients can use to poll for changes.

AFAICT we need to either extend the FinishUpload method of the tus backend implementations or register hooks that can start the preprocessing properly, keep track of progress and finalize the upload by making the file available to the user / moving it to the correct destination.

postrocessing should not produce anything the clients need.

But for all of this we need a workflow. AFAICT the client may not get a new etag until the upload has passed the finishing phase.

[PVince81]

as discussed, we'd need to invent a new TUS extension for async.
the extension would return extra headers to signal that it's going into async mode (like Lazy-Ops)

[refs]

@butonic want to keep this open?

[butonic]

cc @dragotin This boils down to a state machine for uploads. Then file listings could return the state of a file as well: transfer, assembly, preprocessing, finishing, postprocessing. See #214 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

[fschade]

Please open again if the ticket is still relevant