[filesystem] improve download and allow downloading of big files and parallel downloads. #4890

uniibu · 2019-04-12T06:44:33Z

Summary:
This PR improves downloading of files from Theia including:

downloading of big files without any limitation(other than the user's disk storage).
unique download link.
downloads can be paused and/or resumed.
allows parallel downloads in cases a user wanted to use a 3rd party downloader.
adds Copy Download Link menu to support the above.
works on chrome and firefox.
works on non-ascii file names and folder names.

TLDR;
Currently, there are two main concerns regarding downloads. One, the client download is handled in the client background via fetch which stores the blob in the memory. There are limitations using this approach, one, blob can only handle a specific size depending on the browser and this is prone to OOM(Out of Memory) error. Two, the backend that serves the file uses fs.readFile which is not very sufficient specially for big files.

To handle the first issue, instead of fetching the resource/file in the backend, the frontend instead makes an initial request to download a file, and the backend 'prepares' it and responds with a unique id to be used to download. The frontend then attaches this unique download link to the anchor tag with a download property and simulates a click. The download is now handled natively by the browser.

For the second issue,the simplest solution was to use a readable stream piped to the response so that the backend does not need to store the whole file in memory before serving it to the user.

Improvements, since this pr now allows downloading big project files/folders, some of us prefer to use a separate downloader which downloads the files in parallel. So I added support for parallel downloads using Range header. And lastly, to be able to use a separate downloader, you will need the download link, so I added a Copy Download Link menu option which works for singe and multiple(tar) files/folders. Cheers!

Signed-off-by: Uni Sayo unibtc@gmail.com

paul-marechal

Wonderful feature, looks really robust!

I tried a few things despite the link not being copied on Linux, and it works great! Single or multiple files, it archives it with what looks like the closest common parent, works in a multi-root workspace as well!

Although in a multi-root workspace, I tried downloading files from two relatively different places, and I ended up with an archive containing 2 archives. It works, but I wonder if this is intended.

Anyway, last but not least, maybe I missed it, but is there a way to list the downloads?
Either a dedicated view with the different links, or maybe just a quick-pick menu that would open the links once we select an item (since we are having troubles with the copy command)

IIUC you do track downloads already: https://github.com/theia-ide/theia/blob/91943cf34eb8837fb631d7c6119d499da07c6d94/packages/filesystem/src/node/download/file-download-collection.ts#L28-L89

I like these changes.

packages/filesystem/src/browser/download/file-download-service.ts

kittaakos · 2019-04-16T08:38:27Z

The status bar shows, I am offline after copying the download link for a > 4GB file:

Also, when I copy the download link for multiple files and paste it into the browser, I get

If copying the download link for multiple files is not supported, we should disable it on the UI.

uniibu · 2019-04-16T08:50:09Z

If copying the download link for multiple files is not supported, we should disable it on the UI.

Regarding the offline, I've experienced this as well on gitpod, but i have not experienced it on my own server. This is probably something of a server limitation? I'm not 100% sure.

And about the unable to download file, the links actually expire after 1 minute. I don't see any use case of pooling them together for later use, unless you intend to share it to someone else later on which is out of scope of this pr. Another reason for expiring the link is so that, it triggers the clean up of the files generated by archiving multiple files/folders.

kittaakos

but i have not experienced it on my own server.

It happened with my local setup.

And about the unable to download file, the links actually expire after 1 minute.

It wasn't related to timeout or something. When I select two nodes from the navigator, I select Copy Download Link and I paste the link to the browser, I got the error.

packages/filesystem/src/browser/download/file-download-service.ts

packages/filesystem/src/node/download/file-download-handler.ts

packages/filesystem/src/node/download/file-download-collection.ts

kittaakos

I have tried it in Chrome, it works nicely. Thank you for your contributions, @uniibu!
Please, see my comments.

packages/filesystem/src/node/download/file-download-endpoint.ts

packages/filesystem/src/node/download/file-download-handler.ts

packages/filesystem/src/node/download/file-download-collection.ts

packages/filesystem/src/node/download/file-download-handler.ts

packages/filesystem/src/node/download/file-download-collection.ts

packages/filesystem/src/node/download/file-download-handler.ts

This PR improves the download by letting the native browser handle it instead of fetching it in the background. This also adds a unique link which expires in 1 minute. Tested on chrome 73.0 and firefox 60.5 Signed-off-by: Uni Sayo <unibtc@gmail.com>

kittaakos · 2019-04-17T05:24:16Z

packages/filesystem/src/browser/download/file-download-service.ts

@@ -40,8 +37,6 @@ export class FileDownloadService {
    @inject(FileSystem)
    protected readonly fileSystem: FileSystem;

-    @inject(StatusBar)


From the CI:

[lint] ERROR: /home/travis/build/theia-ide/theia/packages/filesystem/src/browser/download/file-download-service.ts:40:1 - Consecutive blank lines are forbidden

kittaakos · 2019-04-17T05:26:49Z

packages/filesystem/src/browser/download/file-download-service.ts

    protected anchor: HTMLAnchorElement | undefined;
-    protected downloadQueue: number[] = [];


All symbols with protected and public (default) visibility are part of the API, if you change it, you have to declare your modifications as a breaking change in the changelog.

kittaakos · 2019-04-17T05:29:23Z

packages/filesystem/src/browser/download/file-download-service.ts

@@ -147,54 +142,68 @@ export class FileDownloadService {
        return this.deferredUpload.promise;
    }

+    protected handleCopy(event: ClipboardEvent, downloadUrl: string) {


All methods and functions with protected and public (default) visibility, must explicitly declare the return type. For instance, here, it is : void.

Please, apply this rule to all the methods you have introduced.

akosyakov · 2019-05-08T05:00:48Z

@kittaakos @marechal-p Is it good? Can it be merged?

kittaakos · 2019-05-08T06:54:00Z

@kittaakos @marechal-p Is it good? Can it be merged?

I can see unresolved conversations, and the build is broken; probably it is not ready for the merge.

uniibu · 2019-05-08T09:07:16Z

Hi guys, sorry for the delay, been busy lately with work. Ill try to finish this week. Cheers

akosyakov · 2019-05-27T07:18:09Z

Does it address #5197 as well?

@uniibu Do you have plans to work on it? Would you mind if someone else tries to finish it up?

uniibu · 2019-05-28T08:57:33Z

@akosyakov Yes for sure. I've been caught up with work and couldn't literally find time right now. So if anyone wants to continue it, it's very much fine with me :) Sorry again guys.

kittaakos · 2019-06-14T08:50:19Z

So if anyone wants to continue it, it's very much fine with me :)

👍

We continue the work here: #5466

akosyakov · 2019-07-04T06:39:53Z

closing it in favor of #5466

akosyakov requested a review from kittaakos April 12, 2019 08:59

paul-marechal reviewed Apr 15, 2019

View reviewed changes

packages/filesystem/src/browser/download/file-download-service.ts Outdated Show resolved Hide resolved

kittaakos reviewed Apr 16, 2019

View reviewed changes

uniibu force-pushed the download-big branch from e04cce6 to 8afd59d Compare April 16, 2019 16:10

kittaakos reviewed Apr 17, 2019

View reviewed changes

akosyakov mentioned this pull request May 8, 2019

Workspace goes offline when try gin to download 700MB-file gitpod-io/gitpod#545

Closed

kittaakos mentioned this pull request Jun 14, 2019

improve download, allow download of big files, works on ff and chrome #5466

Merged

akosyakov closed this Jul 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filesystem] improve download and allow downloading of big files and parallel downloads. #4890

[filesystem] improve download and allow downloading of big files and parallel downloads. #4890

uniibu commented Apr 12, 2019 •

edited by paul-marechal

Loading

paul-marechal left a comment •

edited

Loading

kittaakos commented Apr 16, 2019

uniibu commented Apr 16, 2019

kittaakos left a comment

kittaakos left a comment

kittaakos Apr 17, 2019

kittaakos Apr 17, 2019

kittaakos Apr 17, 2019 •

edited

Loading

akosyakov commented May 8, 2019

kittaakos commented May 8, 2019

uniibu commented May 8, 2019 •

edited

Loading

akosyakov commented May 27, 2019

uniibu commented May 28, 2019

kittaakos commented Jun 14, 2019

akosyakov commented Jul 4, 2019

		protected anchor: HTMLAnchorElement \| undefined;
		protected downloadQueue: number[] = [];

[filesystem] improve download and allow downloading of big files and parallel downloads. #4890

[filesystem] improve download and allow downloading of big files and parallel downloads. #4890

Conversation

uniibu commented Apr 12, 2019 • edited by paul-marechal Loading

paul-marechal left a comment • edited Loading

Choose a reason for hiding this comment

kittaakos commented Apr 16, 2019

uniibu commented Apr 16, 2019

kittaakos left a comment

Choose a reason for hiding this comment

kittaakos left a comment

Choose a reason for hiding this comment

kittaakos Apr 17, 2019

Choose a reason for hiding this comment

kittaakos Apr 17, 2019

Choose a reason for hiding this comment

kittaakos Apr 17, 2019 • edited Loading

Choose a reason for hiding this comment

akosyakov commented May 8, 2019

kittaakos commented May 8, 2019

uniibu commented May 8, 2019 • edited Loading

akosyakov commented May 27, 2019

uniibu commented May 28, 2019

kittaakos commented Jun 14, 2019

akosyakov commented Jul 4, 2019

uniibu commented Apr 12, 2019 •

edited by paul-marechal

Loading

paul-marechal left a comment •

edited

Loading

kittaakos Apr 17, 2019 •

edited

Loading

uniibu commented May 8, 2019 •

edited

Loading