Removing downloaded cache takes very long time

[levz]

//Version
2.8.0

//Source code
Not provided - general question

//Description
I use SimpleCache to download HLS donwload streams as desribed in
https://medium.com/google-exoplayer/downloading-streams-6d259eec7f95

And removal of these downloaded cache takes really long time. (Several hours for 1GB of data)
I suppose this is because of SimpleCache saving every chunk to a separate file and so it then takes a lot of time to remove all one by one.

My question is - can I somehow make SimpleCache save the whole thing into one big file?
I have tried creating my own version of CacheDataSink and sending everything to one file, but it seems to break the content index and download throws exception.

Is there some way to do it with SimpleCache or does saving to single file require a new Cache implementation?

[ojw28]

Most of the overhead is caused by repeatedly updating the cache index on disk. We should batch up the updates instead, and/or come up with an on-disk representation that's more efficient to update.

[levz]

Here is a workaround, until there is a better solution.

I ended with cloning a couple of classes to create a separate SimpleCache folders for each download.
I created clones of DownloadManager and DownloadService. (Could not override them, since they are final).

In MyDownloadManager I create a new helper with a new SimpleCache folder for each download.
And in task run() method, for remove action I simply remove the whole folder. This works fast since there
is no cache index update involved.

[jandresakis]

@levz I see the same issue and was hoping not to do what your talking about in your solution. Did you have to copy/refactor most of the SimpleCache and task logic or just pieces?

[levz]

@jandresakis

No, SimpleCache stays the same.

But you will need to clone DownloadManager with changes that will 1) create a new SimpleCache folder for every download 2) delete cache folder (related to the specific download) from disk when necessary.
And you will have to clone DownloadService to make it work with cloned DownloadManager.

[jandresakis]

@levz thanks for the info. @ojw28 are there currently any plans to do what levs is speaking of? I would like to avoid refactoring/copying and private classes in the exo library if possible to avoid maintaining something that you will provide in the future any way.

[ojw28]

We're planning to simply not write the cache index to disk on key removal (because it's not actually necessary to do so). This should happen pretty soon (e.g. before the end of the year).

[jandresakis]

@ojw28 any updates as to a possible eta for that fix for the slow download removal to find its way into a release?

[ojw28]

This is fixed in the dev-v2 branch. We don't have an ETA for a release that will include it. We're making some pretty big download changes at the moment, so we'll need to decide whether to package this up in a release that comes before those changes all land, or whether to wait and release everything together.

[ojw28]

This will be included in 2.9.3. I tested the change above with an HLS download consisting of 1125 segments, and removal time was reduced from ~16 seconds to ~250 ms.

[jandresakis]

@ojw28 thanks for the early christmas gift ;)