Transfer multiple files and folders

[tanmoyAtb]

closes #1509

a few pointers about this PR -

• I am sharing files one by one, download - upload and next file. Downloading all files first was jamming my memory.
• I did not handle the conflict error, when the other shared folder already has a file with same name. When the server error structure changes, going to do it then.

[render]

Your Render PR Server URL is https://storage-ui-stage-pr-1606.onrender.com.

Follow its progress at https://dashboard.render.com/static/srv-c5cspad0maldmgci5ql0.

[render]

Your Render PR Server URL is https://files-ui-stage-pr-1606.onrender.com.

Follow its progress at https://dashboard.render.com/static/srv-c5cspat0maldmgci5r1g.

[render]

Your Render PR Server URL is https://gaming-ui-stage-pr-1606.onrender.com.

Follow its progress at https://dashboard.render.com/static/srv-c5cspbd0maldmgci5rj0.

[asnaith]

Hey Tanmoy - just checked this out and did testing around the following scenarios

✅ User can share multiple selected files
✅ User can copy multiple shared files to their home directory
✅ User can copy multiple shared files to another shared folder
✅ User can copy a shared folder to their home directory
✅ User can copy a shared folder to another shared folder

All working nicely with the singular toast too.

A couple of minor feedbacks -

• Should the button be "Copy selected" when it's being displayed within the context of a shared folder? We used "copy to" (kebab menu option) and "copy file" dialog headers in this context for single files as we thought sharing from a share sounded like odd terminology.

• Very minor but now that multiple files can be selected to create a shared folder, then the "Keep original file" label should ideally become plural, "Keep original files" when more than one file is selected.

[tanmoyAtb]

Hey Tanmoy - just checked this out and did testing around the following scenarios

✅ User can share multiple selected files ✅ User can copy multiple shared files to their home directory ✅ User can copy multiple shared files to another shared folder ✅ User can copy a shared folder to their home directory ✅ User can copy a shared folder to another shared folder

All working nicely with the singular toast too.

A couple of minor feedbacks -

• Should the button be "Copy selected" when it's being displayed within the context of a shared folder? We used "copy to" (kebab menu option) and "copy file" dialog headers in this context for single files as we thought sharing from a share sounded like odd terminology.
• Very minor but now that multiple files can be selected to create a shared folder, then the "Keep original file" label should ideally become plural, "Keep original files" when more than one file is selected.

Thanks for the amazing feedbacks @asnaith . I'll recheck on the terminology.
To add to the test cases, you can also test sharing multiple files and folders together.
Try cancelling sharing while its on progress for multiple files and folders.

[Tbaut]

this PR has a good amount of awesome things and bonus clean ups, thanks a lot!

All and all, I'd love it if we could have consistent logic in our implementation. If we have a downloadMultipleFiles that uses a reduce to go over each file, and update the toast, I think it'd be great to have it in the implementation as well. Now if you think it's too fancy/hard-to-read/anything, I'm happy to rewrite it with a forEach or something. Just having those 2 using the same type of thinking would be good.

[Tbaut]

I always find a forEach a little more elegant than a for loop. Although here a reduce would be even more elegant so that we don't have to keep things like successCount or totalProgress. The function downloadMultipleFiles is actually built this way.

[tanmoyAtb]

I feel you. forEach isn't blocking, so used a for loop here.
I actually tried using a reduce first. but I was having a hard time getting the progress calculations to work. I'll give the reduce another go, its definitely a good choice for this implementation.
I'll bring you in if I get stuck

[Tbaut]

sure, feel free to reach out, reduce is a little annoying with TS sometimes.

[FSM1]

Ahhh and you wanted it to be blocking so that memory usage doesn't go through the roof?

What about using something like this (shamelessly stolen from SO):

async promiseAllInBatches(task, items, batchSize) {
    let position = 0;
    let results = [];
    while (position < items.length) {
        const itemsForBatch = items.slice(position, position + batchSize);
        results = [...results, ...await Promise.all(itemsForBatch.map(item => task(item)))];
        position += batchSize;
    }
    return results;
}

The only other thing to add then would be the handling of individual failures gracefully, and we could also use different batch sizes depending on device...

[FSM1]

Here is some sample array.reduce code

function runPromisesInSeries(bigArray, getInfoForEveryInnerArgument) {
  try {
    return bigArray.reduce(async (acc, cItem) => {
      const results = await acc
      const data = await getInfoForEveryInnerArgument(cItem)
      results.push(data)
      return results
    }, Promise.resolve([]))
  } catch (err) {
    throw err
  }
}

[asnaith]

Thanks for the amazing feedbacks @asnaith . I'll recheck on the terminology. To add to the test cases, you can also test sharing multiple files and folders together. Try cancelling sharing while its on progress for multiple files and folders.

Nice, I see the toast stating how many were shared successfully prior to the cancellation 👍

[asnaith]

At the moment it's possible to select and share an empty folder. We say that the share was successful but the empty folder is never actually created within the share.

It could lead to some confusion in edge case scenarios where someone wants deliberately share a collection of empty folders (eg thinking of a manager setting up the initial folder structure for their team to populate with files).

[tanmoyAtb]

At the moment it's possible to select and share an empty folder. We say that the share was successful but the empty folder is never actually created within the share.

It could lead to some confusion in edge case scenarios where someone wants deliberately share a collection of empty folders (eg thinking of a manager setting up the initial folder structure for their team to populate with files).

Yes this is an interesting case, I'll update the logic to say No files too share when there are just empty folders !

[Tbaut]

I'm only nit picking at this point, it's a great one 🚀

[Tbaut]

I'm pushing it all the way :) This is still a relic of the past implementation.
If we go all in a reduce, then why not have this integrated the same way as the totalProgress? To do this, change the totalProgress: number to progressState: {totalProgress: number, successCount: number} and return in the reduce {totalProgress: previousProgress + item.size, successCount: successCount++}

[tanmoyAtb]

Yup, thats a very valid approach, but the problem is I need the successCount outside the try... block with the reduce, for example when the user hits cancel.

[tanmoyAtb]

I did try to get the successCount this way, but I couldn't get the successCount to pass through to the scope of the catch block !

[Tbaut]

aaah I didn't realize you're using it outside, my bad, then it's all good :)

[asnaith]

This is great, I tried to share 3 files to a shared folder where one of them already existed. It failed on the one that was already there but continued to upload the others. Nice!