Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI doesn't work satisfactory when adding 10k files #4389

Closed
1 task done
mifi opened this issue Mar 29, 2023 · 1 comment · Fixed by #4402
Closed
1 task done

UI doesn't work satisfactory when adding 10k files #4389

mifi opened this issue Mar 29, 2023 · 1 comment · Fixed by #4402
Labels

Comments

@mifi
Copy link
Contributor

mifi commented Mar 29, 2023

Steps to reproduce

  1. Use a provider like Google Drive
  2. have a folder with 10000 files
  3. select the folder
  4. click Add

Expected behavior

It should show some kind of progress to the user and then allow them use the UI and to upload the files within a reasonable amount of time

Actual behavior

It takes about 5 minutes to load all files (due to pagination on the api towards Google) this is expected and unavoidable - with #4388 this is OK because the user will be able to see what's happening

Then it takes about 2 minutes for all the addFile calls to finish

  • maybe we can use core's addFiles instead (ok done!)
...
Adding remote file
loggers.js:15 [Uppy] [22:14:21] Added file: p599998.png, uppy-p599998/png-1e-image/png-327, mime type: image/png
...
about 20 000 lines
...

Then there's two errors that get spit out:

TransitionGroup.js:246 Uncaught TypeError: Cannot read properties of null (reading 'componentDidLeave')

Then the whole DOM/UI freezes for about 5 minutes and you can't click anything or scroll the list. Only the first few files are shown, without any thumbnail.

After 5 minutes you can scroll again, and you may now start the upload.

Maybe we should have an e2e test for this.

Note: Similar problem also for Drag/drop local files.

@mifi
Copy link
Contributor Author

mifi commented Apr 5, 2023

I just tried 11k files from remote provider (google drive) and now it actually works (after my big PR #4384 was merged). It takes about 7 seconds from the files are finished loading until they show up in the list. so UI is blocked for 7 seconds, and after that it's fully responsive (on Chrome). but I have a MBA M2, so maybe it's cheating. will test some more and try to get the block time down

mifi added a commit that referenced this issue Apr 5, 2023
don't do it for every file added, as it's very slow
instead do the check at the end when all files are added.
this allows us to easily work with 10k+ files
fixes #4389
mifi added a commit that referenced this issue Apr 15, 2023
* show how many files are added when loading

remake of #4388

* add french (cherry pick)

* implement concurrent file listing

* refactor / fix lint

* refactor/reduce duplication

* pull out totals validation

don't do it for every file added, as it's very slow
instead do the check at the end when all files are added.
this allows us to easily work with 10k+ files
fixes #4389

* Update packages/@uppy/core/src/Uppy.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* make restricter.validate validate everything

instead make more specific methods for sub-validation
also rename validateTotals to validateAggregateRestrictions

* improve errors and user feedback

- handle errors centrally so that we can limit the amount of toasts (informers) sent to the users (prevent flooding hundreds/thousands of them)
- introduce FileRestrictionError which is a restriction error for a specific file
- introduce isUserFacing field for RestrictionError

* fix performance issue reintroduced

* improvements

- show "%{count} additional restrictions were not fulfilled" for any restriction errors more than 4
- refactor/rename methods
- improve ghost logic/comments

* improve performance when uploading

- introduce new event "upload-start"  that can contain multiple files
- make a new patchFilesState method to allow updating more files
- unify "upload-start" logic in all plugins (send it before files start uploading)
- defer slicing buffer until we need the data
- refactor to reuse code

* fix e2e build issue

* try to upgrade cypress

maybe it fixes the error

* Revert "fix e2e build issue"

This reverts commit ff3e580.

* upgrade parcel

* move mutation logic to end

* remove FileRestrictionError

merge it with RestrictionError

* fix silly bug

looks like the e2e tests are doing its job 👏

---------

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant