Large file uploads not working under Cloudflare

[orangewolf]

          Team, I'm adding to this ticket because we have a recurring error that feels related to the issues I first noticed with the importer at the top of the ticket. Large file sizes are being rejected from the repository. This has only happened since CloudFlare. I was able to work around the issue last week with the images from the importer, but in general we need to upload large file sizes because we deliver high resolution TIFFs to researchers around the world. Today, I have to work around Hyku and use a different service to deliver files to a user in France because the repo is rejecting the TIFFs I tried to upload.

The repo is rejecting all files over 150 MB, according to my student worker. This file size is obviously wildly too small for our purposes. Can you help us fix this issue fairly urgently? It is a blocker with everything we try to do.

Work I needed to add a TIFF to:
https://adl.b2.adventistdigitallibrary.org/concern/generic_works/20213926_illustrations_of_miller_s_views_of_the_end_of_the_world_in_1843/

Error message and failure to upload:

Dropbox link to the TIFF:
https://www.dropbox.com/scl/fo/mipr9ujtltqgxempmgdrp/APxFmQU3El5vlFC_pRX9zJI?rlkey=c2dz52x6zrowg6knytoe26vnw&st=c314ffy5&dl=0

Originally posted by @KatharineV in #848 (comment)

Testing Instructions

Manually create a work and upload a large tiff image. Ideally use the one the client previously had issues with.
See that the upload completes without error.
Save the work, wait for the jobs to complete.
Ensure the file gets attached in the work's items section and displayed in the work's show page.

[ShanaLMoore]

cloudflare limits how big the files can be through the proxy; it blocks it because of poor performance issues otherwise. There are two path forwards:

1. implement chunk uploading w jquery file uploading and do the same for bulkrax. This would be a big improvement for Hyrax and all of our apps. ~ 1 day
2. Bail the current implementation because the plugin has been deprecated. Switch with something more current. We're not sure if we'll find a chunk file uploader that works with carrier wave and modern rails. This feels like a bigger lift. ~ 3-4 days

[ShanaLMoore]

https://github.com/blueimp/jQuery-File-Upload -> chunked uploads is a feature. this gem is archived/no longer supported
https://github.com/tors/jquery-fileupload-rails
we need to look at the middleware and the resumable=jquery=file upload. that will have the pieces we need.

• look at: courses/upload.html.erb #fileupload

checkout do_upload in the controller

https://github.com/vgantchev/rails-resumable-jquery-fileupload

TODO:

extract the code from https://github.com/vgantchev/rails-resumable-jquery-fileupload

How does this tie in with carrierwave?

[ShanaLMoore]

QA RESULTS: ❌ FAIL

tested on STAGING

Sometimes files fail the valkyrie characertization job. After further investigation, it seems like these files had issues when chunk uploading. We set the chunk size to 10MB, and at the time it's trying to run the im_identify commands we are dealing with the last chunk instead of the full file. So far this has only been observed in staging.