feat(web): determine duplication of upload on client #8825
Conversation
truongsinh
changed the title
|
I like the idea of the checksum header, but I think it might come with a few caveats. Can you create a discussion either here or on discord with a proposal so we can discuss it more in depth? |
Will do. Meanwhile, I hope we can get this merged soon. The checksum header would need most of this PR (calculating the checksum on the client) anyway. There's a prettier build fail. Please let me know if we need any other change for this PR to be merge-ready. |
|
Why exactly do we need/want to try to implement hashing in the web client...? |
I would argue it is the same reason we implemented hashing on mobile client or CLI client (hence, feature parity). That is, to be bandwidth efficient. If a 2GB files with the hash already exist on server client-hashing, the client saves 2GB bandwidth because it does not need to transmit those data. |
|
When you talk about bandwidth I assume you mean while accessing a remote service, since this isn't really a concern on a local network. I would argue that we probably don't even want exact parity between the clients. The mobile app, for example, has an offline database and sync process which enables offline usage. Also, the mobile app doesn't use the timebucket apis. It loads everything locally, while the web client depends on the remote server for everything. In general they serve completely different use cases and I don't know if there is valid use case where remote users are going to use the web client to upload assets (that are large files) that are already on the server. |
Not necessarily. For instance, even with WAN setup assuming a 10MBps upload speed, uploading a single 2GB GoPro file could take as long as 26 minutes. On a LAN, using a somewhat repurposed server, this duration reduces to about 5 minutes for a 2GB file. My typical use case involves uploading large volumes of footage (GoPro, Drone, etc.) as soon as possible when traveling. Often, I'm unable to upload everything daily due to bandwidth constraints. Once home, I continue uploading to ensure all content is transferred to Immich, frequently spending up to 30 minutes to upload hundreds of files, only to discover that fewer than 10% are new assets.
So why can't web app (esp moving toward PWA) have the same 😉. Anyway, it's a different topic. When talking about feature parity, I mentioned both Mobile and (Desktop) CLI as well. What is your thought on feature disparity between Web app and CLI upload? Currently, using the CLI can mitigate some of my issues with bandwidth and time, both on LAN and WAN, though it’s not the most user-friendly solution to force upon users. Also, keep in mind web client hashing is pre-req for #8847 as well. On the other hand, assuming from your perspective, you see no value of web client hashing. Could you express any concern or harm having this on Web client? That is, I want to learn your perspective regarding any reason "not to" have this on web client. Probably if we can sort out your concern, we can get it merged even if some of us don't find value in it. |
We've generally had the stance that for anything beyond uploading just a handful of assets, people should use the CLI rather than the web. The CLI can do folder crawling, make better use of parallelization etc, so for large uploads it's better suited. |
It seems what you describe is a large number of small files ("handful of assets," "folder crawling"). My use case is the opposite, a small number of large files (Drone, 360, GoPro) in a single folder (DCIM). Say 10 files of 4GB (40GB), not 20'000 files of 2MB.
It might be another topic, but given current browser technology, I don't think we are limited to anything in terms of parallelization/concurrency. Probably, it's more about whether we want it in the browser and the maintenance effort. |
|
Do you mind doing some performance benchmarks to see the implications of hashing vs no hashing? Both on a mobile client and on a desktop client, on a local and remote connection? The idea is basically in most situations there is little value add to hashing when uploading files on the web. Basically this means unnecessarily hashing files in most situations. There is a specific situation that you are describing: uploading large, duplicate videos on the web on a slow/remote connection. The consequence is (in addition to more code to maintain/debug/etc) that everything else has to be hashed, always. Maybe a middle ground would be something like adding a user preference for pre-hashing uploads. It is off by default. If enabled, it does client-side hashing (when available). This is how the CLI is supposed to work afaik. It has pre-hashing options, but that can be disabled (if you know everything is new). FWIW I would have preferred to see #8847 built first as this is something the mobile and cli can take advantage of right now. |
Yes, I can spend some time doing so. However, I am not clear on what metrics you are looking at. This is my assumption, please correct: Setup:
Actually, is it fine if I do web and CLI only, but not mobile? It's easier for me to tweak typescript code on server/cli/web, rather than flutter and rebuild android app. Test data:
Yes, this makes sense. I would make that option to be ternary:
AFAIK we already do hashing (on server) regardless, it's about moving hashing (from server) to (and trust) client if the operator wants to. |
This isn't really an option. If we rely on the client and it provides a bad hash, a bunch of existing assumptions go out the window. |
I think the point about performance is will having a file on client take longer for a mix of small and large files than how long it takes the way immich currently works. So how will this PR affect performance? FYI, I'm not part of the immich team, I'm just adding my idea. |
|
We're not going to turn the server hashing off either way. I'm fine if you want to skip the benchmark and just add clientside hashing to the web behind a flag that defaults to false. The server PR for dedupe by hash checksum can be separate from the mobile app using it. There is no expectation that you do the mobile implementation either |
That simplifies things. |
|
Trying to balance both sides here... tldr; add size check prior to hashing, make hashing conditional clientside, and if serverside hashing is also enabled, only hash when the file is between a user-defined threshold |
I actually implemented this, but it doesn't work. Turns out the browser basically dies when the server terminates the request early. Chrome does not provide a way to access the response body (or anything related to the response at all) and just shows a connection reset network error. https://stackoverflow.com/a/18370751
There are security implications to the optimization, so we won't be doing this one either. |
|
This seems to work pretty good. We can leave it enabled by default for secure contexts, etc. and see how it goes. |
This PR brings feature parity of determining asset duplicates to web (only via HTTPS, not HTTP, due to availability
cryptoAPI).That being said, the whole design of
checkBulkUploadfollowed byuploadrequiring 2 roundtrips to the servers has performance and functionality problems. We instead can have a single endpoint that does both checks and uploads using headers for example,x-immich-checksum. Server can then process headers first, and determine whether to consume body stream, or not at all, and return 201. With that approach, we can save bandwidth for duplicated assets even from different users/owners. That is, on server, we can check the whole DB for the checksum. If checksum is found, but the assets belong to another user, we can do hardlink or copy --reflink=always (instead of uploading the whole file again), and create DB record. It would save both bandwidth and storage.