feat(server): efficient full app sync

[fyfrey]

Adds two new endpoints (in a new controller + service) to sync assets for the mobile app.

Compared to the old approach, these DB queries do not use offset/skip and can make use of the available indices. This should prevent the server/DB from overloading during initial/full app assets sync.

Further, the fast/delta sync now includes all owned+partner assets (thereby reducing the number of network requests and full table scans!)

This PR does not include the mobile changes. These will merged separately after 1-2 releases/weeks have passed to avoid all apps from breaking because the server has not yet been updated.

[cloudflare-workers-and-pages]

Deploying immich with    Cloudflare Pages

Latest commit:	8074119
Status:	✅  Deploy successful!
Preview URL:	https://88c8b8cd.immich.pages.dev
Branch Preview URL:	https://feat-server-efficient-sync.immich.pages.dev

View logs

[jrasm91]

This looks pretty good. We're trying to avoid putting everything into the asset controller though. What do you think about splitting these new endpoints out into a controller and service specifically for syncing instead? Maybe endpoints like /sync/assets-full and /sync/assets-delta or similar?

[danieldietzler]

LGTM

[alextran1502]

Hello @fyfrey, thank you for another high-quality PR. Can you add some words to the description about how the full sync vs delta sync works? What are the different mechanisms used between the two?

[fyfrey]

Sure, @alextran1502! It's actually the same as the current mechanism (just more performant on the server side):

Full sync: The app gets all asset information, compares this list with the state in it's database and upserts/removes entries as needed. The list of all assets is fetched in chunks per user, but still made into one giant list per user on the client (so, for 100k+ assets this is still an issue).

Delta sync: The app gets all changed and deleted assets for the user and partners. The server calculates this using the updatedAt property and the audit table (for deletes). So the app retrieves a comparable small set of changes regardless of the library size. It updates its database accordingly.

The app always tries to do a delta sync and only uses the full sync as a fallback: For initial sync when the client DB is empty, if there are too many changes or if the last sync is too long ago.