[Bug] 2.15 - Reported Synchronisation Taking Much Longer (cards and media)

[david-allison]

Thing I noticed after today's update: synchronization takes much longer.

My use-case: I have little over 100'000 cards. Sometimes on desktop I run add-on that adjusts ease factor for all cards.

After that synchronization is quite big. But it become definitely longer after today's update.

/u/szalejot (reddit)

[david-allison]

Note: This is not media-related, just card data [edit - no it's both]

[mikehardy]

Strangely - this is media related for me. I'm experiencing this right now, it is horrific on media sync. The collection was pretty slow as well. When I first saw this reported I blithely said "I don't know, it seems to saturate my 25Mbit pipe?" but at least for a real device (Pixel 4a) at the moment - hooboy this thing is crawling through sand mixed with cold molasses slow

[david-allison]

The media sucks, and has done for a long time (imagine 3GB of AnKing). One of the best parts of the Rust conversion will be moving it to the background like in Anki Desktop.

From memory, you don't sync media much. Is this a regression, or is it also slow for 2.1X as well?

[mikehardy]

regression for me, it used to be a bit stop/starty because of the 25-files-at-once thing but it still saturated my downlink. Now it's crawlin'. I haven't attempted to go back to 2.14, I was just setting up a new phone as one of the things that happens when your mom visits you in your country where there is no postal system or economical package delivery from the US is that she brings all the goodies. Going through 1st time setup on a bunch of new / replacement device :)

[github-actions]

Hello 👋, this issue has been opened for more than 2 months with no activity on it. If the issue is still here, please keep in mind that we need community support and help to fix it! Just comment something like still searching for solutions and if you found one, please open a pull request! You have 7 days until this gets closed automatically

[mikehardy]

This is definitely an issue

[albsch]

I am affected, too. Normal usage (desktop + android app) is becoming annoying with a big deck with media, because continuing on the android app after desktop use introduces a long (up to 10 minutes) sync.

[mikehardy]

we definitely understand, I pinned this to make sure it never stales out - as mentioned by David

The media sucks, and has done for a long time (imagine 3GB of AnKing). One of the best parts of the Rust conversion will be moving it to the background like in Anki Desktop.

And all I can do is say this is an area of serious hard work, it is happening:

https://github.com/ankidroid/Anki-Android/pulls?q=is%3Apr+rust+author%3Adavid-allison-1

It's just a massive massive effort, and critical to get right (it's your card collection access method after all...) so takes time. We appreciate the patience

[GypsyAcidQueen]

Just chiming in to say that this is affecting me too, and to keep the thread alive!

[mikehardy]

Yep - part of the conversion is handling the new V16 schema, and 2 more of those massive pull requests just went in yesterday, with another pending https://github.com/ankidroid/Anki-Android/pulls?q=is%3Apr+author%3Adavid-allison-1+sort%3Aupdated-desc+

It's not easy to do all this stuff right but David's chewing through it at quite a rate

[mikehardy]

For unrelated reasons I had to rebuild my mobile device yesterday, and my experience with media sync is incredibly slow.
In particular to move this issue forward: it's a local bottleneck and has nothing to do with the network. How did I infer this? If you have a collection that has a lot of unused media, you will download it from AnkiWeb (oh so slowly...as we know) but then if you check media you will get a report that you have in my case 6000+ media items that are unused. If you then select to delete them it will take maybe 10 minutes on my OnePlus 6T (a pretty good phone....). I don't believe the network is involved there at all? So this is something about local database interaction. Then when you go to sync those changes (deletes) up to AnkiWeb that is slow also.

So it should be possible to make a performance test for this locally (create collection with unused media, run the delete unused media, expect NN media/second throughput) and profile that + work through this

[david-allison]

Fixed in #9249, it's writing to a "publicly visible" directory that's taking time.

Significantly faster if

• Copy /AnkiDroid to /storage/emulated/0/Android/data/com.ichi2.anki/col
  • This requires a PC and USB
• Set Settings - Advanced - AnkiDroid Directory to /storage/emulated/0/Android/data/com.ichi2.anki/col

Note: Uninstalling AnkiDroid deletes the folder

[mikehardy]

what!? is there some documentation on that ? Is it because of MediaScanner ?

[mikehardy]

(I ask because there are ways to disable it like dropping a .nomedia file in there?)

[david-allison]

what!? is there some documentation on that ? Is it because of MediaScanner ?

My belief is that there's a massive IO performance penalty due to the security associated with enforcing scoped storage (saw a Reddit thread with 6x storage slowdown on Android 11 which tipped me off).

Moving files into the app's private directory is much faster (probably still slower than Android 10).

I got a massive media sync speedup with the directory change above.

[mikehardy]

Fascinating.

[Hxppdv]

My belief is that there's a massive IO performance penalty due to the security associated with enforcing scoped storage (saw a Reddit thread with 6x storage slowdown on Android 11 which tipped me off).

I tried disabling fuse on a rooted phone to bring back support for the old sdcardfs. So far it seems to be working and the sync speed is much faster. It seems that, at least in my case, the slow sync problem is related to fuse aka Scoped Storage.

[BrayanDSO]

Root cause is going to be solved with #5304, so I'm closing this