Debug and fix massive APK size, new in v27.170.

[chrisbobbe]

It's been reported that our APK has gotten really big. The latest beta, v27.174, is 126 MB.

Looking at the GitHub releases, it went from 41.6 MB in v27.169 to 126 MB in v27.170. The most likely-looking thing in the v27.170 release notes is the RN v0.64 upgrade, #4426.

[chrisbobbe]

Hmm!

I ran yarn; tools/android apk at these two consecutive commits, and found a big jump in the size of android/app/build/outputs/apk/release/app-release.apk:

131881527 at 9399897
46433271 at 506e712

So that AGP upgrade is looking like a likely culprit.

[chrisbobbe]

At 506e712, I've been trying to pinpoint what AGP version the issue starts in. I've just tested 3.6.0 and 3.6.4 there, but I got build failures with different outputs on both of those.

[chrisbobbe]

On main, I've tried updating to 4.2.2 and unfortunately the APK was still huge.

[gnprice]

Thanks for the debugging! Continued in a chat thread.

[gnprice]

Here's a (belated) write-up of what we learned in that debugging thread, with some small further points. I'll then file specific issues for subtasks.

The figures below are from the latest release, v27.181.

The APK has a total size of 131988583 bytes, 132.0MB. Of that, the sizes reported by unzip -lv add up to 131.5MB. (The rest is presumably some combination of alignment-padding and metadata.)

The great bulk of the size of the APK comes from the native libraries, .so files:

$ unzip -lv archive/181.apk | head -n -2 | tail -n +4 \
    | grep '\.so$' | perl -lane '$t += $F[2]; END { print $t; }'
124218432

That's 124MB.

• One major factor there is that our APK has four copies of each native library, for the four different ABIs.

  We could instead build a separate APK for each ABI. This would be pretty straightforward to do. It'd mean a tiny bit more work in our release process, uploading four APKs to GitHub instead of one. It'd also potentially mean more complexity for downstream users of the APKs, but @IzzySoft has suggested doing exactly this: Upload the application to F-Droid #1270 (comment) so I think that isn't a concern.

  Of the 124.2MB of native libraries, 32.5MB are for arm64. So the arm64 APK (the most relevant one) would be about 91.7MB smaller than the current combined APK, or about 39.8MB. (More likely a few 100kB smaller still, as some of the padding and metadata would go away too.)

  That's a bit smaller than the 41.6MB of v27.169.

  -> Split APK up by ABI #5295

• A second factor is that the native libraries are stored uncompressed in the APKs; they used to be compressed. The default on that setting flipped in the AGP upgrade mentioned above.

  This was actually a sensible change for upstream to make: when the APK gets installed, the libraries need to be uncompressed anyway so the code can run. If they're compressed in the APK, this means making a copy at install time, so the total storage cost is that of a compressed plus an uncompressed copy. If they're uncompressed in the APK, they can be memory-mapped and run directly from the APK, so there's no need to make a copy.

  This change interacts badly with having libraries for 4 different architectures (because then it replaces 4 compressed copies plus 1 uncompressed, with 4 uncompressed copies.) But we'll fix that.

  Once that's fixed, we could turn on compression of native libraries again, but it would come with a trade-off: it'd mean a smaller APK for distributors; potentially smaller downloads over the network (or not, because HTTP compression might do just as well or better); but a larger installed size for users.

  Specifically, for arm64 we have 32.5MB of native libraries. In v27.169, there were 27.4MB of such libraries and they compressed to 8.9MB, a ratio of 32.5%. If the same ratio held, the 32.5MB would compress to about 10.5MB. So the APK would shrink by about 22.0MB, from about 39.8MB to about 17.8MB; but the installed size on the user's device would grow by about 10.5MB.

• Smaller factors:

  • The largest of those native libraries -- 20.0MB for arm64, well over half the total size of all libraries -- is libjsc.so. We'll get to drop that when we switch to Hermes (Android: Use Hermes #4131), which should be smaller than JSC.
  • Some other native libraries appear to be related to Hermes: libhermes-inspector.so, libhermes-executor-release.so, and three others, totalling (on arm64) 2.6MB. Until we do switch to Hermes, it might be possible to leave those out.
  • The .dex files (Java/Kotlin classes) total about 3.5MB after compression. It may be possible to reduce that with minification, or by identifying libraries we don't actually use and removing those. This could save perhaps a MB or two.
  • The icon fonts (our .ttf files) total 1.3MB after compression. We could save at least half of that by excluding the icon fonts we don't actually use.
    • We could save even more -- probably nearly all of it -- by building our own custom icon font to pick and choose just the specific icons we use.
  • Remaining contributors are (after compression): 1.1MB for our JS code (index.android.bundle); 0.4MB for resources.arsc; 0.3MB for KaTeX fonts; 0.2MB for translations; 0.5MB for everything else.

[gnprice]

@IzzySoft I'd be particularly interested in your perspective on this tradeoff, for after we've split the APKs up by ABI:

[W]e could turn on compression of native libraries again, but it would come with a trade-off: it'd mean a smaller APK for distributors; potentially smaller downloads over the network (or not, because HTTP compression might do just as well or better); but a larger installed size for users.

Specifically, for arm64 we have 32.5MB of native libraries. [… At an estimate,] the 32.5MB would compress to about 10.5MB. So the APK would shrink by about 22.0MB, from about 39.8MB to about 17.8MB; but the installed size on the user's device would grow by about 10.5MB.

Which side of that tradeoff do you think would be best?

[IzzySoft]

I was reading your full text with quite some interest. I've never checked the install part: the APK is kept on-device, but I thought resources were always unpacked. Forgive my ignorance on details here: I'm no Android dev and I never dug into which parts of the APK were unpacked and which not; but there must be a reason why the APK is kept.

We have such per-ABI builds at F-Droid as well. That split certainly has it advantages, as users only need to download the "stuff" needed for their device's architecture. Further, this brings your APKs closer to the 30M limit of my repo, where I then could pick e.g. the armeabi-v7a (which covers most devices, many arm64 ones included).

The compression part is what I cannot estimate – it might need some on-device checking. Should it really bring down the APK size to around 15MB (or even less), I could keep multiple copies in my repo¹. For some low-end devices, size might be a concern – but then compare that to the uncompressed "fat build" shipped now, it'd still be an improvement, right? I'd say it would be a very good compromise worth testing. If compression seems not worth it, it can be turned off again anytime with a new release.

Quick calc: instead of 130M mostly needing no unpacking, it then would be 10M (compressed) plus 30M (unpacked) = 40M on-device, so a clear win with a factor of 3 in savings and an "overhead" of 25% compared to uncompressed which otherwise would mean additional transfer and less copies kept on my server, but a potential factor 4 saving on the user's device. Sounds like a good deal to me, size-wise, if there are no other trade-offs.

---

¹ _{having more than one version in the repo helps when a user wants to install without having the index refreshed, while a newer version was already made available; it avoids the 404 caused otherwise, as the previous version would still be there}

[gnprice]

OK, an update: with #5296, the new armeabi-v7a APK will be down to 35.1MB. Much smaller than the 132.3MB for the all-in-one APK at main!

I've never checked the install part: the APK is kept on-device, but I thought resources were always unpacked. Forgive my ignorance on details here: I'm no Android dev and I never dug into which parts of the APK were unpacked and which not; but there must be a reason why the APK is kept.

Yeah, your instinct that there must be a reason the APK is kept around is right. I believe in general all the Android resources and also the "assets" files remain in the APK and are accessed directly from there, rather than making copies of them elsewhere on the device's storage. So with the change that happened in that AGP upgrade above, the system started doing the same for native libraries (.so files).

I just tried switching back to the old behavior (the switch is called android:extractNativeLibs.) Here are some numbers, with the armeabi-v7a APKs:

• With android build: Split up APKs per ABI #5296 alone: the APK is 35.1MB; when installed, the system reports the app's size is 45.5MB.
• With android build: Split up APKs per ABI #5296 plus setting android:extractNativeLibs="true" to return to the old behavior: the APK is smaller, 17.4MB, but the size after install is larger, 55.2MB.
• OTOH at main before android build: Split up APKs per ABI #5296 (at 9b7eeb5), the APK (which is all-in-one) is 132.3MB, and the app's size upon install is 143MB.

For each of those, I uninstalled the app completely, then installed it fresh with adb install; then long-press on app icon in launcher > App info > Storage & cache, to get a screen like this:

In short, by enabling that setting we could bring the APK size down by another 2x, from 35MB to about 17MB. The cost is that the size as actually installed on the user's device (including the APK) would go up, from about 45MB to 55MB. (Either way, both numbers will be much better than they are today before #5296.)

[IzzySoft]

OK, so we go with that 35M APK for now (so Zulip is at least close to the 30M limit of my repo), and experiment with the 17M version later? If you'd go for the latter, I could increase "versions kept" to 2 (to cover the install issue described above). Or you stick to the 35M to save those 10M (i.e. the size of 2 photos taken) on-device. I'm fine with both – just let me know what you decide for so I can adjust configuration on my end (including matching the correct APK to pull once you release – this might be the currently most important part, as if my updater fetches the wrong one we might be stuck with that if switching meant decreasing versionCode).

[gnprice]

OK, so we go with that 35M APK for now (so Zulip is at least close to the 30M limit of my repo), and experiment with the 17M version later?

Yeah, we'll have that 35MB APK in the upcoming v27.182 release, and see how that goes.

Then I think the next biggest opportunity for making things smaller will be:

• The largest of those native libraries -- 20.0MB for arm64, well over half the total size of all libraries -- is libjsc.so. We'll get to drop that when we switch to Hermes (Use Hermes #4131), which should be smaller than JSC.

I'm hopeful we'll be able to do that in the next month or two; I'll want to give it a try once we next upgrade our React Native version, to v0.65.

There are some further space-saving opportunities in my comment at #5086 (comment) . It'll be interesting to pursue some of those once we're on Hermes.

In the meantime, I'm going to close this issue as done -- the app is still larger than I'd like, but the APKs are a lot smaller than the massive size they became starting in v27.170, and in fact a little smaller than they've been for some time before that. (It looks like in v27.156 back in 2020 the APK was somewhat smaller, and then in v27.157 it grew.)

[IzzySoft]

Thanks @gnprice – any yes, looking forward to that! Should you come close to 15 MB (i.e. the half of the 30M limit), be welcome to send me a ping so I reconfigure for keeping 2 versions in my repo, as discussed. I won't look at the byte, so I'd do that also with 16M per APK 😉