Reproducible Builds

[IzzySoft]

At IzzyOnDroid we support Reproducible Builds (see: Reproducible Builds, special client support and more in our repo). Trying for yours, I was able to successfully generate the APK using below "build recipe", but the resulting APKs were not identical. Differences are only in libc++_shared.so for each of the ABIs, where their size differs by 800 kB for arm64 (yours is almost twice as large as the one from our build):

  -rw-r--r--  0.0 unx    10096 b-     4386 defN 1981-01-01 01:01:02 9734baa0 lib/arm64-v8a/libandroidx.graphics.path.so
- -rw-r--r--  0.0 unx  1822720 b-   508893 defN 1981-01-01 01:01:02 75660256 lib/arm64-v8a/libc++_shared.so
+ -rw-r--r--  0.0 unx  1027408 b-   305190 defN 1981-01-01 01:01:02 0d417b8c lib/arm64-v8a/libc++_shared.so
  -rw-r--r--  0.0 unx  2099504 b-   818001 defN 1981-01-01 01:01:02 91d16a3c lib/arm64-v8a/libconscrypt_jni.so
  -rw-r--r--  0.0 unx     7112 b-     2630 defN 1981-01-01 01:01:02 4f56ac38 lib/arm64-v8a/libdatastore_shared_counter.so
  -rw-r--r--  0.0 unx  1483648 b-   800769 defN 1981-01-01 01:01:02 3b0c45ae lib/arm64-v8a/libffmpegJNI.so
  -rw-r--r--  0.0 unx    80336 b-    22462 defN 1981-01-01 01:01:02 b01eb93b lib/arm64-v8a/libvlcjni.so
  -rw-r--r--  0.0 unx     7252 b-     4064 defN 1981-01-01 01:01:02 9a714d3a lib/armeabi-v7a/libandroidx.graphics.path.so
- -rw-r--r--  0.0 unx  1524360 b-   468153 defN 1981-01-01 01:01:02 81a90518 lib/armeabi-v7a/libc++_shared.so
+ -rw-r--r--  0.0 unx   610272 b-   235378 defN 1981-01-01 01:01:02 e4cdaf91 lib/armeabi-v7a/libc++_shared.so
  -rw-r--r--  0.0 unx  1279936 b-   637276 defN 1981-01-01 01:01:02 123cd601 lib/armeabi-v7a/libconscrypt_jni.so
  -rw-r--r--  0.0 unx     4416 b-     2033 defN 1981-01-01 01:01:02 ffa6438b lib/armeabi-v7a/libdatastore_shared_counter.so
  -rw-r--r--  0.0 unx  1436220 b-   791226 defN 1981-01-01 01:01:02 f367690d lib/armeabi-v7a/libffmpegJNI.so
  -rw-r--r--  0.0 unx    51328 b-    19202 defN 1981-01-01 01:01:02 3295e8ef lib/armeabi-v7a/libvlcjni.so
  -rw-r--r--  0.0 unx     9284 b-     4782 defN 1981-01-01 01:01:02 3d9afc66 lib/x86/libandroidx.graphics.path.so
- -rw-r--r--  0.0 unx  1602440 b-   505426 defN 1981-01-01 01:01:02 e390d8eb lib/x86/libc++_shared.so
+ -rw-r--r--  0.0 unx   993108 b-   336151 defN 1981-01-01 01:01:02 ae13de9f lib/x86/libc++_shared.so
  -rw-r--r--  0.0 unx  2254740 b-   904099 defN 1981-01-01 01:01:02 b728d7f5 lib/x86/libconscrypt_jni.so
  -rw-r--r--  0.0 unx     5148 b-     2330 defN 1981-01-01 01:01:02 db8d550b lib/x86/libdatastore_shared_counter.so
  -rw-r--r--  0.0 unx  1473020 b-   773162 defN 1981-01-01 01:01:02 41ced4ee lib/x86/libffmpegJNI.so
  -rw-r--r--  0.0 unx    76004 b-    25677 defN 1981-01-01 01:01:02 4dcddf64 lib/x86/libvlcjni.so
  -rw-r--r--  0.0 unx    10760 b-     4614 defN 1981-01-01 01:01:02 81bd88ae lib/x86_64/libandroidx.graphics.path.so
- -rw-r--r--  0.0 unx  1632144 b-   484338 defN 1981-01-01 01:01:02 27714799 lib/x86_64/libc++_shared.so
+ -rw-r--r--  0.0 unx  1045960 b-   330990 defN 1981-01-01 01:01:02 2d544d1c lib/x86_64/libc++_shared.so
  -rw-r--r--  0.0 unx  2501448 b-   967374 defN 1981-01-01 01:01:02 5e0ce8d6 lib/x86_64/libconscrypt_jni.so

We're building the exoOnly variant with rbtlog on Debian Bookworm, using JDK 17. Here's the build recipe I've used:

        build:
          - sed -r '/signingConfigs.getByName/d; s/val exoOnly = false/val exoOnly = true/' -i androidApp/build.gradle.kts
          - chmod +x gradlew
          - ./gradlew assembleRelease
          - mv androidApp/build/outputs/apk/noLibs/release/*unsigned.apk /outputs/unsigned.apk
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_user: build

We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.

Looking forward to your reply!

cc @obfusk

[yuroyami]

@IzzySoft I reviewed the hints on reproducible builds and didn't find any false flags that could affect the build's reproducibility. The only concern is the size difference in the C++ shared library. I'm not entirely sure what's causing this size discrepancy but after checking out C++ Library Support - Android NDK, I discovered that the libc++_shared.so library file's size depends on the NDK version used. It's also worth noting that the system can provide a "default" version of this library but I am not sure when that happens. What do you think ?

EDIT: Perhaps I have to specify the NDK version in the build gradle file ? Does your build reproducer take into account this field ?

[IzzySoft]

I can specify the NDK to be used. AFAIK it should usually be discovered automatically, but in case a different one is needed (or the process somehow doesn't find the NDK), it can be specified. Currently, it's left on auto-discovery in the recipe for your app (so it found one, but that might be different from what you've used). If I understand that correctly, selection is then left to sdkmanager – I don't know how that picks its choice.

I'm not an Android dev, so appreciate any "lessons" there: is there a standard way to specify the NDK version to be used, eg. in the build gradle? If so, I happily test it out if you tell me how (I could adjust the build.gradle in the build recipe here which would allow me to work with the current release – or we take an updated version from a different commit, but then we'd also need a matching APK).

I see that AGP allows to specify ndkPath. But if you use a custom location there, I'd need to match that in my recipe as well requiring manual changes whenever you update the NDK. I can also specify the NDK version directly here, which would be the easiest way to test if this really is the culprit. Which exact NDK version were you using?

PS: I just found that one can use something like android.ndkVersion '19.2.5345600' in build.gradle. Just mentioning it here so in case the NDK version solves this, we remember this would probably be the preferred way (no "custom path", just the exact version). I'd expect sdkmanager to pick that up then.

---

TL;DR: first step, if you tell me which exact NDK version you were using, I'd explicitly specify that and give it another try. If that solves the issue, we should try with android.ndkVersion in your build.gradle.

[IzzySoft]

OK, you didn't tell me – but as you've added it to your build.gradle I just tried to explicitly specify NDK 26.3.11579264. No dice, issue persists: libc++_shared.so has huge size differences. The same as above btw.

Not sure if it's related, but cannot hurt to ask: what do you use to compile on? We currently run the build on Debian bookworm with JDK 17.

[yuroyami]

@IzzySoft
I specified the NDK version in the gradle build file but haven't re-released the APKs yet. I was planning to ping you once a new version is out so we can check everything. I'm also unsure if Android Studio used that exact version to build the files, as I have multiple NDK versions installed. Per any chance I assume it picks the latest one?

Anyway, with the NDK version specified, if I release the APKs again and the issue persists, we can rule out an NDK mismatch. But since you've already noticed this, it seems unrelated to the NDK version. Not sure what's going on here.
I build on macOS Sonoma with JDK 17. I'll update you once I release a new version, just to confirm the NDK version mismatch isn't the problem.

[IzzySoft]

Ah, so my action was premature – apologies. I was just walking the backlog and happened to reach your app…

Anyway, with the NDK version specified, if I release the APKs again and the issue persists, we can rule out an NDK mismatch. But since you've already noticed this, it seems unrelated to the NDK version.

Not necessarily– unless the APKs currently attached have been built with exactly that NDK. You can provie me an APK here (just rename it to *.zip so Github allows attaching it, don't put it into another zip – that way I can run it directly from here) and name the matching commit. I then can see if RB succeeds with that.

But yeah, I can also wait for the next release if you prefer.