UnifiedNlp nonfunctional

[Iey4iej3]

Lineage OS for microG, polaris (Xiaomi MIX 2S). After flashing the new build 20191005, I found that UnifiedNlp does not work (I tried three backends: Mozilla Location Service, Apple Wi-Fi, Radiocells.org Unified Network Location Provider Backend). Then I go to the microG settings:

microG Settings -> Self-Check:

Network location provider support

• Android version supported:
  Your Android version is not officially supported. This does not necessarily mean anything.

• System supports location provider:
  Your system does not support this UnifiedNlp package. Either install a matching package or a compatibility Xposed module.

UnifiedNlp status

• UnifiedNlp is registered in system:
  The system did not bind the UnifiedNlp service. If you just installed UnifiedNlp you should try to reboot this device.

See also: https://forum.xda-developers.com/showpost.php?p=80443513&postcount=1600 for another Xiaomi device.

[StableNarwhal]

Same on beryllium with a clean install.

[morten-b]

I have the same issue on beryllium

[Iey4iej3]

Possibly related: microg/GmsCore#940, microg/GmsCore#575

[sepo]

I've the same issue on beryllium after a clean installation of build 20191005.

[mar-v-in]

This is caused by an upstream change introduced Aug 1, see https://review.lineageos.org/q/If363679650179a0a7fad7b01055760b49caf26ca

Contact the responsible LineageOS device maintainer and ask them to set config_enableNetworkLocationOverlay and config_enableFusedLocationOverlay to true (this is the default, so there is actually no need to set it, they just need to remove the code that sets it to false). Feel free to refer them to this post.

[Iey4iej3]

This is caused by an upstream change introduced Aug 1, see https://review.lineageos.org/q/If363679650179a0a7fad7b01055760b49caf26ca

Contact the responsible LineageOS device maintainer and ask them to set config_enableNetworkLocationOverlay and config_enableFusedLocationOverlay to true (this is the default, so there is actually no need to set it, they just need to remove the code that sets it to false). Feel free to refer them to this post.

Thanks. There is an issue at their issue tracker. Let's see what happens.

[Iey4iej3]

(this is the default, so there is actually no need to set it, they just need to remove the code that sets it to false)

In fact, there seems actually no need for these commits. It seems to me that QC location is a proprietary location service which leaks privacy by reporting GPS, tower and WiFi info, along with of course, IP addr, so it is very hard to understand having been introduced into a free and open source OS.

[Iey4iej3]

I have received no response from the LIneageOS side. On the other hand, after having searched LineageOS repo, it seems to me that they have a long history of including the same configurations, at least dated back to cyanogenmod era. @corna Is it possible for us to patch over all QC location usages for all devices?

[olliiiver]

Hi @Iey4iej3. Wouldn't https://gitlab.com/LineageOS/issues/android/issues/1270 fix it?

[Iey4iej3]

Hi @Iey4iej3. Wouldn't https://gitlab.com/LineageOS/issues/android/issues/1270 fix it?

I referred to this issue before, and you see up until now there is no response from LineageOS Team. Note that this commit was introduced by a head developer, therefore it might present an official position of LineageOS. I suppose that the probability of a revert is pretty low, and that it might be up to LOS for microG to patch these settings. However, this is difficult, as far as I understand, it might introduce bootloops? And there are a lot of devices with these configurations introduced, therefore LOS for microG might be too burdened to apply such a patch without breaking any devices.

I don't understand why they introduced such a blob. It is possible that there be more technical difficulties, such as without this setting some blobs simply won't work even the system does not boot properly. It is also possible that LineageOS wants to be closer to vendor builds in order to avoid troubles. The worst case is that they don't care much about privacy or FOSS: they introduce blobs as long as they make the system more "friendly", more "usable" and more "attractive"? Without their response, who knows?

[LuccoJ]

I think if LineageOS doesn't revert this decision, it would be appropriate for LineageOS for microG to patch it out itself, since UnifiedNLP is, after all, a core part of microG, and this is crippling it.

[corna]

We have already dealt with com.qualcomm.location, indeed:

• GmsCore "overrides" (i.e., "removes") the com.qualcomm.location package, so if you include GmsCore from this repo, com.qualcomm.location should be removed automatically
• The necessary configs are overridden by our overlay (applied automatically by docker-lineage-cicd).

This has always worked in the past, I have to understand why it doesn't work anymore.

[Iey4iej3]

We have already dealt with com.qualcomm.location, indeed:

* GmsCore ["overrides" (i.e., "removes") the com.qualcomm.location package](https://github.com/lineageos4microg/android_prebuilts_prebuiltapks/blob/master/GmsCore/Android.mk#L19), so if you include GmsCore from this repo, com.qualcomm.location should be removed automatically

* The necessary configs are overridden by our [overlay](https://github.com/lineageos4microg/docker-lineage-cicd/blob/master/src/signature_spoofing_patches/frameworks_base_config.xml) (applied automatically by [docker-lineage-cicd](https://github.com/lineageos4microg/docker-lineage-cicd/blob/master/src/build.sh#L169-L171)).

This has always worked in the past, I have to understand why it doesn't work anymore.

I can confirm that com.qualcomm.location.apk is removed (not totally clean, say, there are remnant settings in /etc/permissions).

For the configurations, I don't know how Android builds, therefore I cannot tell which one will decisively override the other ones. In LineageOS, the configurations in question are added into DEVICE_PACKAGE_OVERLAYS, see device.mk for example. In the build file, seemingly PRODUCT_PACKAGE_OVERLAYS is loaded before, therefore maybe overridden by, DEVICE_PACKAGE_OVERLAYS. This is very probable that the later onee will override the former ones, by looking at a later line, which is used to override a configuration of something configured before. As a consequence, your configurations are overridden by LineageOS' official configurations.

In addition, they added PRODUCT_PROPERTY_OVERRIDES +=persist.vendor.overlay.izat.optin=rro in the commit. I don't know whether this is related.

[olliiiver]

I've tried to rewrite the overlay command in build.sh, so that the microg overlay will be added later. However, no luck. Unfortunately, the device package is added via brunch just before the build, so the upstream config.xml cannot simply be changed by tools like sed in the build.sh script...

[Iey4iej3]

I've tried to rewrite the overlay command in build.sh, so that the microg overlay will be added later. However, no luck. Unfortunately, the device package is added via brunch just before the build, so the upstream config.xml cannot simply be changed by tools like sed in the build.sh script...

@mar-v-in @corna @olliiiver I think the problem is probably not about config_enableNetworkLocationOverlay and config_enableFusedLocationOverlay in LOS for microG, at least on my polaris device. I have just extracted framework-res.apk, and found that these values are both true. The config_locationProviderPackageNames are com.google.android.gms and com.android.location.fused, while config_networkLocationProviderPackageName is @null.

[Iey4iej3]

I cleaned cache and dalvik cache, and found that despite the Self-Check claim, Mozilla NLP Backend and RadioCell are working. Last time I did not clean these caches, so I don't know whether it affects.
I read from logcat that ichnaea etc is responding.

@StableNarwhal @morten-b @sepo Since you have a different device model than mine and not all of you claimed that you have clean installed, could you please check whether UnifiedNlp works after cleaning caches in case of a dirty install? If not, could you please have a try to extract framework-res.apk to see the values of config_enableNetworkLocationOverlay and config_enableFusedLocationOverlay? Here is a description (in English, but not very reader-friendly) and this is a tutorial (not in English, but you can see these commands containing adb pull /system/framework/framework-res.apk about how to extract the configurations).

[Iey4iej3]

Today I reviewed the issue. The problem seems to be: I need to repeatedly disable and re-enable Location, and disable and re-enable MozillaNLPBackend, wait for around 10 minutes to make it work for the first time, which is very long.

Especially, the LOS booted at 10:19, then I disabled and re-enabled location at 10:25, disabled and re-enabled Mozilla NLP Backend in the microG settings at 10:26, disabled and re-enabled location at 10:29, and finally disabled and re-enabled the Mozilla NLP Backend at 10:30. Apps started to get fused location at 10:33.

a filtered logcat, with keywords gsm,ichnaea and location.

After reading these logs, it seems to me that Mozilla NLP Backend really starts only very lately after some disabling/re-enabling actions. @corna @mar-v-in Is it more a problem on the microG side? Maybe I need to repost the issue in microG repo?

Here are some notes:

1. Look at these two lines:

10-21 10:19:52.172 1258 1324 W LocationProviderProxy-network: Odd, no component found for service com.android.location.service.v3.NetworkLocationProvider
10-21 10:19:52.174 1258 1324 W LocationProviderProxy-fused: Odd, no component found for service com.android.location.service.FusedLocationProvider

2. The service ichnaea started only at 10:26. Which means that it was inactive when the system booted, even though I tried to require location from apps. Look at these lines (a lot of not yet found):

10-21 10:26:43.717 1258 1327 I ActivityManager: Start proc 4762:org.microg.nlp.backend.ichnaea/u0a81 for service org.microg.nlp.backend.ichnaea/.BackendService
10-21 10:26:43.795 4762 4762 D IchnaeaBackendService: No instance found active.
10-21 10:26:44.335 2915 4786 D NlpLocBackendHelper: Binding to: Intent { act=org.microg.nlp.LOCATION_BACKEND pkg=org.microg.nlp.backend.ichnaea cmp=org.microg.nlp.backend.ichnaea/.BackendService }
10-21 10:26:44.342 2915 4787 D NlpLocBackendHelper: Not (yet) bound.
10-21 10:26:44.344 4762 4762 D IchnaeaBackendService: No instance found active.
10-21 10:26:44.344 2915 2915 E ActivityThread: Service org.microg.nlp.location.LocationServiceV2 has leaked ServiceConnection org.microg.nlp.location.BackendHelper@721d2e9 that was originally bound here
10-21 10:26:44.344 2915 2915 E ActivityThread: android.app.ServiceConnectionLeaked: Service org.microg.nlp.location.LocationServiceV2 has leaked ServiceConnection org.microg.nlp.location.BackendHelper@721d2e9 that was originally bound here
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.AbstractBackendHelper.bind(AbstractBackendHelper.java:94)
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.location.BackendFuser.bind(BackendFuser.java:75)
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.location.ThreadHelper.enable(ThreadHelper.java:66)
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.location.ThreadHelper.reload(ThreadHelper.java:39)
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.location.LocationProviderV2.reload(LocationProviderV2.java:59)
10-21 10:26:44.344 2915 2915 E ActivityThread: at org.microg.nlp.location.AbstractLocationService.onHandleIntent(AbstractLocationService.java:77)
10-21 10:27:03.264 4185 4812 D StrictMode: at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
10-21 10:27:03.264 4185 4812 D StrictMode: at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
10-21 10:27:03.264 4185 4812 D StrictMode: at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
10-21 10:27:44.342 2915 4787 D NlpLocBackendHelper: Not (yet) bound.
10-21 10:28:44.342 2915 4787 D NlpLocBackendHelper: Not (yet) bound.
10-21 10:29:13.993 1258 1258 V SettingsProvider: Notifying for 0: content://settings/secure/location_providers_allowed

3. The real services seem to start only at 10:30

10-21 10:30:55.748 4762 4762 D IchnaeaBackendService: No instance found active.
10-21 10:30:57.154 2915 5120 D NlpLocBackendHelper: Binding to: Intent { act=org.microg.nlp.LOCATION_BACKEND pkg=org.microg.nlp.backend.ichnaea cmp=org.microg.nlp.backend.ichnaea/.BackendService }
10-21 10:30:57.160 2915 5123 D NlpLocBackendHelper: Not (yet) bound.
10-21 10:30:57.162 4762 4762 D IchnaeaBackendService: No instance found active.
10-21 10:30:57.165 2915 2915 D NlpLocBackendHelper: Bound to: ComponentInfo{org.microg.nlp.backend.ichnaea/org.microg.nlp.backend.ichnaea.BackendService}
10-21 10:30:57.175 4762 4776 D IchnaeaBackendService: Activating instance at process 4762
10-21 10:30:57.238 4762 4776 D IchnaeaBackendService: Cells: 1
10-21 10:30:57.243 4762 5124 D IchnaeaBackendService: request:
{"radioType":"lte","cellTowers":[{"radioType":"lte","mobileCountryCode":???,"mobileNetworkCode":???,"locationAreaCode":???,"cellId":???,"signalStrength":-107,"psc":171,"asu":33}],"wifiAccessPoints":[],"fallbacks":{"lacf":true,"ipf":false}}

[sepo]

@Iey4iej3 I can confirm, that the values of config_enableNetworkLocationOverlay and config_enableFusedLocationOverlay are both true. The framework-res.apk is from a clean installation of lineage-16.0-20191021-microG-beryllium.

[Iey4iej3]

@mar-v-in There are several issues causing this, both in Mozilla Nlp Backend and microG. First, microG does not Nlp backends after the boot. Second, Mozilla Nlp Backend does not create an instance properly after started which worsen the issue: I tried other backends like Apple WiFi and local GSM Service, which works only one or two minutes after started triggered by re-enabling in settings of UnifiedNlp in microG.

[StableNarwhal]

I'm honestly considering going back to regular LOS with minimal GAPPS. I have nothing but the greatest respect for all the volunteers working on microg and the automatic LOS+microg build, but this setup just doesn't seem viable for a production grade setting anymore.
I'm currently on a July build for beryllium due to all the independent hickups and an increasing number of apps just refuse to work with microg, ancient bug reports never got fixed.

This is completely OT and I apologise, but after a few years of running a microg setup and converting all the convenient account choices to privacy friendly alternatives I guess I'll have to give up - at least for now.

[mar-v-in]

@Iey4iej3 given your log it looks like their are still traces of qualcomm location in that build. Right before the first LocationProviderProxy-network: Odd, no component found for service com.android.location.service.v3.NetworkLocationProvider, there is a ServiceWatcher: com.qualcomm.location not found, so it's still searching for a wrong package.

Another known issue with some setups of file-based encryption will log com.google.android.gms resolves service com.android.location.service.v3.NetworkLocationProvider, but has wrong signature, ignoring before the Odd line.

[Iey4iej3]

@olliiiver @mar-v-in Now I understand the situation! Just look at this commit:

# GNSS

PRODUCT_PROPERTY_OVERRIDES += persist.vendor.overlay.izat.optin=rro

I googled persist.vendor.overlay and understood that these objects are compiled into vendor partition, so that they can perform a runtime overriding over other static settings! How DIRTY that trick is!!! I fetched /vendor/overlay/framework-res__auto_generated_rro.apk, decoded and grep -R qualcomm, found the result as expected:

framework-res__auto_generated_rro/res/values/bools.xml:

<bool name="config_enableNetworkLocationOverlay">false</bool>
<bool name="config_enableFusedLocationOverlay">false</bool>

framework-res__auto_generated_rro/res/values/:

<string name="config_networkLocationProviderPackageName">com.qualcomm.location</string>
<string name="config_fusedLocationProviderPackageName">com.qualcomm.location</string>
<string name="config_comboNetworkLocationProvider">com.qualcomm.location</string>

@corna Is it possible to patch over these rro overrides?

By the way, the result of grep -R com.qualcomm.location is:

:/system # grep -R com.qualcomm.location
Binary file ./lib64/libxt_native.so matches
Binary file ./lib64/liblocationservice_jni.so matches
./etc/permissions/privapp-permissions-com.qualcomm.location.xml: <privapp-permissions package="com.qualcomm.location">
./etc/permissions/privapp-permissions-com.qualcomm.location.xml: <privapp-permissions package="com.qualcomm.location.XT">
./etc/permissions/com.qualcomm.location.xml:* file com.qualcomm.location.xml
./etc/selinux/plat_service_contexts:com.qualcomm.location.izat.IzatService u:object_r:izat_service:s0
./vendor/etc/selinux/vendor_seapp_contexts:user=system seinfo=platform name=com.qualcomm.location.XT isPrivApp=true domain=location_app type=system_app_data_file
Binary file ./vendor/lib/liblbs_core.so matches
Binary file ./vendor/lib/libalarmservice_jni.so matches
Binary file ./vendor/overlay/framework-res__auto_generated_rro.apk matches
Binary file ./vendor/lib64/liblbs_core.so matches
Binary file ./vendor/lib64/libalarmservice_jni.so matches

[Iey4iej3]

@corna Is it possible to patch over PRODUCT_PROPERTY_OVERRIDES? Seemingly this appears very early in product.mk and as far as I understand, the earlier the option appears, the more priority it has. It might be possible to patch product.mk, but I don't know how to override properly the setting persist.vendor.overlay.izat.optin=rro.

[Iey4iej3]

@corna Another idea is to insert around https://github.com/lineageos4microg/docker-lineage-cicd/blob/e724032a86cfd469b97c82027a7617a83149f05e/src/build.sh#L171 a similar line:

sed -i "1s;^;PRODUCT_PROPERTY_OVERRIDES += persist.vendor.overlay.izat.optin=none\n;" "vendor/$vendor/config/common.mk"

to override the value rro (I don't know whether this will work, since I don't find a manual about all available values of that persist.vendor.overlay.izat.optin). Seemingly LOS Team does not want to remove these blobs.

[StableNarwhal]

microg/GmsCore#964

Does this mean this issue is also fixed? Apologies for the ignorant questions, I currently just don't have the time to test around myself.

[Iey4iej3]

@StableNarwhal No, that does not fix for beryllium and polaris.

That crashing and non-registered issue is about a bug introduced in the latest build of microG where proguard "optimizes" out some essential codes.

This issue is not an issue on the microG side. As far as I understand now, it is due to the fact that LineageOS introduced some codes and blobs (from MIUI) which specify Qualcomm Location to be the only accepted location provider (in an exclusive fashion) and consequently, UnifiedNlp should not work as long as these codes persist. People tried to ask the LineageOS Team to revert these commits, but seemingly they simply ignored this request, and therefore it is up to LOS for microG, say @corna, to find solutions to patch over these codes properly. These codes are pretty dirty. They not only set Qualcomm Location exclusively as the only accepted location provider at compile time, but also apply these configurations at runtime by rro, which effectively invalidates the previous patch of LOS for microG.

(NB: In fact, there are several devices with similar codes, that is to say, there are already several devices on which theoretically LOS for microG should not work, but seemingly nobody reported before. beryllium and polaris are only some of latest devices that introduced these codes)

[StableNarwhal]

Thanks for the very detailed synopsis. I normally wouldn't use any project which knowingly betrayed their core values for this long without actually telling people, unfortunately LOS seems to have a bit of a monopoly regarding privacy-friendly after-market ROMS for non-pixel devices.

Even if the awesome LOS4microg maintainers fix this issue, LOS, at least in my current understanding, could break this and many other improvements this fork provides literally ever day and successfully ignore the backlash of our very minor sub-community.

[Iey4iej3]

I opened a pull request lineageos4microg/docker-lineage-cicd#65 which I hope to be helpful.

[Iey4iej3]

Hi @olliiiver, do you understand how the build process parses PRODUCT_PROPERTY_OVERRIDES? I tried to add a line with no luck. I extracted the /vendor/build.prop:
build.prop
I searched the keyword ADDITIONAL VENDOR BUILD PROPERTIES in the LineageOS github tree and apparently it is generated by Makefile, which should be the collection of PRODUCT_PROPERTY_OVERRIDES. Therefore the added line is not taken into account.

[Iey4iej3]

Globally, we need to understand how the LOS builder treat the overlays. Now some of settings are built into the vendor image, not the system image. Especially we need to understand why there is a setting in the system image that config_enableNetworkLocationOverlay is true, while there is an overlay in the vendor image which resets it to be false. After having understood how they are distinguished, we need to find a way to override vendor image settings, since seemingly they are preferred over the settings in the system image.

[siebenbrodt]

Possible post-install workaround (I haven't tried it yet myself) for beryllium: recompile /vendor/overlay/framework-res__auto_generated_rro.apk with corrected values.

[Iey4iej3]

Possible post-install workaround (I haven't tried it yet myself) for beryllium: recompile /vendor/overlay/framework-res__auto_generated_rro.apk with corrected values.

This is not the "correct" way in my eyes. It is in fact, first decompile, then modify, and finally repack. (I don't know how the signature works for rro, but if later Google add a signature verification then it would be very difficult to modify) This is no better than, for example, revert the commit manually to build.

[siebenbrodt]

It seems that the story came to an end – the commit will be reverted due to "bla bla bla" and "all the whining".
https://review.lineageos.org/q/I77a82306b21030056e30d43eda4e504963554309

[morten-b]

I can confirm that it is now functional again on beryllium. I think the issue can be closed.

[Iey4iej3]

Let's close it for the moment. If other devices are still affected (it might be the case that they just reverted codes for some devices), feel free to open it. And a side note: the issue in our building script is not fixed, used to override LOS upstream settings. However, nobody has spare time to understand and fix it, and given that the related settings are reverted upstream, it is not urgent to fix it.

[ghost]

Still an issue in 2024. Please fix