java.lang.NullPointerException in NetworkInterfacesUtil

[Thomas-Vos]

• org.eclipse.californium:californium-core:2.6.3
• org.eclipse.californium:scandium:2.6.3

I received the following stack trace in my app:

Fatal Exception: java.lang.NullPointerException: Attempt to invoke interface method 'boolean java.util.Enumeration.hasMoreElements()' on a null object reference
       at org.eclipse.californium.elements.util.NetworkInterfacesUtil.initialize(NetworkInterfacesUtil.java:129)
       at org.eclipse.californium.elements.util.NetworkInterfacesUtil.isMultiAddress(NetworkInterfacesUtil.java:400)
       at org.eclipse.californium.core.coap.Request.setDestinationContext(Request.java:900)
       at org.eclipse.californium.core.coap.Request.setURI(Request.java:514)
       at org.eclipse.californium.core.coap.Request.setURI(Request.java:457)
       at ...

Related Californium code (2.6.3):

https://github.com/eclipse/californium/blob/d6c24198d6891a432ebe67b3044263ef503dcd2e/element-connector/src/main/java/org/eclipse/californium/elements/util/NetworkInterfacesUtil.java#L128-L129

It looks like NetworkInterface.getNetworkInterfaces() can return null in some cases, so this should be handled correctly in Californium. You can actually see that it can return null in the Android source code here: https://cs.android.com/android/platform/superproject/+/master:libcore/ojluni/src/main/java/java/net/NetworkInterface.java;l=420;drc=29ea579e1b0541c320448582b6695f0fb9e8aa82

[boaks]

Fixing the NullPointerException is one thing, but broadcast-address detection an local link MTU will not be available. If you want a specific other MTU configure it, otherwise the "common" defaults are used.

[boaks]

Please try out the provided PR. If it works for you, I schedule a bugfix release 2.6.4 for next Thursday.

[Thomas-Vos]

Thanks for the quick fix, I appreciate that.

I have not tried to reproduce the issue myself yet. The crash was reported through analytics and multiple users got the crash. I am not sure yet in what situations this can be null, so it could be hard to reproduce without more information.

The reported manufacturers so far are Huawei and Mobvoi Android devices on Android 8/9.

[boaks]

Maybe turn off wifi and mobile data and then start the app?
I will try it also on my own.

[boaks]

javadoc - Returns null if no network interfaces could be found on this machine

[Thomas-Vos]

I tried that but no crashes, NetworkInterface.getNetworkInterfaces() still returns a non-null value. Also, if it would be this easy to reproduce the issue, I would have gotten many more reported crashes. This bug affects just a very few users in my app.

I looked into the crash again, and noticed that the crash was only reported on Wear OS devices so far. Wear OS devices are smart watches, which are basically just Android devices. However, Internet does work a bit different than on Android phones. Wear OS devices could be using just WiFi. But it could also be a proxy over Bluetooth to the connected Android phone. I guess something there works a bit different which causes NetworkInterface.getNetworkInterfaces() to return null in some cases. Maybe it returns null if the watch is switching between WiFi and Bluetooth? I am not sure.

[boaks]

An additional concern from my side is, that even if this null is fixed, I guess, starting the Connector (and so the open the socket) will also fail.

[Thomas-Vos]

Yes that could be the case but wouldn't that be better than a NullPointerException?

[boaks]

I don't know. My feeling is, it will not work, either for the null or the failing network. We will see ...

[Thomas-Vos]

I understand. However, just wrapping the setURI function with a try/catch to catch NullPointerException does not make sense to me. I would prefer the network request to fail or at least that a better exception is thrown.

[boaks]

It's not about not fixing the null, it's more about the bug after the bugfix .-).
We will see. Without test-environment it will get hard to see, how that situation should be handled.

[boaks]

I merged the PRs.

I would appreciate, if it could be tested, what else will be required.

For 2.6.x we are somehow limited to bugfixes.
Other changes requires a new review for Californium (that last is far more older than 1 year), which would create the most benefit, if the 3.0.0 is the reviewed version (my current plan: September).

The "feature: runs on changing network environment" wasn't considered for 2.6.
I'm not sure, if that could be made working without changing the API. We will see, what else will be required.
(e.g. for 3.0 I added NetwrokInterfacesUtil.clear() in 03a18eb for better support. That is not that mandatory, only for using broadcasts and auto detected MTU).

At least, I would propose to release 2.6.4 only after your feedback about this "overall issue".

[Thomas-Vos]

Ok, I found out why NetworkInterface.getNetworkInterfaces() sometimes returns null in my Wear OS app. Currently making a sample project.

[Thomas-Vos]

Sample project (requires Android 6.0+):
https://github.com/Thomas-Vos/CaliforniumIssue1664

I was only able to get it to print null on a real phone. Also it does not print null every time, but after switching on/off Wifi a few times it does print null for me.

1. Turn on Wifi on your phone
2. Run the sample project
3. Logs will show the following:

Network interfaces: [name:dummy0 (dummy0), name:rmnet6 (rmnet6), name:wlan0 (wlan0), name:p2p0 (p2p0), name:lo (lo)]

4. Now turn off Wifi (but keep the app visible) and the logs should show this (but not every time):

Network interfaces: null
Network interfaces: [name:dummy0 (dummy0), name:rmnet6 (rmnet6), name:p2p0 (p2p0), name:lo (lo)]

5. If you see no null, just toggle Wifi a few times so it connects/disconnects to a network.

See MainActivity.java for more details (I added some comments)

[boaks]

Nice, if it returns null, what happens then when you (try) to send the request?
(Requires to test with the current 2.6.x branch, the 2.6.4-SNAPSHOT)

[Thomas-Vos]

Reproduces the NullPointerException with Californium 2.6.3: https://github.com/Thomas-Vos/CaliforniumIssue1664/tree/californium_2.6.3

I used the URL coap://35.185.40.182:5683/ in the sample (which is the Californium test server).

With Californium 2.6.4-SNAPSHOT the CoapResponse is null. Project here: https://github.com/Thomas-Vos/CaliforniumIssue1664/tree/californium_2.6.4

This looks expected to me as bindProcessToNetwork blocks all sockets if network (in this case WiFi) is lost even if there is mobile data available.

Anything else you want me to try?

[boaks]

Anything else you want me to try?

No, thanks. It's more about, if that "works for you". If so, should I schedule 2.6.4 for Thursday?

[Thomas-Vos]

For me this works fine as it no longer crashes but just receives a null response. (expected behaviour as network is not reachable). A new release would be great.

I noticed that Californium does not work again after network is restored. Http requests work fine but Californium not. I also use the new NetworkInterfacesUtil.clear() function in 3.0.0-SNAPSHOT. Sample project here: https://github.com/Thomas-Vos/CaliforniumIssue1664/tree/californium_3.0.0

The interesting code is here: https://github.com/Thomas-Vos/CaliforniumIssue1664/blob/californium_3.0.0/app/src/main/java/com/example/californiumissue1664/MainActivity.java

See logs below (I added some explanations below):

2021-07-12 15:17:10.310 15430-16547/com.example.californiumissue1664 I/System.out: Network interfaces: [name:dummy0 (dummy0), name:rmnet6 (rmnet6), name:wlan0 (wlan0), name:p2p0 (p2p0), name:lo (lo)]
Now I turn off Wifi
2021-07-12 15:17:15.113 15430-16547/com.example.californiumissue1664 I/System.out: Network interfaces: null
2021-07-12 15:17:15.122 15430-16601/com.example.californiumissue1664 I/System.out: doHttpRequest result: java.net.UnknownHostException: Unable to resolve host "example.com": No address associated with hostname
2021-07-12 15:17:15.150 15430-16601/com.example.californiumissue1664 I/System.out: Coap response: null
As you can see above HTTP and Californium requests both fail which is expected.

The app now calls bindProcessToNetwork(null) and interfaces are available again.
2021-07-12 15:17:15.152 15430-16601/com.example.californiumissue1664 I/System.out: Network interfaces: [name:dummy0 (dummy0), name:rmnet6 (rmnet6), name:p2p0 (p2p0), name:lo (lo)]
After sleeping for one second the HTTP request is successful but Californium still fails.
2021-07-12 15:17:16.575 15430-16601/com.example.californiumissue1664 I/System.out: doHttpRequest result size: 1256
2021-07-12 15:17:16.584 15430-16601/com.example.californiumissue1664 I/System.out: Coap response: null

If I comment out the first Californium request and only execute the second one when network is available again, it works fine. So it seems Californium is still remembering something from the first failed request. Is this expected? For the second request I create a new CoapClient and also called NetworkInterfacesUtil.clear().

[boaks]

Is this expected?

That is more what I'm afraid of. That's that kind of bug after bugfix I has already mentioned.

AFAIK, Californium does a lot on "startup" but doesn't retry that. It is not "implemented" to run in changing Network environments.

The NetworkInterfacesUtil.clear() should not really be the show-stopper or maker, at least, if no broadcast requests are used. The MTU would mainly affect x509 handshakes, but the defaults should also work.

I checkout your sample and try, If I find the cause for failing the second request.

[boaks]

One idea would be, that the EndpointManager keeps the "broken" Endpoint on the first access of getDefaultEndpoint(String scheme).

Therefore I would try to use EndpointManager.reset(). I'm not sure, if it works. As I wrote, that are not the cases, which have been considered.

(I will also try to reproduce it. I'm not that sure, if I'm successful.)

[Thomas-Vos]

Great, just tried adding EndpointManager.reset() and now it works again! I added it after the NetworkInterfacesUtil.clear() line.

The reproducer is maybe not the best, but it works on my phone (Samsung Galaxy S21 Ultra on Android 11). It is hard to make a good sample project if I do not know the exact cause of the issue in my real app.

[boaks]

The reproducer is maybe not the best

But much better than no-one :-).

As I wrote, for 2.6 I don't think the NetworkInterfacesUtil.clear() is required. if you want to use it with 2.6.4, could you please check it with that version and without the NetworkInterfacesUtil.clear()?

(I still on try to reproduce it ... until now, without luck .. 6x wifi off-on).

[Thomas-Vos]

Ok, that works too. Californium 2.6.4-SNAPSHOT with EndpointManager.reset().

[Thomas-Vos]

Does your phone have Wifi and mobile data? Because the reproducer requires both. Basically what it does is:

1. connect Wifi manually
2. the app forces all sockets to only work on Wifi - connectivityManager.bindProcessToNetwork(network)
3. disconnect Wifi manually
4. onLost callback is called
5. network interfaces are now null (but not always it seems). If null, the code continues:
6. http and coap requests fail
7. clear binding - connectivityManager.bindProcessToNetwork(null) - the app will now use mobile data
8. network interfaces are available again (mobile data)
9. waits one second
10. now http works again, and we are trying to get coap working too, using the clear/reset functions

[boaks]

I used the emulator .. on my Samsung A50 it works ...

[boaks]

I tested it now a couple of times. Works on my side.
I think, we should go for 2.6.4 on Thursday, or?

[Thomas-Vos]

Great to hear. Thursday is fine for me. Thanks for all the help.

[boaks]

You're welcome.

[boaks]

Hi @Thomas-Vos ,

I currently in preparation for starting to "switch to Californium 4.0". Removing the deprecates, simplifying some historical grown names and updating the minimal java version to support (java 7 gets more and more pain).

Do you still use Californium for Android? If so, what is t the minimal java version required for you supporting Android?

[sbernard31]

(Just to let you know current Leshan 2.0.0 minimal version is Java 8)

[sbernard31]

The corresponding Leshan issue about that move :eclipse-leshan/leshan#924
The most important comment in this issue : eclipse-leshan/leshan#924 (comment)
An old slack overflow question related to that https://stackoverflow.com/a/54130025/5088764 (not sure it helps)

Note that this is now possible to compile with recent JVM targeting older JVM without toolchains : eclipse-leshan/leshan#1412

Minimal java version to build Leshan is 11 but artifacts are compatible with java 8

[boaks]

Thanks Simon for this information!
I also consider java 8 as a good minimal java version to support.

[Thomas-Vos]

Hi @boaks,

Thanks for letting me know about this. My app still supports Android 5.0 and higher (Java 8 is starting from Android 8.0).

However, there is now a way to use Java 8 language features and APIs on older Android versions like Android 5: https://developer.android.com/studio/write/java8-support

So basically, you just publish Californium which uses Java 8 APIs. The library users targeting old Android versions need to follow the guide mentioned above. The app and library code is automatically rewritten to use the backported APIs (called desugaring), like described on the page.

I think (almost?) all Java 8 APIs are supported. Supported syntax is described here, and here backported Java 8 and Java 11 APIs. I don't think you will run into any issues.

Already using it for other libraries and it is working great. So you do not have to do anything to support this in Californium. But maybe it's a good idea to mention it in the release notes so Android users are aware of this change and can implement this.

[boaks]

I currently started to work on Californium 4. I hope to prepare the first "update PR" end of next week. I would then be very welcome, if you could check, if that update works for you.

[boaks]

The first bunch of changes are available in branch 4.0.x, see PR #2297 .

@Thomas-Vos

If you could spend some time to check, if that works for you with Android, that would be great.
If it's currently not possible and you would like to postpone that, let me know.

[Thomas-Vos]

@boaks, thanks for letting me know. At the moment I am really busy with other tasks so unfortunately I currently do not have the time to test this in detail.

Furthermore, in my app Hue Essentials I am using Californium with the IKEA TRADFRI gateway. However, this gateway is no longer available and the upgraded version does not use coap. So my userbase that actually needs Californium is shrinking so updating the library is also less of a priority for me.

[boaks]

Thanks for your answer.