Instrumentation tests

[tobrun]

tldr: Change the way how we do testing on Mapbox Maps SDK for Android.

Flaky instrumentation tests

We have been attempting to stabilise our instrumented test harness using Espresso, which executes tests by applying view matching on top of our MapView, but failed to find a resolution. 1 out of 5000 executions can still lead to a test run not being started by properly.

Native libraries

We have been using instrumentation tests to workaround limitations with standard unit tests that are run on the jvm of the executing machine. Main reason is that it's not afaik possible to load the our native libraries on the hosting machine as part of an test execution. Atm you will hit an exception when calling into a native method.

Above are the two main factors driving this change. We need to revisit the approach we take to testing:

Proposal

All java code should be tested with unit tests:

• we use roboelectric to avoid missing Android SDK api's
• we code against interfaces to hide the native implementation of classes

Instrumentation tests that validate the jni to core integration without relying on a MapView:

• can be build up from solely depending on NativeMapView
• no java logic may be tested with instrumented tests

Instrumentation tests that still require ViewMatching:

• still needed for sanity test of activities
• still needed for render tests
• still needed for performance tests
• we can lower the amount of tests that depends on ViewMatching from 600 to > 100

With above we cover the full stack and have a strict division between what is tested when.

cc @mapbox/maps-android

[LukasPaczos]

This will also possibly allow us to run those instrumented unit tests on emulators, ruling out flakiness and availability issues of real devices on device farms.

[danesfeder]

Tagging @mapbox/navigation-android here as this is great test work for us to look into as well. Thanks for digging into this @tobrun

[tobrun]

Flagging that we are still hitting a lot of flaky instrumentation tests for our auto generated activity tests:

They fail with:

java.lang.RuntimeException: java.util.concurrent.ExecutionException: android.support.test.espresso.IdlingResourceTimeoutException: Wait for [FinishLoadingStyleIdlingResource] to become idle timed out

We should prioritse fixing this @mapbox/maps-android