CI tests are timing out

[tobrun]

Since roughly 8 days, we have started to see that CI tests that are timing out. At first it looked the test were running slower but now noticing that some occurrences seem to stop execution. For now the android build has been made non required for merging.

Capturing from the logs:

08-18 23:08:39.091: I/TestRunner(5106): failed: testBackgroundOpacityAsConstant(com.mapbox.mapboxsdk.testapp.style.BackgroundLayerTest)
08-18 23:08:39.091: I/TestRunner(5106): ----- begin exception -----
08-18 23:08:39.092: I/TestRunner(5106): java.lang.RuntimeException: Could not launch intent Intent { act=android.intent.action.MAIN flg=0x10000000 cmp=com.mapbox.mapboxsdk.testapp/.activity.espresso.EspressoTestActivity } within 45 seconds. Perhaps the main thread has not gone idle within a reasonable amount of time? There could be an animation or something constantly repainting the screen. Or the activity is doing network calls on creation? See the threaddump logs. For your reference the last time the event queue was idle before your activity launch request was 1503122874085 and now the last time the queue went idle was: 1503122874085. If these numbers are the same your activity might be hogging the event queue.
08-18 23:08:39.092: I/TestRunner(5106): 	at android.support.test.runner.MonitoringInstrumentation.startActivitySync(MonitoringInstrumentation.java:360)
08-18 23:08:39.092: I/TestRunner(5106): 	at android.support.test.rule.ActivityTestRule.launchActivity(ActivityTestRule.java:219)
08-18 23:08:39.092: I/TestRunner(5106): 	at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:268)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.Suite.runChild(Suite.java:128)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.Suite.runChild(Suite.java:27)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
08-18 23:08:39.092: I/TestRunner(5106): 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
08-18 23:08:39.092: I/TestRunner(5106): 	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
08-18 23:08:39.092: I/TestRunner(5106): 	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
08-18 23:08:39.092: I/TestRunner(5106): 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
08-18 23:08:39.092: I/TestRunner(5106): ----- end exception -----

[tobrun]

I have done some tests with a commit older as 8 days old and not seeing the issue there:

• https://circleci.com/gh/mapbox/mapbox-gl-native/19101
• https://circleci.com/gh/mapbox/mapbox-gl-native/19102
• https://circleci.com/gh/mapbox/mapbox-gl-native/19103

Update: to validate, reran tests above:

• https://circleci.com/gh/mapbox/mapbox-gl-native/19639
• https://circleci.com/gh/mapbox/mapbox-gl-native/19640
• https://circleci.com/gh/mapbox/mapbox-gl-native/19641
• https://circleci.com/gh/mapbox/mapbox-gl-native/19647
• https://circleci.com/gh/mapbox/mapbox-gl-native/19648
• https://circleci.com/gh/mapbox/mapbox-gl-native/19649
• https://circleci.com/gh/mapbox/mapbox-gl-native/19650

all of them succeeded

[tobrun]

Follow test with 336259d and not seeing the issue

• https://circleci.com/gh/mapbox/mapbox-gl-native/19110
• https://circleci.com/gh/mapbox/mapbox-gl-native/19132

Update: to validate, reran tests above:

• https://circleci.com/gh/mapbox/mapbox-gl-native/19644
• https://circleci.com/gh/mapbox/mapbox-gl-native/19643
• https://circleci.com/gh/mapbox/mapbox-gl-native/19651
• https://circleci.com/gh/mapbox/mapbox-gl-native/19651
• https://circleci.com/gh/mapbox/mapbox-gl-native/19652

all if them succeeded

[tobrun]

This issue will be potentially fixed with #9816 as the source for this regression is cd8eb13

[brunoabinader]

This will be fixed in #9842.

[jfirebaugh]

@tobrun #9846 seems to show that cd8eb13 is not the issue (or at least not the only issue).

[tobrun]

Apologies @brunoabinader @jfirebaugh tracked this down to a059c74. Reverting this change in #9851.

[jfirebaugh]

💯 🙏 @tobrun for tracking that down.