Machine test-aws-ubuntu1804-armv8-1 fails testSCCMLTests1_openj9_1 with openjdk11 Aarch64

[LongyuZhang]

Machine test-aws-ubuntu1804-armv8-1 fails testSCCMLTests1_openj9_1 Test 26 for nightly openjdk11 Aarch64 with timeout error, https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.functional_aarch64_linux/1.

After testing with other machines (test-packet-ubuntu1604-armv8-2, test-aws-rhel76-armv8-2, test-aws-rhel76-armv8-4, test-packet-ubuntu1604-armv8-1), they all passed this test, with links https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4002 - 4005, only the same machine as nightly build (test-aws-ubuntu1804-armv8-1) still hangs at testSCCMLTests1_openj9_1 Test 26 (I manually cancelled after hanging for 1 hour) https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4007/. Thanks.

Related issue: eclipse-openj9/openj9#10741

Failure Output:

21:24:09  Testing: Test 26: CMVC 168131 : Create a non persistent cache
21:24:09  Test start time: 2020/09/29 01:24:08 Coordinated Universal Time
21:24:09  Running command: "/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux/openjdkbinary/j2sdk-image/bin/java"  -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=ShareClassesCMLTests,nonpersistent -version
21:24:09  Time spent starting: 1 milliseconds
21:34:08  ***[TEST INFO 2020/09/29 01:34:08] ProcessKiller detected a timeout after 600000 milliseconds!***
21:34:08  INFO: getUnixPID() has failed indicating this is not a UNIX System.'Debug on timeout' is currently only supported on Linux.
21:34:08  
21:34:08  
Cancelling nested steps due to timeout
06:34:24  Sending interrupt signal to process
06:34:26  Time spent executing: 33016381 milliseconds
06:34:26  Test result: FAILED

[LongyuZhang]

Recent openjdk11 Aarch64 sanity.functional nightly passed on another machine "test-aws-rhel76-armv8-4", https://ci.adoptopenjdk.net/view/all/job/Test_openjdk11_j9_sanity.functional_aarch64_linux/2/.

[pshipton]

FYI https://openj9.slack.com/archives/C8312LCV9/p1602551615048900

there’s a pthread_cond_signal bug affecting glibc 2.27. Worth being aware of this if unexplained deadlocks are occurring: https://sourceware.org/bugzilla/show_bug.cgi?id=25847

[andrew-m-leonard]

Happened again on Windows:
https://ci.adoptopenjdk.net/job/Test_openjdk8_j9_sanity.functional_x86-64_windows_xl/10/console

23:06:24  Test start time: 2020/11/12 23:06:24 Coordinated Universal Time
23:06:24  Running command: "C:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_windows_xl/openjdkbinary/j2sdk-image\\bin\\java"   -Xdump:system+heap+snap+jit:none -cp "C:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_windows_xl/openjdk-tests///..//jvmtest\functional\cmdline_options_testresources\cmdlinetestresources.jar" VMBench.GPTests.GPTest abort
23:06:24  Time spent starting: 12 milliseconds
23:06:24  Time spent executing: 394 milliseconds
23:06:24  Test result: PASSED
23:06:24  Output from test:
23:06:24   [OUT] Invoking abort!
23:06:24   [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2020/11/12 23:06:24 - please wait.
23:06:24   [ERR] JVMDUMP032I JVM requested Java dump using 'C:\Users\jenkins\workspace\Test_openjdk8_j9_sanity.functional_x86-64_windows_xl\openjdk-tests\TKG\test_output_16052206202425\cmdLineTest_sigabrtHandlingTest_0\javacore.20201112.230624.9048.0001.txt' in response to an event
23:06:24   [ERR] JVMDUMP010I Java dump written to C:\Users\jenkins\workspace\Test_openjdk8_j9_sanity.functional_x86-64_windows_xl\openjdk-tests\TKG\test_output_16052206202425\cmdLineTest_sigabrtHandlingTest_0\javacore.20201112.230624.9048.0001.txt
23:06:24   [ERR] JVMDUMP013I Processed dump event "abort", detail "".
23:06:24  >> Success condition was found: [Output match: Processing dump event "abort"]
23:06:24  >> Required condition was found: [Output match: Processed dump event "abort"]
23:06:24  
23:06:24  Testing: -XX:+HandleSIGABRT
23:06:25  Test start time: 2020/11/12 23:06:24 Coordinated Universal Time
23:06:25  Running command: "C:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_windows_xl/openjdkbinary/j2sdk-image\\bin\\java"   -Xdump:system+heap+snap+jit:none -XX:+HandleSIGABRT -cp "C:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_windows_xl/openjdk-tests///..//jvmtest\functional\cmdline_options_testresources\cmdlinetestresources.jar" VMBench.GPTests.GPTest abort
23:06:25  Time spent starting: 4 milliseconds
23:46:36  ***[TEST INFO 2020/11/12 23:46:25] ProcessKiller detected a timeout after 2400000 milliseconds!***
23:46:36  INFO: getUnixPID() has failed indicating this is not a UNIX System.'Debug on timeout' is currently only supported on Linux.
23:46:36  Output from test:
23:46:36   [OUT] Invoking abort!
23:46:36   [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2020/11/12 23:06:25 - please wait.
23:46:36   [ERR] JVMDUMP032I JVM requested Java dump using 'C:\Users\jenkins\workspace\Test_openjdk8_j9_sanity.functional_x86-64_windows_xl\openjdk-tests\TKG\test_output_16052206202425\cmdLineTest_sigabrtHandlingTest_0\javacore.20201112.230625.77392.0001.txt' in response to an event

[andrew-m-leonard]

and another: https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.functional_aarch64_linux/35/console

[pshipton]

I believe these are separate problems. Pls open an OpenJ9 issue for the hang in javacore creation in the -XX:+HandleSIGABRT test.

[pshipton]

Created eclipse-openj9/openj9#11198 for the hang on Windows.