Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-release triage of weekly test runs #2754

Closed
smlambert opened this issue Jul 18, 2021 · 16 comments
Closed

Pre-release triage of weekly test runs #2754

smlambert opened this issue Jul 18, 2021 · 16 comments

Comments

@smlambert
Copy link
Contributor

smlambert commented Jul 18, 2021

This report represents this view:

Screen Shot 2021-07-18 at 11 11 43 AM

Notable:


TRSS link
Build URL https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/1709/
Started by upstream project "build-scripts/weekly-openjdk8-pipeline" build number 30

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_0
hotspot_jre_0 => deep history 0/2 passed | possible issues
jdk_jmx_0 => deep history 0/3 passed | possible issues
jdk_jfr_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_sparcv9_solaris
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.perf_x86-32_windows
renaissance-als_0 => deep history 0/5 passed | possible issues
renaissance-chi-square_0 => deep history 0/5 passed | possible issues
renaissance-dec-tree_0 => deep history 0/5 passed | possible issues
renaissance-gauss-mix_0 => deep history 0/5 passed | possible issues
renaissance-log-regression_0 => deep history 0/5 passed | possible issues
renaissance-movie-lens_0 => deep history 0/5 passed | possible issues

Test_openjdk8_hs_sanity.perf_sparcv9_solaris
renaissance-scala-kmeans_0 => deep history 0/6 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows_testList_0
hotspot_jre_0 => deep history 0/2 passed | possible issues
jdk_jmx_0 => deep history 0/3 passed | possible issues
jdk_jfr_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_sanity.system_arm_linux
TestJlmRemoteClassAuth_2 => deep history 12/13 passed | possible issues
TestJlmRemoteClassNoAuth_2 => deep history 8/13 passed | possible issues
TestJlmRemoteMemoryAuth_2 => deep history 10/13 passed | possible issues
TestJlmRemoteMemoryNoAuth_2 => deep history 7/13 passed | possible issues
TestJlmRemoteThreadNoAuth_2 => deep history 9/13 passed | possible issues

Test_openjdk8_hs_sanity.perf_x86-64_solaris
renaissance-scala-kmeans_0 => deep history 0/7 passed | possible issues

[Test_openjdk8_hs_sanity.openjdk_arm_linux] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_arm_linux/247/) Triage Done
jdk_util_2 => deep history 1/13 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows_testList_1
jdk_jmx_1 => deep history 0/3 passed | possible issues
jdk_jfr_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_linux
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_2
hotspot_jre_1 => deep history 0/4 passed | possible issues
jdk_management_0 => deep history 0/3 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_1
jdk_management_1 => deep history 0/3 passed | possible issues
jdk_jmx_1 => deep history 0/3 passed | possible issues
jdk_jfr_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_1
jdk_jfr_2 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_arm_linux
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_1
hotspot_jre_2 => deep history 0/2 passed | possible issues
jdk_net_2 => deep history 1/4 passed | possible issues
jdk_jfr_2 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_solaris
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_2
jdk_beans_2 => deep history 1/2 passed | possible issues
jdk_nio_2 => deep history 1/4 passed | possible issues
jdk_tools_2 => deep history 0/4 passed | possible issues
jdk_imageio_2 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0
hotspot_jre_0 => deep history 0/2 passed | possible issues
hotspot_jre_1 => deep history 0/2 passed | possible issues
hotspot_jre_2 => deep history 0/2 passed | possible issues
jdk_net_0 => deep history 0/2 passed | possible issues
jdk_net_1 => deep history 0/2 passed | possible issues
jdk_net_2 => deep history 0/2 passed | possible issues
jdk_nio_1 => deep history 1/2 passed | possible issues
jdk_security2_0 => deep history 0/2 passed | possible issues
jdk_security2_1 => deep history 0/2 passed | possible issues
jdk_security2_2 => deep history 0/2 passed | possible issues
jdk_jmx_0 => deep history 0/2 passed | possible issues
jdk_jmx_1 => deep history 0/2 passed | possible issues
jdk_jmx_2 => deep history 0/2 passed | possible issues
jdk_rmi_0 => deep history 0/2 passed | possible issues
jdk_rmi_1 => deep history 0/2 passed | possible issues
jdk_rmi_2 => deep history 0/2 passed | possible issues
jdk_tools_0 => deep history 0/2 passed | possible issues
jdk_tools_1 => deep history 0/2 passed | possible issues
jdk_tools_2 => deep history 0/2 passed | possible issues
jdk_jfr_0 => deep history 0/2 passed | possible issues
jdk_jfr_1 => deep history 0/2 passed | possible issues

  • extended.openjdk on x64_solaris times out at 10hrs, increased default TIME_LIMIT to 20 hrs to see results

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_2
hotspot_jre_1 => deep history 0/3 passed | possible issues
jdk_beans_1 => deep history 1/2 passed | possible issues
jdk_imageio_0 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64le_linux
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_x86-64_mac_testList_2
jdk_instrument_0 => deep history 0/3 passed | possible issues

Test_openjdk8_hs_extended.openjdk_aarch64_linux
hotspot_jre_0 => deep history 0/4 passed | possible issues
hotspot_jre_1 => deep history 1/4 passed | possible issues
jdk_management_0 => deep history 0/4 passed | possible issues
jdk_management_1 => deep history 0/4 passed | possible issues
jdk_jmx_0 => deep history 0/4 passed | possible issues
jdk_jmx_1 => deep history 0/4 passed | possible issues
jdk_jfr_0 => deep history 0/4 passed | possible issues
jdk_jfr_1 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.perf_arm_linux
dacapo-xalan_0 => deep history 1/4 passed | possible issues
renaissance-als_0 => deep history 0/4 passed | possible issues
renaissance-chi-square_0 => deep history 0/4 passed | possible issues
renaissance-dec-tree_0 => deep history 0/4 passed | possible issues
renaissance-gauss-mix_0 => deep history 0/4 passed | possible issues
renaissance-log-regression_0 => deep history 0/4 passed | possible issues
renaissance-movie-lens_0 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_1
jdk_security4_1 => deep history 0/3 passed | possible issues
jdk_jmx_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_2
hotspot_jre_1 => deep history 0/3 passed | possible issues
jdk_management_0 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_sanity.openjdk_ppc64_aix
jdk_util_0 => deep history 9/13 passed | possible issues
jdk_util_1 => deep history 9/13 passed | possible issues

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_0
jdk_management_2 => deep history 0/4 passed | possible issues
jdk_jmx_2 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_0
hotspot_jre_0 => deep history 0/2 passed | possible issues
jdk_security4_0 => deep history 0/2 passed | possible issues
jdk_jmx_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-32_windows
⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_sanity.system_x86-64_solaris
TestJlmRemoteMemoryAuth_0 => deep history 5/7 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_mac_testList_1
jdk_nio_1 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0 Triage done
hotspot_jre_0 => deep history 0/1 passed | possible issues
jdk_jmx_0 => deep history 0/2 passed | possible issues
jdk_jfr_1 => deep history 0/1 passed | possible issues

Test_openjdk8_hs_extended.openjdk_sparcv9_solaris_testList_0
hotspot_jre_0 => deep history 0/2 passed | possible issues
jdk_security2_0 => deep history 0/2 passed | possible issues
jdk_jmx_0 => deep history 0/2 passed | possible issues
jdk_jfr_0 => deep history 0/2 passed | possible issues
jdk_imageio_0 => deep history 0/2 passed | possible issues

[Test_openjdk8_hs_sanity.openjdk_x86-64_windows] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/509/) Triage Done
⚠️ Test Job Failed ⚠️

[Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_0] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_0/10/) Triage Done
jdk_jmx_2 => deep history 0/4 passed | possible issues

[Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/9/) Triage Done
jdk_management_1 => deep history 1/2 passed | possible issues
jdk_jmx_1 => deep history 0/2 passed | possible issues
jdk_jfr_0 => deep history 0/1 passed | possible issues

@smlambert
Copy link
Contributor Author

Note: jdk_jfr target seems to be failing on many platforms with "needs -nativepath" error (jdk/jfr/event/sampling/TestNative.java testcase in that target that requires it), perhaps need to do a similar thing that has been done for jdk_foreign_native target? or simply exclude jdk/jfr/event/sampling/TestNative.java in problem list if its failing everywhere.

FYI @sophia-guo

@smlambert
Copy link
Contributor Author

smlambert commented Jul 19, 2021

jdk_jmx target failing with:

[2021-07-17T14:16:43.687Z] STDERR:
[2021-07-17T14:16:43.687Z] /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_0/aqa-tests/openjdk/openjdk-jdk/jdk/test/javax/management/remote/mandatory/connection/RMIConnector_NPETest.java:64: error: method shutdown in class RMID cannot be applied to given types;
[2021-07-17T14:16:43.687Z]             rmid.shutdown(rmidPort);
[2021-07-17T14:16:43.687Z]                 ^
[2021-07-17T14:16:43.687Z]   required: no arguments
[2021-07-17T14:16:43.687Z]   found: int
[2021-07-17T14:16:43.687Z]   reason: actual and formal argument lists differ in length
[2021-07-17T14:16:43.687Z] 1 error
[2021-07-17T14:16:43.687Z] 

This is https://bugs.openjdk.java.net/browse/JDK-8066588, which appears resolved, check why we do not have the fix in our mirror.

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 19, 2021

We don't build test images for jdk8 so I will exclude it for 8.

Curious thing is the test isn't excluded for jdk11 however it doesn't run in jdk11. https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_hs_extended.openjdk_x86-64_linux_testList_2/7/testReport/

Tried with jdk_custom with jdk11 and looks like it failed. https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/152/

Turns out for jdk11 the test was excluded in upstream https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/test/jdk/ProblemList.txt#L959

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 19, 2021

Re #2754 (comment)

Yes, looks like it was backported to jdk8

JDK-8263145 | 8u301 | Yoshiki Sato | P3 | Resolved | Fixed | b01
https://bugs.openjdk.java.net/browse/JDK-8263145

Though I don't quite understand the backport process. Check the HG https://hg.openjdk.java.net/jdk8u/jdk8u/ and github https://github.com/openjdk/jdk8u/blob/master/jdk/test/javax/management/remote/mandatory/connection/RMIConnector_NPETest.java the change is not there.

Exclude the test as the backport should be propagated correctly in upstream repo.

@smlambert
Copy link
Contributor Author

Issues on x86_solaris:

  • Looks like there is an issue with creating TAP reports on the x86_solaris machines (likely a machine config issue or jenkins user permission issue).
  • jdk_jfr testcases causing crashes (see core files generated in attached test output file) / should exclude this target on this platform for now
  • hitting the 10hr timeout for some test runs (possibly because of the many crashes in jdk_jfr). I have increased the TIME_LIMIT to 20hrs.
  • sun/tools/jstatd testcases failing when trying to run jps, with malformed host identifier usage error
Start jps:
[/export/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/openjdkbinary/j2sdk-image/bin/jps -J-d64 -J-XX:+UsePerfData -v build-packet_esxi-solaris10u11-x64-2.adoptopenjdk.net:46975]
Malformed Host Identifier: build-packet_esxi-solaris10u11-x64-2.adoptopenjdk.net:46975
usage: jps [-help]
       jps [-q] [-mlvV] [<hostid>]

Definitions:
    <hostid>:      <hostname>[:<port>]
  • com/sun/crypto/provider/Mac/MacClone.java fails with java.lang.CloneNotSupportedException: SHA-256

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 19, 2021

Issues for perf on sparcv9_solaris: https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.perf_x86-64_solaris/10/console

Test renaissance-scala-kmeans_0 failed without any error message. Suspect the way the test is triggered may have some issue. Looking at the passing test on linux, there is message Warning!!! Affinity is NOT set. Affinity tool may NOT be installed/supported.
Believe that something wrong with dynamically set CPU affinities added in #1917

08:32:18  Warning!!! Affinity is NOT set. Affinity tool may NOT be installed/supported.
08:32:18  Running EXEC_CMD_WITH_AFFINITY= /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/openjdkbinary/j2sdk-image/bin/java -jar /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/aqa-tests/TKG/../../jvmtest/perf/renaissance/renaissance.jar --json /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/aqa-tests/TKG/../TKG/output_16265245182190/renaissance-scala-kmeans_0/scala-kmeans.json scala-kmeans

Will remove dynamically setting CPU affinities and need someone to look at the issues in run_with_affinity.sh

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 20, 2021

jdk_util test java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java intermittently failed on arm build
Test_openjdk8_hs_sanity.openjdk_arm_linux
It is also machine related. Suspect related with arm memory size? And this is not a regression failure, which should not block the release, will exclude it? @smlambert

1/10 failed build-equinix-ubuntu1604-armv7l-1 : https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/162/tapResults/
7/10 failed test-aahlenst-armv7-raspbian-pi4-1 : https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/163/tapResults/

09:19:54  test org.openjdk.tests.java.util.stream.MapOpTest.testOps("ArrayList.asList:pseudorandom", java.util.stream.TestData$AbstractTestData$RefTestData@98fbd2): failure
09:19:54  java.lang.AssertionError: 1 failure(s) for test data: RefTestData[ArrayList.asList:pseudorandom]
09:19:54  1: java.lang.Error: PAR_STREAM_FOR_EACH_CLEAR_SIZED: java.lang.ArrayIndexOutOfBoundsException
09:19:54  java.lang.ArrayIndexOutOfBoundsException

@smlambert
Copy link
Contributor Author

re: #2754 (comment) yes, thanks @sophia-guo

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 20, 2021

Test_openjdk8_hs_sanity.openjdk_x86-64_windows failed with fatal: unable to access 'https://github.com/eclipse-openj9/openj9.git/'. Either it's a network issue or a machine issue? Re-ran: will see if there are specific test failures

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/510/

09:38:22  get functional test material...
09:38:22  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
09:38:45  fatal: unable to access 'https://github.com/eclipse-openj9/openj9.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 

Test_openjdk8_hs_sanity.openjdk_x86-64_windows : rerun green

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 21, 2021

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1
jdk_management_1 => deep history 1/2 passed | possible issues

Test com/sun/management/OperatingSystemMXBean/TestTotalSwap.java fails when run in container exclude for platform run in container. Suppose only for linux we run tests in docker.

@sophia-guo
Copy link
Contributor

sophia-guo commented Jul 21, 2021

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0

hotspot_jre_0 => deep history 0/1 passed | possible issues:

Following test failures were reported in https://bugs.openjdk.java.net/browse/JDK-8180723?focusedCommentId=14086592&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14086592 . Closed as cannot reproduce, which should be reopen.

compiler/rtm/locking/TestUseRTMAfterLockInflation.java.TestUseRTMAfterLockInflation
compiler/rtm/locking/TestUseRTMForInflatedLocks.java.TestUseRTMForInflatedLocks
compiler/rtm/locking/TestUseRTMForStackLocks.java.TestUseRTMForStackLocks
compiler/rtm/method_options/TestUseRTMLockElidingOption.java.TestUseRTMLockElidingOption

jdk/jfr/event/compiler/TestCompilerCompile.java.TestCompilerCompile : timeout. Running in the docker. Rerun on nondocker, passed:
https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/171/

Program `/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960351ms)

jdk/jfr/jcmd/TestJcmdDumpPathToGCRoots.java error message: "Execution failed: `main' threw exception: java.lang.Exception: Expected chains but found none " issue opened #2766

@sophia-guo
Copy link
Contributor

Test_openjdk8_hs_extended.openjdk_x86-32_windows
Failed tests specific to x86-32:

jdk/jfr/event/compiler/TestCompilerPhase.java.TestCompilerPhase

 Stack TraceExecution failed: `main' threw exception: java.lang.RuntimeException: No events: expected false, was true  Standard Output Standard Error 2 sec 2
jdk/jfr/event/oldobject/TestObjectDescription.java.TestObjectDescription 

Stack TraceProgram `E:\workspace\Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_1\openjdkbinary\j2sdk-image\bin\java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960006ms).  Standard Output Standard Error

@karianna
Copy link
Contributor

Recently for jdk8u312-b07 - the same security failures happened https://trss.adoptium.net/output/test?id=616ff8ccc6182d02178e14af

@SehrishHussain
Copy link
Contributor

Another test fail related to jdk-jfr for openjdk18, ppc64le_linux:
jdk/jfr/jmx/TestGetRecordings.java

Jenkins URL: https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.openjdk_ppc64le_linux_testList_0/7/#showFailuresLink
TRSS Link: https://trss.adoptium.net/allTestsInfo?buildId=616d7c2bc6182d021780004e&limit=5&hasChildren=true

@smlambert
Copy link
Contributor Author

Closing this as both 'completed' and stale.

@zzambers
Copy link
Contributor

We don't build test images for jdk8 so I will exclude it for 8.

Jdk8 actually does not have test images.
Test was fixed by JDK-8301143:
jdk/jfr/event/sampling/TestNative.java

however I have found that it still fails on MacOS in grinder:

...
14:57:40  + OS=Darwin
14:57:40  + '[' Darwin '!=' Linux ']'
14:57:40  + '[' Darwin '!=' Darwin ']'
14:57:40  + '[' x = x ']'
14:57:40  +++ which gcc
14:57:40  ++ readlink -f /usr/bin/gcc
14:57:40  readlink: illegal option -- f
14:57:40  usage: readlink [-n] [file ...]
14:57:40  + TESTGCC=
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants