Elasticsearch tests requires jdk not jre

[smlambert]

Elasticsearch jdk8 tests are failing as we are basing our test docker images from AdoptOpenJDK docker images that are jre not jdk builds. Our images are using xxx but should maybe use for jdk11: https://github.com/AdoptOpenJDK/openjdk-docker/blob/master/11/jdk/ubuntu/Dockerfile.openj9.nightly.full (jdk11u-nightly tag)
for jdk8: https://github.com/AdoptOpenJDK/openjdk-docker/blob/master/8/jdk/ubuntu/Dockerfile.openj9.nightly.full (jdk8u-nightly tag)

From recent grinder output:
23:32:57 Using docker image default Java
23:32:57 JAVA_BIN is: /opt/java/openjdk/bin
23:32:57 openjdk version "1.8.0_202"
23:32:57 OpenJDK Runtime Environment (build 1.8.0_202-b08)
23:32:57 Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190205_213 (JIT enabled, AOT enabled)
23:32:57 OpenJ9 - 90dd8cb40
23:32:57 OMR - d2f4534b
23:32:57 JCL - d002501a90 based on jdk8u202-b08)

23:37:15 FAILURE: Build failed with an exception.
23:37:15
23:37:15 * What went wrong:
23:37:15 A problem occurred configuring project ':distribution'.
23:37:15 > jps executable not found; ensure that you're running Gradle with the JDK rather than the JRE
23:37:15

Additionally, there was a workaround added at https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/thirdparty_containers/elasticsearch/dockerfile/elasticsearch-test.sh#L59-L63 for OpenJ9 prior to that implementation supporting jps tool. That workaround may not actually be working, as seen by this output:
23:32:58 /elasticsearch-test.sh: line 61: /opt/java/openjdk/bin/jps: Permission denied
23:32:58 chmod: cannot access '/opt/java/openjdk/bin/jps': No such file or directory
but should no longer be needed as OpenJ9 now has the jps tool.

From another Grinder output, jdk11 Elasticsearch tests fail with:
02:28:39 FAILURE: Build failed with an exception.
02:28:39
02:28:39 * What went wrong:
02:28:39 Could not determine java version from '11.0.2'.

[sophia-guo]

This is actually a common issue. The test hotspot docker images from AdoptOpenJDK with 8 and 11 is JDK. However the test openj9 docker images from AdoptOpenJDK with 8 and 11 is jre. Will get more information and open an issue in https://github.com/AdoptOpenJDK/openjdk-docker, which suppose provide JDK as default.

[smlambert]

I think we just need to use the right tag in order to pull the correct docker image from docker hub, otherwise we just pull "latest" tag which is for a jre image.

https://hub.docker.com/r/adoptopenjdk/openjdk8-openj9/tags
https://medium.freecodecamp.org/an-introduction-to-docker-tags-9b5395636c2a

Usage:
FROM imageName[:tagName]

[sophia-guo]

The thing is for hotspot we are also pulling "latest", which is JDK image instead. Guess there is difference between hotspot and openj9, say hotspot is a default one and corresponding has different behavior? Will try to get more information.

[sophia-guo]

Guess https://github.com/AdoptOpenJDK/openjdk-docker readme is out of date

[sophia-guo]

There are jre (102M) and latest (171M) tag under https://hub.docker.com/r/adoptopenjdk/openjdk8-openj9/tags. So latest should be for JDK?

[smlambert]

Best to double-check what we are downloading and then check with @dinogun and/or raise an openjdk-docker issue then.

[sophia-guo]

Another issue is I'm thinking should we also use nightly tag instead of latest as we expect to run the docker image every night?

[smlambert]

I think selecting which tag to use is also related to #773 - what build job or pipeline should be triggering these tests. Currently they are triggered from the main build pipelines, but the docker builds are not built via the main build pipelines, so we need to also correct that.

What build job creates the nightly images at docker hub? and what is the best way to trigger these test jobs from it?

@dinogun @karianna @johnoliver - is there an intent to integrate docker builds into the overall build pipeline? or will they remain separate?

[karianna]

I think selecting which tag to use is also related to #773 - what build job or pipeline should be triggering these tests. Currently they are triggered from the main build pipelines, but the docker builds are not built via the main build pipelines, so we need to also correct that.

What build job creates the nightly images at docker hub? and what is the best way to trigger these test jobs from it?

@dinogun @karianna @johnoliver - is there an intent to integrate docker builds into the overall build pipeline? or will they remain separate?

I would argue the docket builds should also go in, as when cutting down to make slim images we could introduce regressions. I think we'll need to carefull omit certain test areas though.

[sophia-guo]

jre issue opened. AdoptOpenJDK/openjdk-docker#122

[smlambert]

Agree with #974 (comment) - we should clearly state which tests use which images and make a plan to test a range of the ones produced.

[sophia-guo]

Using nightly in my personal branch (also delete the workaround) got:

20:24:23  BUILD SUCCESSFUL in 11m 1s
20:24:23  473 actionable tasks: 473 executed
20:24:23  Elasticsearch Build - Completed
20:24:23  Running elasticsearch tests :

and

20:34:16  JUnit4 test failed, ant output was:
20:34:16     [junit4] <JUnit4> says olá! Master seed: 4303BBF64C04856A
20:34:16     [junit4] JVM J0:     1.30 ..   261.08 =   259.77s
20:34:16     [junit4] JVM J1:     1.55 ..   264.43 =   262.89s
20:34:16     [junit4] JVM J2:     1.17 ..   261.52 =   260.34s
20:34:16     [junit4] Execution time total: 4 minutes 24 seconds
20:34:16     [junit4] Tests summary: 1017 suites (1 ignored), 6554 tests, 1 suite-level error, 80 ignored (80 assumptions)
20:34:16  
20:34:16  :core:test FAILED
20:34:16  
20:34:16  FAILURE: Build failed with an exception.
20:34:16  
20:34:16  * What went wrong:
20:34:16  Execution failed for task ':core:test'.
20:34:16  > There were test failures: 1017 suites (1 ignored), 6554 tests, 1 suite-level error, 80 ignored (80 assumptions) [seed: 4303BBF64C04856A]
20:34:16  
20:34:16  * Try:
20:34:16  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
20:34:16  
20:34:16  * Get more help at https://help.gradle.org

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1278/consoleFull

[sophia-guo]

For jdk11 failure "Could not determine java version from '11.0.2'.' Looks like a gradle support issue,

https://stackoverflow.com/questions/54358107/gradle-could-not-determine-java-version-from-11-0-2

gradle/gradle#8671

Update gradle version may helps. I will try that.

[sophia-guo]

Revisit by trying with adoptopenjdk/openjdk8-openj9:latest and original test material can't reproduce the error 'ps executable not found; ensure that you're running Gradle with the JDK rather than the JRE'. https://ci.adoptopenjdk.net/view/work%20in%20progress/job/Grinder_Sandbox/314/consoleFull

Will update.

[sophia-guo]

Discussion about #974 (comment) will be continued in #773.