Skip to content

Reproducing Test Failures Locally

Jump to bottom
Lan Xia edited this page May 30, 2024 · 36 revisions

General steps

These are the general steps that we apply to running a great variety of tests. We use TestKitGen, a thin veneer used to standardize the diverse set of test frameworks employed by the underlying tests.

  1. Ensure test machine is set up with test prereqs.

  2. Build or download/unpack the SDK you want to test to /someLocation

  3. export TEST_JDK_HOME=/someLocation // set test JDK home. On windows, the windows path format is expected. (i.e., TEST_JDK_HOME=C:\someLocation )

  4. git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)

  5. cd aqa-tests

  6. ./get.sh

  7. cd TKG

  8. export environment variables suitable for the SDK under test (i.e., export BUILD_LIST=functional )

  9. make compile // fetches test material and compiles it, based on build.xml files in the test directories

  10. make _< someTestTarget > // executes the test target (can be test group, level, level.group or specific test). i.e., openjdk (all tests in openjdk group), sanity.functional (all functional tests labelled at sanity level), extended.system (all system tests labelled at extended level), jdk_math (the specific jdk_math target defined as part of openjdk group), MauveMultiThreadLoadTest_0 (the first variation of the specific system test called MauveMultiThreadLoadTest), etc

Degrees of Freedom in Testing

Below is a list of common scenarios. As you can see, the approach to running the different groups of tests is very similar. With minimal changes to environment variables and target names, a great variety of test scenarios can be run. These instructions can be used for running any group of tests on any JDK version, any JDK implementation, and any platform/spec. For more details, please see the OpenJ9 Test Documentation.

There are a number of degrees of freedom by which you can generate, compile and execute test targets. Some variants are:

  • test group [functional|system|openjdk|external|perf]
  • level [sanity|extended|special]
  • jdk version - [8|9|10|11]
  • jdk impl - [openj9|hotspot|sap|ibm]
  • spec - [linux_x86-64_cmprssptrs|linux_x86-64| etc] - as listed in buildspecs

JDK_VERSION, JDK_IMPL and SPEC can be auto-detected by our test framework, so setting those environment variables explicitly is no longer required.

If needed, users can turn off the auto-detection by setting AUTO_DETECT=off. If users set AUTO_DETECT off, then JDK_VERSION, JDK_IMPL and SPEC values are required.

Specific Test recipes (demonstrative, not an exhaustive list)

Test group JDK version JDK impl Platform/Spec Env variables
functional 8 openj9 x64_linux/linux_x86-64_cmprssptrs export TEST_JDK_HOME=/someLocation
export BUILD_LIST=functional
system 10 openj9 x64_linux/linux_x86-64_cmprssptrs export TEST_JDK_HOME=/someLocation
export BUILD_LIST=system
openjdk 11 hotspot x64_mac/macos_x86-64 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=openjdk
openjdk 11 openj9 s390x_linux/linux_390-64_cmprssptrs export TEST_JDK_HOME=/someLocation
export BUILD_LIST=openjdk
export JDK_CUSTOM_TARGET=java/lang/Class/GetModuleTest.java
external 8 openj9 x86_linux/linux_x86-64_cmprssptrs export TEST_JDK_HOME=/someLocation
export BUILD_LIST=external
perf 11 openj9 x86_linux/linux_x86-64_cmprssptrs export TEST_JDK_HOME=/someLocation
export BUILD_LIST=perf

Functional tests

Run extended functional tests on jdk8_x86-64_linux_openj9 SDK

  • Build or unpack the jdk8_x86-64_linux_openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export BUILD_LIST=functional
  • make compile
  • make _extended.functional
export NATIVE_TEST_LIBS=<path to native test libraries> (default to native-test-libs folder at same level as TEST_JDK_HOME)

Run the sanity tests in the functional/JLM_Tests directory on jdk10_x86-64_linux_openj9 SDK

  • Build or unpack the jdk10_x86-64_linux_openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export BUILD_LIST=functional/JLM_Tests
  • make compile
  • make _sanity

System tests

Run sanity system tests on jdk10_x86-64_linux_openj9 SDK

  • Build or unpack the jdk10_x86-64_linux_openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export BUILD_LIST=system
  • make compile
  • make _sanity.system

OpenJDK tests

Run sanity openjdk tests on jdk11_x64_macos_hotspot SDK

  • Build or unpack the jdk11_x64_macos_hotspot SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export BUILD_LIST=openjdk
  • make compile
  • make _sanity.openjdk

Note: if tests need native libraries, please export test image path. This is used to construct -nativepath in openjdk.mk. For example,

export TESTIMAGE_PATH = "$WORKSPACE/jdkbinary/openjdk-test-image" (to /yourTestImageLocation)

Run specific test class from openjdk tests on jdk11-linux-s390x-openj9 SDK

For this example, let's run a currently excluded test, java/lang/Class/GetModuleTest.java found in exclude file

  • Build or unpack the jdk11-linux-s390x-openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export environment variables matching the SDK under test
export BUILD_LIST=openjdk 
export TARGET=jdk_custom

export JDK_CUSTOM_TARGET=test/jdk/java/lang/Class/GetModuleTest.java // <- in jdk9+, tests are in test/jdk/*

or

export JDK_CUSTOM_TARGET=jdk/test/java/lang/Class/GetModuleTest.java // <- in jdk8, tests are in jdk/test/*

depends on openjdk test path

  • make compile
  • make _jdk_custom

External tests

Run scala tests in external group on jdk8-x86-64_linux_openj9 SDK (in Docker container)

  • Build or unpack the jdk8-x86-64_linux_openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG
  • export environment variables matching the SDK under test ( export BUILD_LIST=external/scala and export EXTRA_DOCKER_ARGS="-v $TEST_JDK_HOME:/opt/java/openjdk" )
  • make compile
  • make _scala_test

Perf tests

Run idle_micro bench in perf group on jdk11-x86-64_linux_openj9 SDK

  • Build or unpack the jdk11-x86-64_linux_openj9 SDK to /someLocation
  • export TEST_JDK_HOME=/someLocation
  • git clone https://github.com/adoptium/aqa-tests.git (to /testLocation)
  • cd aqa-tests
  • ./get.sh
  • cd TKG export BUILD_LIST=perf/idle_micro
  • make compile
  • make _IdleMicrobenchmark_J9
Clone this wiki locally