Below is an example to run all sanity.functional tests against a JDK8 Linux x86-64 cmprssptrs OpenJ9 SDK:
cd openj9/test/TestConfig
export JAVA_BIN=/my/openj9/sdk/bin
make -f run_configure.mk // generates makefiles
make compile // downloads test related material/libs
// and compiles test material
make _sanity.functional // runs tests
Please read OpenJ9 Test User Guide for details and other examples. For example, if you wish to test a JDK11 version, you would export JDK_VERSION=11 (default JDK_VERSION=8).
Please read Prerequisites.md for details on what tools should be installed on your test machine to run tests.
While the OpenJ9 Test User Guide gives a more complete list of OpenJ9 test use cases, there are some frequently asked questions or common use cases by OpenJ9 developers listed below.
By default, make compile
compiles all tests. This is the safest way
to ensure all the test code needed has been compiled. However, there is a
way to shortcut the compilation process to reduce compilation time. If
BUILD_LIST
is set, make compile
will only compile the folder names
names that match within BUILD_LIST
.
export BUILD_LIST=functional/TestUtilities,functional/Java8andUp
make compile
Adding a testNG test as an example:
- adding a single test class to an existing directory
- update testng.xml to add the test class to a existing or create a new
- If the new is created in testng.xml, playlist.xml should
be updated to add the new based on playlist.xsd
Supported test groups are
functional|system|openjdk|external|perf|jck
. It is required to provide one group per test in playlist.xml.
- adding additional new test methods for new Java10 functionality
- test should be automatically picked up
Please refer to the video and tutorial that describes how to add container-based 3rd party application tests (run inside of Docker images). These tests are added and run in the automated test builds at the AdoptOpenJDK project.
In playlist.xml, to disable a test target, add
<disabled>Reason for disabling test, should include issue number</disabled>
inside the <test>
element that you want to disable.
- Disable an individual test class
- testNG test
add a line to
TestConfig/resources/excludes/latest_exclude_$(JDK_VERSION).txt
file with issue number and specific specs to disable
- testNG test
add a line to
org.openj9.test.java.lang.management.TestOperatingSystemMXBean 123 linux_x86-64
Test can be run with different levels, groups and types or combination of two (i.e., level.group, level.type, group.type) or three (i.e., level.group.type)
Supported levels are sanity|extended
Supported groups are functional|system|openjdk|external|perf|jck
Supported groups are regular|native
make _sanity
make _functional
make _extended.perf
make _sanity.native
make _extended.functional.native
If a test is disabled using <disbaled>
tag in playlist.xml, it can be executed through specifying the test target or add disabled
in front of regular target.
make _testA
make _disabled.sanity.functional
make _disabled.extended
The example below executes all of the sanity tests found within the JIT_Test directory
make -C ../functional/JIT_Test -f autoGen.mk _sanity
or
cd ../functional/JIT_Test; make -f autoGen.mk _sanity
Please read How-to Run customized JCK test targets for details.
User can run tests against different jdk version and/or jdk implementation. While the default values of these variables match a typical use case for OpenJ9 developers, there are also many cases where developers need to verify features for a specific version or compare behaviour against a particular implementation.
There is no extra step needed. By default, AUTO_DETECT is turned on, and the test framework will auto detect SPEC, JDK_IMPL, and JDK_VERSION. Please read Configure environment for details and examples.
- test results summary
At the end of each run, test results summary will be printed:
TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
cmdLineTester_javaAssertions_0
cmdLineTester_LazyClassLoading_0
...
FAILED test targets:
TestAttachAPIEnabling_SE80_0
TestFileLocking_SE80_0
TOTAL: 84 EXECUTED: 84 PASSED: 82 FAILED: 2 SKIPPED: 0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
You can find the failed test output in console output.
- TAP result
A simple standardized TAP output is produced at the end of a test run, to provide developers with a convenient summary of the test results. It is also necessary as the tests used to verify OpenJ9 use a variety of test output formats. This summary is a way to standardize the output which allows CI tools to present results in a common way.
SKIPPED
tests
If a test is skipped, it means that this test cannot be run on this platform due to jvm options, platform requirements and/or test capabilities.
failed.mk
will be generated if there is any failed test target.
We can rerun failed tests as following:
make _failed
failed.mk
will be over-written each test run. If you want to
'save it', you can make a copy of the generated failed.mk
file.