Support JUnit 4 via the VintageTestEngine

[dzou]

Many thanks for this great project.

I wanted to ask about junit-platform-native: Is there future plans to make this compatible with JUnit4 running tests via junit-vintage-engine?

[sbrannen]

The initial goal is support for the JUnit Platform in general, with specific support for JUnit Jupiter.

However, some preliminary work has been done to provide built-in support for JUnit Vintage:

native-build-tools/common/junit-platform-native/src/main/java/org/graalvm/junit/platform/JUnitPlatformFeature.java

Lines 94 to 95 in acf1474

	RuntimeClassInitialization.initializeAtBuildTime("org.junit.vintage.engine.support.UniqueIdReader");
	RuntimeClassInitialization.initializeAtBuildTime("org.junit.vintage.engine.support.UniqueIdStringifier");

If you are experiencing specific issues, please provide a sample application demonstrating what you are encountering or provide stack traces, debug output, etc. that could help the team assess the problem.

As I mentioned above, the current "support" for JUnit Vintage is preliminary which means you likely have to add some manual configuration for yourself -- for example, reflection config for JUnit Vintage internals. Though, in theory it should be possible to get JUnit 4 tests running via the JUnit Vintage TestEngine.

[dzou]

I see, thanks for the clarification. I'll give running the JUnit4 tests a try and followup in this thread what configurations I used if I'm able to get it working.

[dzou]

I played around with it and was able to get JUnit 4 working with the following additional classes marked for build-time initialization:

--initialize-at-build-time=org.junit.runner.Description,org.junit.vintage.engine.descriptor.RunnerTestDescriptor,org.junit.runners.JUnit4,org.junit.runners.BlockJUnit4ClassRunner

Link to example: https://github.com/dzou/junit4-experiment

I'll try this out on a bigger set of tests and see if it still works.

[sbrannen]

I played around with it and was able to get JUnit 4 working with the following additional classes marked for build-time initialization:

Great!

Thanks for sharing, and please do keep us posted on your progress.

See also: #59 (comment)

[gradinac]

Hey @dzou! We've added really basic support for the JUnit Vintage engine as part of #59 - could you try out the latest changes on master and see if they work for your samples?

While it's at a really basic level right now, it should hopefully grow over time and encompass more JUnit features out of the box

[dzou]

Great! I'll give it a try and will followup.

[dzou]

@gradinac -- Hey, thanks for making those changes. It works great for the simple test repo!

I just tried it on a more complex repo of tests here: https://github.com/dzou/java-pubsub
(To try locally, cd google-cloud-pubsub and run mvn test -Pnative.)

I am getting an error that looks like this:

[junit-platform-native] Running in 'test discovery' mode. Note that this is a fallback mode.
Fatal error:org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:111)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:85)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:92)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:67)
	at org.graalvm.junit.platform.JUnitPlatformFeature.registerTestPlan(JUnitPlatformFeature.java:121)
	at org.graalvm.junit.platform.JUnitPlatformFeature.beforeAnalysis(JUnitPlatformFeature.java:89)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:701)
	at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:701)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:563)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:476)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.NoClassDefFoundError: org/apache/log/Logger
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
	at java.base/java.lang.Class.privateGetPublicMethods(Class.java:3191)
	at java.base/java.lang.Class.getMethods(Class.java:1904)
	at org.junit.platform.commons.util.ReflectionUtils.getDefaultMethods(ReflectionUtils.java:1481)
	at org.junit.platform.commons.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:1454)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1396)
	at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1380)
	at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1366)
	at org.junit.vintage.engine.descriptor.TestSourceProvider.lambda$findMethod$1(TestSourceProvider.java:64)
	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1133)
	at java.base/java.util.Collections$SynchronizedMap.computeIfAbsent(Collections.java:2682)
	at org.junit.vintage.engine.descriptor.TestSourceProvider.findMethod(TestSourceProvider.java:64)
	at org.junit.vintage.engine.descriptor.TestSourceProvider.findTestSource(TestSourceProvider.java:45)
	at org.junit.vintage.engine.discovery.RunnerTestDescriptorPostProcessor.addChildrenRecursively(RunnerTestDescriptorPostProcessor.java:63)
	at org.junit.vintage.engine.discovery.RunnerTestDescriptorPostProcessor.applyFiltersAndCreateDescendants(RunnerTestDescriptorPostProcessor.java:45)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.junit.vintage.engine.discovery.VintageDiscoverer.discover(VintageDiscoverer.java:50)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:63)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:103)
	... 16 more
Caused by: java.lang.ClassNotFoundException: org.apache.log.Logger
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 46 more

I guess the puzzling part is that the class it is complaining about org.apache.log.Logger isn't on my project classpath; I'm curious if you might have seen a similar issue before.

I have also gotten this error for the com.oracle.mxtool.junit.MxJUnitRequest$Builder if you try running the mvn test -Pnative command at the root directory of this (junit4 branch): https://github.com/dzou/junit4-experiment/tree/junit4
Also not on my classpath. I'm wondering where these classes are coming from.

[dzou]

I did some further debugging, trying to understand what tests the junit vintage engine picks up.

I basically found that once we get into VintageDiscover.java there's a lot of random log and provider classes that get picked up in the RunnerTestDescriptors. Then you see later in the list it's normal test classes:

And then I'm failing when it tries to resolve the org.apache.log.Logger reference from that LogKitLogger class. I am just curious why non tests classes end up getting picked up.

EDIT: Oh, I guess it must be because I'm falling back into test-discovery mode. I still see this error even though I have this dependency added. Maybe I am missing something; will debug a bit more tomorrow.

[WARNING] Test configuration file wasn't found. Build will now fallback to test discovery mode.
[WARNING] Add following dependency to use the test listener mode:
[WARNING] <dependency>
[WARNING]     <groupId>org.graalvm.buildtools</groupId>
[WARNING]     <artifactId>junit-platform-native</artifactId>
[WARNING]     <version>0.9.2-SNAPSHOT</version>
[WARNING]     <scope>test</scope>
[WARNING] </dependency>

EDIT 2: Something also worth noting is that JUnitPlatformFeature (line 89) the access.getApplicationClassPath() will also include the GraalVM JAR under the lib/svm/ folder on the classpath, which seems to explain where the com.oracle.mxtool.junit.MxJUnitRequest error was coming from.

[dzou]

Ok, I think I got it to work.

It looks like in my project setup something got messed up with the maven-surefire-plugin. In my project, my maven-surefire-plugin is defined in a parent pom two levels up. And then it seemed like when I did mvn test -Pnative, it didn't do the work of generating that test ID file I suppose. Or maybe it did but not in the expected place? Not sure. Here's the project I was working on: https://github.com/dzou/java-pubsub

So to fix the problem, I define it again in the native profile right before the native-maven-plugin and it seemed to work!

My setup below:

 <profile>
      <id>native</id>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
          </plugin>
          <plugin>
            <groupId>org.graalvm.buildtools</groupId>
            <artifactId>native-maven-plugin</artifactId>
            <version>0.9.2-SNAPSHOT</version> <!-- or newer version -->
            <executions>
              <execution>
                <id>test-native</id>
                <goals>
                  <goal>test</goal>
                </goals>
                <phase>test</phase>
              </execution>
            </executions>
            <configuration>
              <buildArgs>
                <buildArg>--no-fallback</buildArg>
                <buildArg>--no-server</buildArg>
              </buildArgs>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

But yes, it looks JUnit4 is working well @gradinac 🙂.

[gradinac]

Hey @dzou! Thank you very much for the detailed analysis! :) It seems like the tests were indeed running in the discovery mode - this particular one seems to come from the JUnit test discovery internals. For the discovery mode, we will most likely need to somehow tell JUnit to ignore these failures and just skip over the descriptor from which they originated.
However, as you've also mentioned, this is a fallback mode and is currently not meant to be used unless absolutely necessary - we are also working on discovering tests without needing to actually run them in the default mode as part of #76

As for JUnit4, that's great news! :) We currently have the most basic tests in place for it and no additional automatic configuration , but hopefully that support will grow over time

[dzou]

Cool. I'll close this for now since JUnit4 looks to be supported well; if any issues emerge in the future then people can file separate issues.

[sbrannen]

@dzou, thanks for providing the detailed analysis of the errors you encountered!

It seems like the tests were indeed running in the discovery mode - this particular one seems to come from the JUnit test discovery internals. For the discovery mode, we will most likely need to somehow tell JUnit to ignore these failures and just skip over the descriptor from which they originated.

This actually stems from the fact that the JUnitPlatformFeature does not configure an "include class name" filter.

For example, the JUnit Platform ConsoleLauncher always configures the standard include pattern (if the user does not override that default with custom patterns), as can be seen here:

https://github.com/junit-team/junit5/blob/b5e2f16916e4a339e8e3747c4af6f4b247dba6df/junit-platform-console/src/main/java/org/junit/platform/console/options/AvailableOptions.java#L178-L179

Build tools and IDEs do something similar. For details on what Maven Surefire includes by default, consult the official documentation.

Something also worth noting is that JUnitPlatformFeature (line 89) the access.getApplicationClassPath() will also include the GraalVM JAR under the lib/svm/ folder on the classpath, which seems to explain where the com.oracle.mxtool.junit.MxJUnitRequest error was coming from.

That's the other part of the puzzle. The JUnitPlatformFeature currently adds everything to the classpath roots that JUnit uses for classpath scanning.

The combination of scanning all JARs in the classpath and NOT having a class name or package name filter results in the JUnit Vintage TestEngine attempting to load every single class in the classpath via reflection in order to search for @Test methods.

Thus, if the fallback discovery mode is to be retained, it must properly configure either a class name or package name include filter, and it probably should not blindly add everything to the classpath for classpath scanning.

However, I believe that @lazar-mitrovic intends to remove the fallback discovery mode. So that may become a moot point.

/cc @melix as background information for #76.