JUnit5 Integration for Scala

[sageserpent-open]

So far, if one wants to write a parameterised test in Scala using Americium, the way to do it is to choose a testing framework - so Scalatest, MUnit, uTest and then embed the entire supply within a single test body. This works, but:

1. The test framework doesn't know about each trial, so it doesn't show them; the whole test just passes or fails without any clear idea as to the progression through the trials.
2. There is no way of running a specific trial, in contrast with the JUnit5 integration in Java where IntelliJ or Visual Studio uses JUnit5 to allow clinking on an individual trial to run it in isolation.
3. The JUnit5 integration makes it obvious as to whether shrinkage is occurring or not.

Now, it is possible to use the existing JUnit5 integration in Scala code - and this can also pull in Scalatest assertions via scalatestplus-junit5 , but this needs the Trials instances to be defined directly in the test class via @TestInstance(Lifecycle.PER_CLASS), as the code in TrialsTestExtension looks for static fields in the test class, not in the Scala companion object instance. Furthermore, the Trials instances have to be converted to the Java API prior to being used directly via @TrialsTest, or being ganged together to make a configured SupplyToSyntax for use by @ConfiguredTrialsTest.

It all feels a bit hokey - and I personally do not like the loose association between named trials instances and Scala test function arguments; that is forced on folk writing Java as a necessary evil because of the way annotations are defined in Java, but it would be nice to be have proper type-checking on the Scala side.

A spike in Scala yields this:

  @TestFactory
  def foo: java.util.Iterator[DynamicNode] =
    (sourcesTrials.javaTrials and sourcesTrials.javaTrials and sourcesTrials.javaTrials and minimumSizeFractionTrials.javaTrials)
      .withLimit(100)
      .testIntegrationContexts()
      .asScala
      .map(context =>
        dynamicTest(
          s"Foo - ${context.caze}",
          () => println(s"Case: ${context.caze}")
        )
      )
      .asJava

This yields something similar to the Java JUnit5 integration experience in terms of seeing clickable trials in IntelliJ, but with some finessing could be packaged up into a principled Scala utility.

The use of JUnit5's dynamic tests sacrifices the full test lifecycle support that comes with test templates - so there is no support for @BeforeEach / @BeforeAfter at the level of each trial's execution, but Scala folk used to the Scalatest + Scalacheck integration already have to put up with this. They would use the functional approach to resource management and bracket the test code in either a simple try ... finally block, or use RAII either via the standard Using or perhaps Cats' Resource, so we can settle for just having JUnit5 manage setup / teardown at the suite level or around the entire test factory run.

[sageserpent-open]

Some spike code from a client project:

  @TestFactory
  def foo: util.Iterator[DynamicNode] =
    bar(
      sourcesTrials and sourcesTrials and sourcesTrials and minimumSizeFractionTrials,
      (caze: (FakeSources, FakeSources, FakeSources, Double)) =>
        if caze._4 > 0.7 then Trials.reject()
        else if caze._4 > 0.5 then throw new RuntimeException("Drat!")
        else println(s"Case: $caze")
    )

  private def bar[Caze](
      trials: TrialsScaffolding[Caze],
      block: Caze => Unit
  ): util.Iterator[DynamicNode] =
    trials.trials.javaTrials
      .withLimit(100)
      .testIntegrationContexts()
      .asScala
      .map(context =>
        dynamicTest(
          s"Foo - ${customPrettyPrinter(context.caze)}",
          () =>
            val eligible =
              try
                context
                  .inlinedCaseFiltration()
                  .executeInFiltrationContext(
                    () => block(context.caze()),
                    Array(classOf[TestAbortedException])
                  )
              catch
                case throwable: Throwable =>
                  context.caseFailureReporting().report(throwable)
                  throw throwable
              end try
            end eligible

            if !eligible
            then throw new TestAbortedException
            end if
        )
      )
      .asJava

Now this is messy because of the need to drop back to the Java API to get at the TestIntegrationContext instances, but some manual testing shows it does the job.

The plan is to introduce .testIntegrationContexts into the Scala API form of TrialsScaffolding so that some extensions can be written, something like:

@TestFactory // JUnit5 dynamic test annotation.
def parameterisedTest: util.Iterator[DynamicTest] =
    <trials instance>.withLimit(10).dynamicTests(<parameterised test lambda>)

[sageserpent-open]

A nice twist on this if the implementation goes smoothly would be to generalise SupplyToSyntax by extracting a parent interface, TrialsFactoring.TestIntegrationContexts - thus allowing both Scala and Java code to enjoy the tighter coupling between the actual trials instance(s) and the test's arguments.

The only reason for the current approach is because JUnit5's @TestTemplate is geared up to generate test cases outside of the actual test running, and has to use annotations to do this, so one has to live with what Java annotations can support - stringly-typed code.

[sageserpent-open]

This went out in release 1.15.1, Git commit SHA: 907ad0d .

[sageserpent-open]

Should add some overloads in the Java and Scala APIs to allow multi-argument parameterised test to be used against a supplier built from several ganged trials.

[sageserpent-open]

Also need to update the GitHub Wiki to advertise this.

[sageserpent-open]

Support for multi-argument tests went out in release 1.15.2, Git SHA: 9d53fa8 .