Exception thrown inside a test crashes the runner

[mbovel]

Hi!

Running the following code (munit.scala):

//> using lib "junit:junit:4.11"

import org.junit.runner.notification.RunListener
import org.junit.runner.{Description, JUnitCore}

class FatalErrorSuite extends munit.FunSuite:
  test("simple test") { () }
  test("failed test") {
    def loop(x: Int): Int = loop(x) + loop(x)
    loop(0)
  }
  test("another test") { () }

@main def main = 
  val junitCore = JUnitCore()
  val pointsCounter = PointsCounter()
  junitCore.addListener(pointsCounter)
  val result = junitCore.run(classOf[FatalErrorSuite])
  println(pointsCounter.runCount)

class PointsCounter(var runCount: Int = 0) extends RunListener:
  override def testStarted(description: Description): Unit =
    runCount += 1

Prints 3 as I would expect with munit 0.7:

➜ scala-cli run -d "org.scalameta::munit::0.7.27" munit.scala  
Compiling project (Scala 3.1.3, JVM)
Compiled project (Scala 3.1.3, JVM)
3

But prints 2 with munit 1.0.0-M1:

➜ scala-cli run -d "org.scalameta::munit::1.0.0-M1" munit.scala
Compiling project (Scala 3.1.3, JVM)
Compiled project (Scala 3.1.3, JVM)
2

And prints 2 with munit 1.0.0-M2 and stays stuck:

➜ scala-cli run -d "org.scalameta::munit::1.0.0-M2" munit.scala
Compiling project (Scala 3.1.3, JVM)
Compiled project (Scala 3.1.3, JVM)
2
^C

Seems like a bug; is it?

[tgodzik]

Thanks for reporting! Looks like something we should take a look into. @olafurpg do you have any idea what is going on here?

It would be good to finally release 1.0.0, so this would need to be fixed for that.

[olafurpg]

Thank you for the detailed reproduction! If I have to guess at a quick glance, the regression appears to have happened when adding support for async fixtures. A good place to look into fixing this issue would be this line here where we catch NonFatal exceptions

munit/munit/shared/src/main/scala/munit/MUnitRunner.scala

Line 291 in 92710a5

case NonFatal(ex) =>

We should be able to reproduce this situation by capturing println statements inside a "framework suite" similar to this one here https://github.com/scalameta/munit/blob/92710a507339d20368d251feadf66e4f9f4e1840/tests/shared/src/main/scala/munit/FailFrameworkSuite.scala