Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MUnit only reporting a handful of test results #138

Closed
sloshy opened this issue May 30, 2020 · 8 comments
Closed

MUnit only reporting a handful of test results #138

sloshy opened this issue May 30, 2020 · 8 comments

Comments

@sloshy
Copy link

sloshy commented May 30, 2020 •
edited
Loading

I have converted my test suites from ScalaTest to MUnit, and for some reason MUnit only appears to be reporting a subset of my expected test results. At first I only had a single test suite, in a large file of several hundred lines, and it was reporting only 3 results. I have since moved my tests into several, smaller suites, and now it only reports results from the very last suite.

Here is some sample output, from SBT:

sbt:mymedialog-mediadb> test
[info] Compiling 7 Scala sources to /home/ryan/Projects/mymedialog-mediadb/target/scala-2.13/test-classes ...
io.medialog.mediadb.NewEditionRouteSuite:
io.medialog.mediadb.NameRouteSuite:
io.medialog.mediadb.RenameRouteSuite:
io.medialog.mediadb.ItemRouteSuite:
io.medialog.mediadb.ModifyLocalesRouteSuite:
  + POST -> /[type]/[id]/names/modify-locales should fail on an invalid type string 0.897s
  + POST -> /[type]/[id]/editions/new should fail on an invalid type string 0.914s
  + POST -> /[type]/[id]/names/rename should fail on an invalid type string 0.917s
  + GET -> /[type]/[id] should fail on an invalid type string 0.919s
  + POST -> /[type]/[id]/names should fail on an invalid type string 0.925s
  + POST -> /new should add unique, new items only 0.366s
[info] Passed: Total 6, Failed 0, Errors 0, Passed 6
[success] Total time: 9 s, completed May 30, 2020, 4:31:30 PM

As you can see, the first 4 suites look almost as if they are being skipped over, but the final suite is the only one that appears to get ran at all.

The tests for all of the above suites have the following in common:

  • All use the same base suite, a combination of ScalaCheckSuite with some custom configuration and common functions (some scalacheck generators, HTTP4S DSLs and implicits, mainly)
  • All tests are ScalaCheck property tests using the property function
  • All tests are 100% synchronous

The code in question is on a private repository. If it helps to resolve the issue, I can grant temporary access to the repository to a project maintainer for MUnit (it's a personal project anyway). I can also share more specific code samples as needed, but the tests are pretty deeply tied into my domain logic so I can only share so much at a time.

EDIT: Some additional details:

  • Scala 2.13.2
  • VS Code & Metals w/ Bloop on WSL Ubuntu 20.04
  • MUnit 0.7.7
  • GraalVM 20.1.0 - JDK 11
@sloshy
Copy link
Author

sloshy commented May 30, 2020

Some additional information: I get different results when running my suites in SBT and Bloop. In Bloop, it reports from at least one test in each suite, but skips many of the others, and in SBT it only reports the last several.

➜  mymedialog-mediadb git:(master) bloop test root
io.medialog.mediadb.ItemRouteSuite:
  + GET -> /[type]/[id] should fail on an invalid type string 0.913s
  + POST -> /new should add unique, new items only 0.204s
Execution took 1.12s
io.medialog.mediadb.ModifyLocalesRouteSuite:
2 tests, 2 passed
All tests in io.medialog.mediadb.ItemRouteSuite passed

  + POST -> /[type]/[id]/names/modify-locales should fail on an invalid type string 0.031s
Execution took 31ms
1 tests, 1 passed
All tests in io.medialog.mediadb.ModifyLocalesRouteSuite passed

io.medialog.mediadb.NameRouteSuite:
  + POST -> /[type]/[id]/names should fail on an invalid type string 0.013s
Execution took 13ms
1 tests, 1 passed
All tests in io.medialog.mediadb.NameRouteSuite passed

io.medialog.mediadb.NewEditionRouteSpec:
  + POST -> /[type]/[id]/editions/new should fail on an invalid type string 0.011s
Execution took 11ms
1 tests, 1 passed
All tests in io.medialog.mediadb.NewEditionRouteSpec passed

io.medialog.mediadb.RenameRouteSuite:
  + POST -> /[type]/[id]/names/rename should fail on an invalid type string 0.011s
Execution took 11ms
1 tests, 1 passed
All tests in io.medialog.mediadb.RenameRouteSuite passed

===============================================
Total duration: 1.18s
All 5 test suites passed.
===============================================

@olafurpg
Copy link
Member

Thank you for reporting!

Do you get different output if you enable/disable forking?

Do the tests have conflicting names?

Have you tried to minimize the issue into something you can share publicly?

@olafurpg
Copy link
Member

What is the output if you run the tests with testOnly -- +l --verbose?

@sloshy
Copy link
Author

sloshy commented May 30, 2020

If I enable Test / fork := true then SBT reports similar results to Bloop. That second suggestion though is very interesting. It seems like all of the tests are being ran this way:

sbt:mymedialog-mediadb> testOnly -- +1 --verbose
io.medialog.mediadb.ModifyLocalesRouteSuite started
io.medialog.mediadb.ModifyLocalesRouteSuite:
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
  + POST -> /[type]/[id]/names/modify-locales should fail on an invalid type string 0.931s
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.ModifyLocalesRouteSuite.POST -> /[type]/[id] started
Test run io.medialog.mediadb.ModifyLocalesRouteSuite finished: 0 failed, 0 ignored, 7 total, 3.19s
io.medialog.mediadb.ItemRouteSuite started
io.medialog.mediadb.ItemRouteSuite:
io.medialog.mediadb.ItemRouteSuite.GET -> /[type]/[id] started
  + GET -> /[type]/[id] should fail on an invalid type string 0.013s
io.medialog.mediadb.ItemRouteSuite.GET -> /[type]/[id] started
io.medialog.mediadb.ItemRouteSuite.GET -> /[type]/[id] started
io.medialog.mediadb.ItemRouteSuite.POST -> /new should add unique, new items only started
  + POST -> /new should add unique, new items only 0.067s
Test run io.medialog.mediadb.ItemRouteSuite finished: 0 failed, 0 ignored, 4 total, 0.144s
io.medialog.mediadb.NewEditionRouteSuite started
io.medialog.mediadb.NewEditionRouteSuite:
io.medialog.mediadb.NewEditionRouteSuite.POST -> /[type]/[id] started
  + POST -> /[type]/[id]/editions/new should fail on an invalid type string 0.018s
io.medialog.mediadb.NewEditionRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.NewEditionRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.NewEditionRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.NewEditionRouteSuite.POST -> /[type]/[id] started
Test run io.medialog.mediadb.NewEditionRouteSuite finished: 0 failed, 0 ignored, 5 total, 3.833s
io.medialog.mediadb.NameRouteSuite started
io.medialog.mediadb.NameRouteSuite:
io.medialog.mediadb.NameRouteSuite.POST -> /[type]/[id] started
  + POST -> /[type]/[id]/names should fail on an invalid type string 0.011s
io.medialog.mediadb.NameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.NameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.NameRouteSuite.POST -> /[type]/[id] started
Test run io.medialog.mediadb.NameRouteSuite finished: 0 failed, 0 ignored, 4 total, 0.332s
io.medialog.mediadb.RenameRouteSuite started
io.medialog.mediadb.RenameRouteSuite:
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
  + POST -> /[type]/[id]/names/rename should fail on an invalid type string 0.009s
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
io.medialog.mediadb.RenameRouteSuite.POST -> /[type]/[id] started
Test run io.medialog.mediadb.RenameRouteSuite finished: 0 failed, 0 ignored, 6 total, 0.684s
[info] Passed: Total 6, Failed 0, Errors 0, Passed 6
[success] Total time: 10 s, completed May 30, 2020, 6:34:19 PM

It almost looks like it might have something to do with the names of the tests. In the --verbose logs it seems like it only gets the first several characters of each test suite.

@sloshy
Copy link
Author

sloshy commented May 31, 2020 •
edited
Loading

Ok, new update: I prefixed all my tests with the # they are in the current file (first test is prefixed with 1, second with 2, etc) and it correctly reports the test results now for all of my tests, with and without forking. I believe this has to be naming-related, then.

Full output:

➜  mymedialog-mediadb git:(master) btr
io.medialog.mediadb.ItemRouteSuite:
  + 1 GET -> /[type]/[id] should fail on an invalid type string 0.605s
  + 2 GET -> /[type]/[id] should 404 on a nonexistent item 0.13s
  + 3 GET -> /[type]/[id] should return valid items as JSON 0.218s
  + 4 POST -> /new should add unique, new items only 0.128s
Execution took 1.08s
4 tests, 4 passed
All tests in io.medialog.mediadb.ItemRouteSuite passed

io.medialog.mediadb.ModifyLocalesRouteSuite:
  + 1 POST -> /[type]/[id]/names/modify-locales should fail on an invalid type string 0.022s
  + 2 POST -> /[type]/[id]/names/modify-locales should 404 on a nonexistent item 0.239s
  + 3 POST -> /[type]/[id]/names/modify-locales should fail when the base name does not exist 0.162s
  + 4 POST -> /[type]/[id]/names/modify-locales should fail if all locales being added already exist 0.119s
  + 5 POST -> /[type]/[id]/names/modify-locales should add locales to name that do not already exist 0.261s
  + 6 POST -> /[type]/[id]/names/modify-locales should fail if attempting to remove every locale for a name 0.077s
  + 7 POST -> /[type]/[id]/names/modify-locales should remove known locales for a name 0.119s
Execution took 1s
7 tests, 7 passed
All tests in io.medialog.mediadb.ModifyLocalesRouteSuite passed

io.medialog.mediadb.NameRouteSuite:
  + 1 POST -> /[type]/[id]/names should fail on an invalid type string 0.007s
  + 2 POST -> /[type]/[id]/names should allow adding a new name to an existing item 0.094s
  + 3 POST -> /[type]/[id]/names should not allow duplicate names 0.063s
  + 4 POST -> /[type]/[id]/names should 404 on a nonexistent item 0.056s
Execution took 0.22s
4 tests, 4 passed
All tests in io.medialog.mediadb.NameRouteSuite passed

io.medialog.mediadb.NewEditionRouteSuite:
  + 1 POST -> /[type]/[id]/editions/new should fail on an invalid type string 0.007s
  + 2 POST -> /[type]/[id]/editions/new should 404 on a nonexistent item 0.035s
  + 3 POST -> /[type]/[id]/editions/new should not allow adding the default edition 0.019s
  + 4 POST -> /[type]/[id]/editions/new should not allow adding editions that are already added 0.03s
  + 5 POST -> /[type]/[id]/editions/new should allow adding valid editions to items 1.785s
Execution took 1.88s
5 tests, 5 passed
All tests in io.medialog.mediadb.NewEditionRouteSuite passed

io.medialog.mediadb.RenameRouteSuite:
  + 1 POST -> /[type]/[id]/names/rename should fail on an invalid type string 0.014s
  + 2 POST -> /[type]/[id]/names/rename should 404 on a nonexistent item 0.161s
  + 3 POST -> /[type]/[id]/names/rename should fail if the item exists but name does not exist 0.081s
  + 4 POST -> /[type]/[id]/names/rename should fail if the rename would be idempotent 0.056s
  + 5 POST -> /[type]/[id]/names/rename should fail if the new name exists already 0.095s
  + 6 POST -> /[type]/[id]/names/rename should rename existing names to a new non-existing name 0.095s
Execution took 0.5s
6 tests, 6 passed
All tests in io.medialog.mediadb.RenameRouteSuite passed

===============================================
Total duration: 4.68s
All 5 test suites passed.
===============================================

To reproduce the issue it seems that if you have tests with names similar to mine (minus the number prefixes) it would happen again. Seems like it cuts out the test name after /[type]/[id]

@olafurpg
Copy link
Member

olafurpg commented Jun 1, 2020

Thanks for the details! I'm able to reproduce, opened #140 which fixes the issue.

@olafurpg
Copy link
Member

olafurpg commented Jun 1, 2020

Triggered a v0.7.8 release with the fix

@SethTisue
Copy link
Contributor

good one :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SethTisue @sloshy @olafurpg