Minor Aesthetic Improvement to judge Output

[CFiggers]

This is mostly just to share how I'm using judge, not to pull focus to aesthetics/bikeshedding when functionality is still your focus. :) Feel more than free to ignore.

Taking some inspiration from pyrmont/testament, I made some minor aesthetic updates to my local fork of judge.

Original:

$ judge
running test: imported-function
1 passed 0 failed 0 skipped 0 unreachable

Updated:

$ judge
running test: imported-function

-------------------------------------------
 1 passed 0 failed 0 skipped 0 unreachable 
-------------------------------------------

The effect of this is minor when there's only a few tests being run, but the more get added (somewhere around 10 or so) the more this helps readability (IMO).

To make this change I updated just a couple lines of runner.janet:

# runner.janet starting at line 234

-  (eprintf "%i passed %i failed %i skipped %i unreachable"
-    (ctx :passed) (ctx :failed) (ctx :skipped) unreachable)

+  (eprintf "")
+  (let [msg (string " "
+                    (ctx :passed) " passed "
+                    (ctx :failed) " failed "
+                    (ctx :skipped) " skipped "
+                    unreachable " unreachable ")]
+    (eprintf (string/repeat "-" (length msg)))
+    (eprintf msg)
+    (eprintf (string/repeat "-" (length msg))))
+  (eprintf "")

You're more than welcome to borrow, modify, improve upon, or disregard entirely however you may see fit.

[ianthehenry]

Yeah, the test output is pretty ugly right now and I've been meaning to overhaul it. I mostly use Judge from my text editor and I diff the source file against the .tested file instead of looking at the output, so I don't have very good intuitions for how the output should look on the command line.

Since you are probably the only person using Judge right now, I'm curious if you have thoughts about:

• Distinguishing "there was no expected value but I filled one in" from "there was an expected value but it was wrong"
• Not printing "running test..." for every single test, only printing failing tests and the summary line
• Interactively prompting about patching source files (instead of the binary --accept option)
• Grouping output by file in some way

Do you use the source-patching capabilities at all? I know it's a weird workflow and I could imagine just using Judge as a "regular" test runner.

[CFiggers]

Good questions. In order:

• I don't find it confusing that "there was no expected value" and "the expected value was wrong" aren't distinguished. The colorized diffs make sense to me, especially because I know when I haven't provided an expected value. Then again, it might be pretty easy in that case to have the first line print in blue instead of red.
• I personally would probably like to see "running tests in [file name]" and then have only failed tests print with a little bit of white space at the left side to "indent" them under the file they're found in. Maybe leave in a "-v, --verbose" flag to still print "running test: [name]" every time.
• An interactive mode would be valuable to me. I end up approximating that anyway by running each test by name and then running the same command a second time with "-a" appended. An interactive mode would basically exactly automate the workflow I'm doing manually.
• See answer two—yes, grouping by file is probably the default that I would go with.

So, rolling that all together, imagine something like this:

$ judge

 running tests in: src/joule.janet
 running tests in: test/test1.janet
 running tests in: test/test2.janet

   test failed: find modal works while scanning previous find results

     - (test [foo])
     + (test [foo] [bar])

 running tests in: test/test3.janet

----------------------------------------------------
 Found 16 tests in 4 files containing 21 assertions
 15 passed 1 failed 0 skipped 0 unreachable 
----------------------------------------------------

As to your last question, yes absolutely. I use -a quite a lot, especially when first authoring the test. My default is to just leave the second argument to (test) blank and let judge populate it for me. At that point I'll examine it to see if it's what I expect it to be, and if not go fix it and if so immediately re-run with -a.

[CFiggers]

For context, I'm using judge primarily with VS Code from the integrated terminal. It looks like this:

[ianthehenry]

Something I was thinking could be nice is to print the entire source of a failing test in a kind of diff format:

$ judge
# testing src/file.janet

 (deftest "some test name"
   (def something "hi")
-  (test something))
+  (test something "hi"))

 Verdict [ynqa?]

Instead of just the test name. And then optionally prompt for an action in interactive mode.

(Maybe with dots to indicate successful tests? Maybe useful if tests are slow?)

Hmm... I might get some time to play with this this weekend.

[CFiggers]

Perhaps judge --trial enters the interactive mode... 😉

[ianthehenry]

upvote

[ianthehenry]

@CFiggers Okay! I've added an interactive mode to Judge, and reworked the output substantially. I think it's a big improvement but I'm curious to get your thoughts.

[CFiggers]

judge --interactive sparks joy 😁