RFC: Add information to passing tests #36879

mmiller-max · 2020-08-02T09:20:00Z

Summary

With this PR, the information (tested expression and the value) of a succfessul test will be stored in the Pass Result rather than these fields being set to nothing. This will be the case for both @test and @test_throws. There will be no change to DefaultTestSet behaviour, as the current behaviour of record(ts:DefaultTestSet, t::Pass) discards any information in the Pass result and just increments the number of successful tests. Storing the information allows other testsets to use this information, for example in test report generation.

More info

Following the discussion in the issue, it seems like the changes made by #20150 mean that information can be stored when tests Pass as the whole result is discarded by DefaultTestSets.

This allows other CustomTestSets to use this information, which is important if this information is required, for example by a test case management suite.

mmiller-max · 2020-08-03T03:13:36Z

To retain the same output for a Pass as before this PR, Base.show needed to be modified. I think the code that I removed from Base.show was never being used (as those fields were always nothing) but this needs confirming by someone with a wider knowledge than me.

This PR is ready for a code review.

mmiller-max · 2020-08-05T10:46:40Z

Please can someone review this? Cheers

StefanKarpinski · 2020-08-05T16:03:26Z

Giving a summary of the effect of this would be helpful. Who would be good to review? GitHub suggests @JeffBezanson, but that may just be because the did some sweeping change that touched this code.

timholy · 2020-08-05T22:30:47Z

Maybe @oxinabox who filed the issue.

Would also be great to see some numbers on whether/how this affects the time to run Julia's own tests.

mmiller-max · 2020-08-06T00:18:16Z

Thanks both for your comments.

I'll run a comparison on make test-all (if that's the best way to do it?) tonight (Australian time), and here's a brief summary of the effect:

With this PR, the information (tested expression and the value) of a succfessul test will be stored in the Pass Result rather than these fields being set to nothing. This will be the case for both @test and @test_throws. There will be no change to DefaultTestSet, as the current behaviour of record(ts:DefaultTestSet, t::Pass) discards any information in the Pass result and just increments the number of successful tests. Storing the information allows other testsets to use this information, for example in test report generation.

The storing of information in the Pass object means that Base.show(io::IO, t::Pass) has to be modified so that the output when a test passes is unchanged. The removed lines from this function print the Pass information if not nothing, which on master it always is and therefore the printing was never being done. I think removing these lines should have no effect.

mmiller-max · 2020-08-08T04:48:01Z

Julia test times averaged across two runs:

With this PR: 42.5 mins
On a master commit (7c0cb30): 44.5 mins

The difference could be just what else was running on my machine at the same time, it would be useful if someone else could repeat the test so the results don't just rely on one laptop.

I have been struggling to actually run Julia's tests as the generating precompile statements step hangs (>1 hr) whilst building, so have had to resort to running make test-all inside a container. This in turns means the sockets and file tests fail, but hopefully its still representative. The same failures occur with and without the changes. (Any help with this would be appreciated, but don't want to cloud up this PR).

oxinabox · 2020-08-11T22:07:10Z

Maybe @oxinabox who filed the issue.

Note that @mmiller-max is now working on the package that I was working on when I filed that issue.
(TestReports.jl)

I will review this PR presently

oxinabox

I think this needs tests that it actually is storing the correct thing.

oxinabox · 2020-08-11T22:09:32Z

stdlib/Test/src/Test.jl

-    elseif t.test_type === :test && t.data !== nothing
-        # The test was an expression, so display the term-by-term
-        # evaluated version as well
-        print(io, "\n   Evaluated: ", t.data)


Why don't we want to display this?
Given someone has created a test result object, why not show its fields?

I can see it being argued either way.

One the one hand the information is there so why not display it to give the user more information, but on the other hand if a test has passed the user may not want the details of the passing of that test to fill up the REPL. I typically only see the Test Passed output when manually stepping through tests to find the failing ones, and wouldn't be interested in knowing the details of the ones that passed. Normally the tests are in testsets and therefore the pass information isn't shown either way.

I kept it to the latter as that's what was originally happening before the changes here, albeit because that information wasn't stored so couldn't be displayed anyway.

I'm leaning towards not displaying it, but its definitely worth a discussion.

Fair enough. I am not sure when this is displayed, except if @test is used on its out, in the REPL.
do @testsets call display on the tests with in them? If so, i agree we want to minimize noise and not display this

I think it's only displayed when @test is used on its own in the REPL, as you say. @testsets do not call display so makes no difference for them.

I still think it is more helpful to not display this information, but actually thinking about it @test_throws already displays the exception type, so to be consistent we should either remove that printing too, or keep them all in.

Is this a question for slack/a wider group of people?

Current situation is:

julia> @test 1==1 Test Passed julia> @test_throws ErrorException throw(ErrorException("Msg")) Test Passed Thrown: ErrorException

Full printing:

julia> @test 1==1 Test Passed Expression: 1 == 1 Evaluated: 1 == 1 julia> @test_throws ErrorException throw(ErrorException("Msg")) Test Passed Expression: throw(ErrorException("Msg")) Thrown: ErrorException

Or no printing:

julia> @test 1==1 Test Passed julia> @test_throws ErrorException throw(ErrorException("Msg")) Test Passed

stdlib/Test/src/Test.jl

mmiller-max · 2020-10-18T22:02:47Z

Bump.

I think the following two points need addressing to move this PR forward:

Deciding on the test output, as per this discussion. Who needs to be involved with the decision?
Getting someone else to do a benchmark to check performance of Julia's tests.

Cheers

mmiller-max · 2021-01-30T05:38:58Z

Bump

Just wondering if this has been discussed on a triage call or if that's still pending?

vtjnash · 2021-04-22T19:37:30Z

This looks fine to me. We needed to make sure not to store the information (by default), but it didn't need to be removed this early. Please ping me when you rebase for review. For the questions:

Please remove the printing changes, as those aren't applicable to the rest of the PR and will let us merge this without issue.
If it passes on CI, that should be good enough for confirming performance is good.

mmiller-max · 2021-04-28T08:04:59Z

@vtjnash thanks for following up. Rebased and ready for another review. Without the changes for printing there are quite a few doctest changes due to Passes now containing information that can be printed.

vtjnash

lgtm

stdlib/Test/src/Test.jl

Closes JuliaLang#25483

This restores the behaviour of how `@test` shows a test pass in Julia v1.0 - v1.6, reverting the display changes that resulted when we began storing more info in the `Pass` type in JuliaLang#36879. Closes JuliaLang#43794.

This restores the behavior of how `at-test` shows a test pass in Julia v1.0 - v1.6, reverting the display changes introduced when storing more info in the `Pass` type in #36879, closes #43794.

This restores the behavior of how `at-test` shows a test pass in Julia v1.0 - v1.6, reverting the display changes introduced when storing more info in the `Pass` type in JuliaLang#36879, closes JuliaLang#43794.

mmiller-max changed the title ~~Add information to passing tests~~ WIP: Add information to passing tests Aug 2, 2020

mmiller-max force-pushed the mm/add-test-pass-information branch from 858c0fb to f54b2a4 Compare August 3, 2020 00:14

mmiller-max changed the title ~~WIP: Add information to passing tests~~ RFC: Add information to passing tests Aug 3, 2020

mmiller-max force-pushed the mm/add-test-pass-information branch from f54b2a4 to 116a4f0 Compare August 5, 2020 10:42

oxinabox reviewed Aug 11, 2020

View reviewed changes

mmiller-max force-pushed the mm/add-test-pass-information branch from 116a4f0 to 0492c37 Compare August 23, 2020 08:12

oxinabox mentioned this pull request Oct 9, 2020

WIP make metatesting passes to match fails JuliaDiff/ChainRulesTestUtils.jl#68

Open

mmiller-max mentioned this pull request Oct 14, 2020

V1.0.0 Roadmap JuliaTesting/TestReports.jl#48

Closed

17 tasks

StefanKarpinski added the triage This should be discussed on a triage call label Nov 16, 2020

vtjnash removed the triage This should be discussed on a triage call label Apr 22, 2021

mmiller-max force-pushed the mm/add-test-pass-information branch 2 times, most recently from 2380b12 to d2a3576 Compare April 27, 2021 22:50

Add information to passing tests (JuliaLang#25483)

a67d42e

mmiller-max force-pushed the mm/add-test-pass-information branch from d2a3576 to a67d42e Compare April 28, 2021 02:52

vtjnash approved these changes Apr 28, 2021

View reviewed changes

vtjnash added the forget me not PRs that one wants to make sure aren't forgotten label Apr 28, 2021

vtjnash merged commit 6b10500 into JuliaLang:master May 29, 2021

simeonschaub removed the forget me not PRs that one wants to make sure aren't forgotten label May 29, 2021

vtjnash reviewed Jun 3, 2021

View reviewed changes

stdlib/Test/src/Test.jl Show resolved Hide resolved

shirodkara pushed a commit to shirodkara/julia that referenced this pull request Jun 9, 2021

Test: Add information to passing tests (JuliaLang#36879)

def0433

Closes JuliaLang#25483

KristofferC mentioned this pull request Jun 30, 2021

make "Add information to passing tests" a bit more backwards compatible #41424

Merged

johanmon pushed a commit to johanmon/julia that referenced this pull request Jul 5, 2021

Test: Add information to passing tests (JuliaLang#36879)

32b9ea5

Closes JuliaLang#25483

This was referenced Jan 13, 2022

Only show "Test Passed" when a test passes, not what was evaluated #43794

Closed

Show only the "Test Passed" message when a test passes #43795

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Add information to passing tests #36879

RFC: Add information to passing tests #36879

mmiller-max commented Aug 2, 2020 •

edited

Loading

mmiller-max commented Aug 3, 2020

mmiller-max commented Aug 5, 2020

StefanKarpinski commented Aug 5, 2020

timholy commented Aug 5, 2020

mmiller-max commented Aug 6, 2020

mmiller-max commented Aug 8, 2020 •

edited

Loading

oxinabox commented Aug 11, 2020

oxinabox left a comment •

edited

Loading

oxinabox Aug 11, 2020

mmiller-max Aug 13, 2020

oxinabox Aug 13, 2020 •

edited

Loading

mmiller-max Aug 23, 2020

mmiller-max commented Oct 18, 2020

mmiller-max commented Jan 30, 2021

vtjnash commented Apr 22, 2021

mmiller-max commented Apr 28, 2021

vtjnash left a comment

RFC: Add information to passing tests #36879

RFC: Add information to passing tests #36879

Conversation

mmiller-max commented Aug 2, 2020 • edited Loading

Summary

More info

mmiller-max commented Aug 3, 2020

mmiller-max commented Aug 5, 2020

StefanKarpinski commented Aug 5, 2020

timholy commented Aug 5, 2020

mmiller-max commented Aug 6, 2020

mmiller-max commented Aug 8, 2020 • edited Loading

oxinabox commented Aug 11, 2020

oxinabox left a comment • edited Loading

Choose a reason for hiding this comment

oxinabox Aug 11, 2020

Choose a reason for hiding this comment

mmiller-max Aug 13, 2020

Choose a reason for hiding this comment

oxinabox Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

mmiller-max Aug 23, 2020

Choose a reason for hiding this comment

mmiller-max commented Oct 18, 2020

mmiller-max commented Jan 30, 2021

vtjnash commented Apr 22, 2021

mmiller-max commented Apr 28, 2021

vtjnash left a comment

Choose a reason for hiding this comment

mmiller-max commented Aug 2, 2020 •

edited

Loading

mmiller-max commented Aug 8, 2020 •

edited

Loading

oxinabox left a comment •

edited

Loading

oxinabox Aug 13, 2020 •

edited

Loading