Libtest json output #46450

Gilnaa · 2017-12-02T15:51:53Z

A revisit to my last PR.

Events are now more atomic, printed in a flat hierarchy.

For the normal test output:

running 1 test
test f ... FAILED

failures:

---- f stdout ----
	thread 'f' panicked at 'assertion failed: `(left == right)`
  left: `3`,
 right: `4`', f.rs:3:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    f

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

The JSON equivalent is:

{ "type": "suite", "event": "started", "test_count": "1" }
{ "type": "test", "event": "started", "name": "f" }
{ "type": "test", "event": "failed", "name": "f" }
{ "type": "suite", "event": "failed", "passed": 0, "failed": 1, "allowed_fail": 0, "ignored": 0,  "measured": 0, "filtered_out": "0" }
{ "type": "test_output", "name": "f", "output": "thread 'f' panicked at 'assertion failed: `(left == right)`
  left: `3`,
 right: `4`', f.rs:3:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.
" }

rust-highfive · 2017-12-02T15:52:06Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @TimNN (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

killercup · 2017-12-03T15:03:35Z

cc @rust-lang/dev-tools -- we talked a bit about this two weeks ago. One of the things we kinda settled on was that we want the test system to be extensible, so people can experiment with alternative test runners, etc.

This does look like a nice addition to the default testing tool, though. I think @matklad might be one of the first production user (of anything that allows easily parsing the test results in IntelliJ), so I'm especially interested in whether this is something he can use.

matklad · 2017-12-03T15:20:30Z

This does look like a nice addition to the default testing tool, though.

+:100:, having a machine readable output out of the box is hugely important for tool interoperability, and implementing it is much easier then generic "custom test runners" thing.

The main comment I have is that we absolutely need the docs which describe, in detail, the machine readable format. We probably also want to put this format through the RFC process?

Regarding the implementation, I think we should replace newlines in strings with \n, so that each JSON message always occupies a single line. That is, the following message would be hard to deal with in online fashion:

{ "type": "test_output", "name": "f", "output": "thread 'f' panicked at 'assertion failed: `(left == right)`
  left: `3`,
 right: `4`', f.rs:3:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.
" }

It's also not clear what happens with stdout and stderr of the tests, but I've kinda convinced myself that the problem of associating output with a particular test is in general unsolvable, so I am fine if the output is just interleaved with JSON messages.

killercup · 2017-12-03T15:46:36Z

We probably also want to put this format through the RFC process?

Yeah, probably. I'm kinda hoping to have this be a limited/focussed RFC that skips a whole bunch of edge cases (i.e., literally only do what this PR does), or land this as an unstable feature. Otherwise, we're probably opening the door to discuss any and all features we could ever want in this format…

I think we should replace newlines in strings with \n

Oh yes, I totally forgot to comment on this. JSON does not allow lines breaks in strings, they have to be escaped.

Gilnaa · 2017-12-04T11:12:13Z

They're all great ideas, I'll try to escape \n today and I'll try to work on the RFC by the end of the weekend

killercup · 2017-12-04T18:33:12Z

I think it'd be a good idea to use an actual JSON library to write these. The compiler has its own rustc_serialize, but this format looks like a very good fit for serde's tagged enum types. Not how easy/desirable it is to depend on serde{_json, _derive,} here, but it's already used in the bootstrap scripts and cargo.

Gilnaa · 2017-12-04T19:04:01Z

TBH, I tried adding serde as a dep of libtest and had some obscure errors. (didn't investigate)

It seemed simple enough to just roll-my-own, but I agree it's probably better to use it now that I've realized escaping is not that simple.

Gilnaa · 2017-12-04T19:16:56Z

RFC here:
rust-lang/rfcs#2234

shepmaster · 2017-12-09T15:48:12Z

This PR appears to be blocked on rust-lang/rfcs#2234

Gilnaa · 2017-12-10T07:56:08Z

I see now that there's a desire to add multiple new formats (TAP, etc.).

I can split this PR to merge only the refactoring needed for multiple output-formats, and have JSON as a separate PR.

bors · 2017-12-10T19:07:18Z

☔ The latest upstream changes (presumably #46620) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · 2017-12-13T02:34:32Z

We discussed this at the dev-tools meeting yesterday. We concluded that we don't need to wait for an RFC and we should land this now as a way to experiment and iterate on machine-readable test output. Everything should be behind an unstable feature flag.

The long-term solution is going to be deeply intertwined with pluggable test runners.

bjorn3 · 2017-12-11T07:32:59Z

src/libtest/formatters.rs

+
+        Ok(())
+    }
+}


Missing trailing newline

Gilnaa · 2017-12-13T15:12:24Z

I've made a few fixes suggested in the RFC.

The only issue left is how to mark it as unstable/nightly only.
The attribute is unavailable in this context, and even if it were, I don't think it would be relevant for CLI options.

@alexcrichton , maybe you have an idea?

Edit:
Actually, now that I think about it, libsyntax generates the calls to libtest, maybe it's possible for it to tell it if the compiler is nightly or stable

alexcrichton · 2017-12-13T17:19:28Z

Unfortunately we're not really set up right now to have unstable flags in libtest, so the infrastructure would have to be added here as well.

In general though the way our unstable flags look like is:

To enable an unstable flag like this you also need to explicitly enable unstsable flags. Typically this means you pass something like -Z unstable-options.
That means we'd need to add a flag to libtest like -Z which takes an argument like unstable-options
The -Z flag can then only be passed on the nightly channel. If it's passed on beta/stable it's a hard error to pass the -Z flag.

Gilnaa · 2017-12-13T18:14:23Z

Cool, and how should libtest obtain the stability/instability status?

alexcrichton · 2017-12-13T18:36:59Z

This is how it's done in libsyntax, and unfortunately that probably would just need to be duplicated for libtest.

Gilnaa · 2017-12-14T10:04:12Z

r? @TimNN (am I doing this right?)

TimNN · 2017-12-14T10:31:00Z

src/libtest/lib.rs

@@ -519,7 +530,13 @@ pub fn parse_opts(args: &[String]) -> Option<OptRes> {
        None if quiet => OutputFormat::Terse,
        Some("pretty") | None => OutputFormat::Pretty,
        Some("terse") => OutputFormat::Terse,
-        Some("json") => OutputFormat::Json,
+        Some("json") => {
+            if !allow_unstable {


This is not quite what @alexcrichton mentioned in #46450 (comment): There should be a -Z unstable-options flag, which may only be passed on nightly (the allow_unstable() function above), and only if -Z unstable-options is passed should json output be allowed (in other words: the user needs to explicitly enable unstable flags, before being allowed to pass the json flag).

Ohh, got it.

Gilnaa · 2017-12-15T05:44:28Z

json is behind -Z now

Gilnaa · 2018-01-26T19:35:46Z

@kennytm All problems should be solved™

kennytm · 2018-01-26T19:54:39Z

@bors r=nrc

bors · 2018-01-26T19:54:40Z

📌 Commit 100ead3 has been approved by nrc

bors · 2018-01-26T20:34:49Z

⌛ Testing commit 100ead3 with merge 83389e2...

Libtest json output A revisit to my [last PR](#45923). Events are now more atomic, printed in a flat hierarchy. For the normal test output: ``` running 1 test test f ... FAILED failures: ---- f stdout ---- thread 'f' panicked at 'assertion failed: `(left == right)` left: `3`, right: `4`', f.rs:3:1 note: Run with `RUST_BACKTRACE=1` for a backtrace. failures: f test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out ``` The JSON equivalent is: ``` { "type": "suite", "event": "started", "test_count": "1" } { "type": "test", "event": "started", "name": "f" } { "type": "test", "event": "failed", "name": "f" } { "type": "suite", "event": "failed", "passed": 0, "failed": 1, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": "0" } { "type": "test_output", "name": "f", "output": "thread 'f' panicked at 'assertion failed: `(left == right)` left: `3`, right: `4`', f.rs:3:1 note: Run with `RUST_BACKTRACE=1` for a backtrace. " } ```

bors · 2018-01-26T22:40:07Z

💔 Test failed - status-appveyor

kennytm · 2018-01-27T07:49:06Z

src/test/run-make/libtest-json/Makefile

+	$(RUSTC) --test f.rs
+	$(call RUN,f) -Z unstable-options --test-threads=1 --format=json > $(OUTPUT_FILE) || true
+
+	cat $(OUTPUT_FILE) | $(PYTHON) validate_json.py


Please quote the environment variable like "$(PYTHON)". On Windows the path is C:\Python27\python.exe, and shell strips the backslashes without the quotes.

[02:04:23] ---- [run-make] run-make\libtest-json stdout ---- [02:04:23] [02:04:23] error: make failed [02:04:23] status: exit code: 2 [02:04:23] command: "make" [02:04:23] stdout: [02:04:23] ------------------------------------------ [02:04:23] PATH="/c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu:C:\projects\rust\build\x86_64-pc-windows-gnu\stage2\bin:/c/projects/rust/build/x86_64-pc-windows-gnu/stage0-tools/x86_64-pc-windows-gnu/release/deps:/c/projects/rust/build/x86_64-pc-windows-gnu/stage0-sysroot/lib/rustlib/x86_64-pc-windows-gnu/lib:/c/Program Files (x86)/Inno Setup 5:/c/Python27:/c/projects/rust/mingw64/bin:/usr/bin:/c/Perl/site/bin:/c/Perl/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Program Files/7-Zip:/c/Program Files/Microsoft/Web Platform Installer:/c/Tools/GitVersion:/c/Tools/PsTools:/c/Program Files/Git LFS:/c/Program Files (x86)/Subversion/bin:/c/Program Files/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/110/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn/ManagementStudio:/c/Tools/WebDriver:/c/Program Files (x86)/Microsoft SDKs/TypeScript/1.4:/c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/IDE/PrivateAssemblies:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI/wbin:/c/Ruby193/bin:/c/Tools/NUnit/bin:/c/Tools/xUnit:/c/Tools/MSpec:/c/Tools/Coverity/bin:/c/Program Files (x86)/CMake/bin:/c/go/bin:/c/Program Files/Java/jdk1.8.0/bin:/c/Python27:/c/Program Files/nodejs:/c/Program Files (x86)/iojs:/c/Program Files/iojs:/c/Users/appveyor/AppData/Roaming/npm:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/MSBuild/14.0/Bin:/c/Tools/NuGet:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/CommonExtensions/Microsoft/TestWindow:/c/Program Files/Microsoft DNX/Dnvm:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Apache/Maven/bin:/c/Python27/Scripts:/c/Tools/NUnit3:/c/Program Files/Mercurial:/c/Program Files/LLVM/bin:/c/Program Files/dotnet:/c/Tools/curl/bin:/c/Program Files/Amazon/AWSCLI:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/Extensions/Microsoft/SQLDB/DAC/140:/c/Program Files/Git/cmd:/c/Program Files/Git/usr/bin:/c/Tools/vcpkg:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/Yarn/bin:/c/Program Files (x86)/Microsoft SQL Server/140/Tools/Binn:/c/Program Files/Microsoft SQL Server/140/Tools/Binn:/c/Program Files/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/nodejs:/c/ProgramData/chocolatey/bin:/c/Program Files/PowerShell/6.0.0:/c/Program Files/erl9.2/bin:/c/Users/appveyor/AppData/Local/Yarn/bin:/c/Users/appveyor/AppData/Roaming/npm:/c/Program Files/AppVeyor/BuildAgent:/c/projects/rust:/c/projects/rust/handle" 'C:\projects\rust\build\x86_64-pc-windows-gnu\stage2\bin\rustc.exe' --out-dir /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu -L /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu --test f.rs [02:04:23] PATH="/c/projects/rust/build/x86_64-pc-windows-gnu/stage0-tools/x86_64-pc-windows-gnu/release/deps:/c/projects/rust/build/x86_64-pc-windows-gnu/stage0-sysroot/lib/rustlib/x86_64-pc-windows-gnu/lib:/c/Program Files (x86)/Inno Setup 5:/c/Python27:/c/projects/rust/mingw64/bin:/usr/bin:/c/Perl/site/bin:/c/Perl/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Program Files/7-Zip:/c/Program Files/Microsoft/Web Platform Installer:/c/Tools/GitVersion:/c/Tools/PsTools:/c/Program Files/Git LFS:/c/Program Files (x86)/Subversion/bin:/c/Program Files/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/110/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn:/c/Program Files/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn/ManagementStudio:/c/Tools/WebDriver:/c/Program Files (x86)/Microsoft SDKs/TypeScript/1.4:/c/Program Files (x86)/Microsoft Visual Studio 12.0/Common7/IDE/PrivateAssemblies:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI/wbin:/c/Ruby193/bin:/c/Tools/NUnit/bin:/c/Tools/xUnit:/c/Tools/MSpec:/c/Tools/Coverity/bin:/c/Program Files (x86)/CMake/bin:/c/go/bin:/c/Program Files/Java/jdk1.8.0/bin:/c/Python27:/c/Program Files/nodejs:/c/Program Files (x86)/iojs:/c/Program Files/iojs:/c/Users/appveyor/AppData/Roaming/npm:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/MSBuild/14.0/Bin:/c/Tools/NuGet:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/CommonExtensions/Microsoft/TestWindow:/c/Program Files/Microsoft DNX/Dnvm:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Apache/Maven/bin:/c/Python27/Scripts:/c/Tools/NUnit3:/c/Program Files/Mercurial:/c/Program Files/LLVM/bin:/c/Program Files/dotnet:/c/Tools/curl/bin:/c/Program Files/Amazon/AWSCLI:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft Visual Studio 14.0/Common7/IDE/Extensions/Microsoft/SQLDB/DAC/140:/c/Program Files/Git/cmd:/c/Program Files/Git/usr/bin:/c/Tools/vcpkg:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/Yarn/bin:/c/Program Files (x86)/Microsoft SQL Server/140/Tools/Binn:/c/Program Files/Microsoft SQL Server/140/Tools/Binn:/c/Program Files/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/nodejs:/c/ProgramData/chocolatey/bin:/c/Program Files/PowerShell/6.0.0:/c/Program Files/erl9.2/bin:/c/Users/appveyor/AppData/Local/Yarn/bin:/c/Users/appveyor/AppData/Roaming/npm:/c/Program Files/AppVeyor/BuildAgent:/c/projects/rust:/c/projects/rust/handle:C:\projects\rust\build\x86_64-pc-windows-gnu\stage2\lib\rustlib\x86_64-pc-windows-gnu\lib" /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu/f -Z unstable-options --test-threads=1 --format=json > /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu/libtest-json-output.json || true [02:04:23] cat /c/projects/rust/build/x86_64-pc-windows-gnu/test/run-make/libtest-json.stage2-x86_64-pc-windows-gnu/libtest-json-output.json | C:\Python27\python.exe validate_json.py [02:04:23] [02:04:23] ------------------------------------------ [02:04:23] stderr: [02:04:23] ------------------------------------------ [02:04:23] /bin/sh: C:Python27python.exe: command not found [02:04:23] make: *** [Makefile:10: all] Error 127 [02:04:23] [02:04:23] ------------------------------------------ [02:04:23] [02:04:23] thread '[run-make] run-make\libtest-json' panicked at 'explicit panic', tools\compiletest\src\runtest.rs:2883:9 [02:04:23] note: Run with `RUST_BACKTRACE=1` for a backtrace. [02:04:23] [02:04:23] [02:04:23] failures: [02:04:23] [run-make] run-make\libtest-json [02:04:23] [02:04:23] test result: FAILED. 172 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out [02:04:23] [02:04:23] thread 'main' panicked at 'Some tests failed', tools\compiletest\src\main.rs:476:22

Is it enough? Does it work in msys2 whatever? It seems like it wants "/c/python27/...."

@Gilnaa should be enough, msys2 can recognize Windows paths.

what a time to be alive

kennytm · 2018-01-27T09:53:38Z

@bors r=nrc

bors · 2018-01-27T09:53:39Z

📌 Commit 8b7f1d0 has been approved by nrc

bors · 2018-01-27T10:57:01Z

⌛ Testing commit 8b7f1d0 with merge 6b99ade...

Libtest json output A revisit to my [last PR](#45923). Events are now more atomic, printed in a flat hierarchy. For the normal test output: ``` running 1 test test f ... FAILED failures: ---- f stdout ---- thread 'f' panicked at 'assertion failed: `(left == right)` left: `3`, right: `4`', f.rs:3:1 note: Run with `RUST_BACKTRACE=1` for a backtrace. failures: f test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out ``` The JSON equivalent is: ``` { "type": "suite", "event": "started", "test_count": "1" } { "type": "test", "event": "started", "name": "f" } { "type": "test", "event": "failed", "name": "f" } { "type": "suite", "event": "failed", "passed": 0, "failed": 1, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": "0" } { "type": "test_output", "name": "f", "output": "thread 'f' panicked at 'assertion failed: `(left == right)` left: `3`, right: `4`', f.rs:3:1 note: Run with `RUST_BACKTRACE=1` for a backtrace. " } ```

bors · 2018-01-27T13:40:05Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nrc
Pushing 6b99ade to master...

farodin91 · 2018-02-11T21:09:26Z

Is there any support in cargo for this feature?

Gilnaa · 2018-02-12T04:45:17Z

@farodin91 on nightly, cargo test -- -Z unstable-options --format=json

Not sure if that is unstable-options or unstable-features

kornelski · 2018-02-13T07:52:59Z

Is there an active RFC or a tracking issue for this feature?

I know there's been discussion about overhauling testing in general, but I'm curious when/if this particular feature may become stable.

rust-highfive assigned TimNN Dec 2, 2017

kennytm added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 3, 2017

GuillaumeGomez added the T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. label Dec 4, 2017

Gilnaa mentioned this pull request Dec 4, 2017

Added libtest_json_output rust-lang/rfcs#2234

Closed

shepmaster added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 9, 2017

Gilnaa force-pushed the libtest_json_output branch from dcd49b9 to 266f26d Compare December 12, 2017 20:10

nrc added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Dec 13, 2017

bjorn3 requested changes Dec 13, 2017

View reviewed changes

src/libtest/formatters.rs Outdated

Ok(())

}

}

Copy link

Member

bjorn3 Dec 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing trailing newline

Gilnaa force-pushed the libtest_json_output branch from 93e347a to debe263 Compare December 13, 2017 19:15

TimNN reviewed Dec 14, 2017

View reviewed changes

Gilad Naaman added 2 commits January 26, 2018 19:46

libtest: Failing benchmarks no longer crash the harness.

cb86f38

Updated Cargo commit hash

100ead3

Gilnaa force-pushed the libtest_json_output branch from ab10869 to 100ead3 Compare January 26, 2018 17:57

kennytm reviewed Jan 27, 2018

View reviewed changes

libtest: Fixed call to python in run-make

8b7f1d0

Gilnaa force-pushed the libtest_json_output branch from 3804e0a to 8b7f1d0 Compare January 27, 2018 09:50

kennytm added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 27, 2018

bors merged commit 8b7f1d0 into rust-lang:master Jan 27, 2018

dwrensha mentioned this pull request Jan 28, 2018

TestOpts::quiet -> TestOpts.format Manishearth/compiletest-rs#99

Merged

Mark-Simulacrum mentioned this pull request Feb 12, 2018

Compiler slowdown on lage module in nightly #48153

Closed

Gilnaa mentioned this pull request Mar 25, 2018

Tracking issue for libtest JSON output #49359

Open

lzybkr mentioned this pull request Jun 29, 2018

cargo test -- --logfile should support common CI formats rust-lang/cargo#5670

Closed

zackmdavis mentioned this pull request Sep 2, 2018

Make json test output formatter represent "test_count" as num #53867

Merged

ehuss mentioned this pull request Apr 10, 2019

Summarize output result of all test binaries run rust-lang/cargo#4324

Open

varkor mentioned this pull request Jul 22, 2019

libtest: add --show-output flag to print stdout of successful tests #62600

Merged

Libtest json output #46450

Libtest json output #46450

Conversation

Gilnaa commented Dec 2, 2017 • edited Loading

rust-highfive commented Dec 2, 2017

killercup commented Dec 3, 2017

matklad commented Dec 3, 2017

killercup commented Dec 3, 2017 • edited Loading

Gilnaa commented Dec 4, 2017

killercup commented Dec 4, 2017 • edited Loading

Gilnaa commented Dec 4, 2017

Gilnaa commented Dec 4, 2017

shepmaster commented Dec 9, 2017

Gilnaa commented Dec 10, 2017

bors commented Dec 10, 2017

nrc commented Dec 13, 2017

bjorn3 Dec 11, 2017

Choose a reason for hiding this comment

Gilnaa commented Dec 13, 2017 • edited Loading

alexcrichton commented Dec 13, 2017

Gilnaa commented Dec 13, 2017

alexcrichton commented Dec 13, 2017

Gilnaa commented Dec 14, 2017

TimNN Dec 14, 2017

Choose a reason for hiding this comment

Gilnaa Dec 14, 2017

Choose a reason for hiding this comment

Gilnaa commented Dec 15, 2017

Gilnaa commented Jan 26, 2018

kennytm commented Jan 26, 2018

bors commented Jan 26, 2018

bors commented Jan 26, 2018

bors commented Jan 26, 2018

kennytm Jan 27, 2018

Choose a reason for hiding this comment

Gilnaa Jan 27, 2018

Choose a reason for hiding this comment

kennytm Jan 27, 2018

Choose a reason for hiding this comment

Gilnaa Jan 27, 2018

Choose a reason for hiding this comment

kennytm commented Jan 27, 2018

bors commented Jan 27, 2018

bors commented Jan 27, 2018

bors commented Jan 27, 2018

farodin91 commented Feb 11, 2018

Gilnaa commented Feb 12, 2018

kornelski commented Feb 13, 2018

Gilnaa commented Dec 2, 2017 •

edited

Loading

killercup commented Dec 3, 2017 •

edited

Loading

killercup commented Dec 4, 2017 •

edited

Loading

Gilnaa commented Dec 13, 2017 •

edited

Loading