Turn reporters into seperate module #492
Comments
|
+1 from me but I'd like to retain the convenience that mocha already provides for backwards compat |
|
Yes |
|
Meaning mocha-reporters would be a dependency of mocha and used whenever spec is used. |
|
I would vote against TAP personally, and just use JSON, but either way this would be a nice feature because not only can we decouple but it makes outputting multiple reports (cov, markdown etc) in parallel really simple since you can just |
|
+1 for JSON format as it is easier to consume. |
|
+1 for JSON, if you want tap you can just do |
|
another nice thing with this is that we could easily combine output. Right now for example in Express I have two make targets, |
|
👍 for me, running dozens of tests in separate processes, would love consolidated reporting for them. |
|
I just wanted to come back to this. At the moment, our events aren't JSON serializable, I'd like to cut down what we report to be JSON serializable, that way I wont need to use the insane ForbesLindesay/jssn I've created to serialize them. At the moment you need a serializer that can handle circular references and maintain the prototype chain. Getting that to work in IE6 was a P.I.T.A. |
|
@visionmedia you said your interested in this change. If I were to do I would be tempted to have the intermediate format be TAP. so you could do Obviously there would be a JS api and The only breaking change would be "how to write custom reporters" and maybe default mocha to TAP. Alternatively we can have a json like TAP interface on stdout & stdin. If I work on this i'll probably not do all reporters at the same time but do them a bunch at the time and get the ball rolling earlier. |
|
The problem with TAP is that it encodes a fraction of the information that mocha provides and that mocha-reporters consume. It wouldn't be that much of a stretch to envision a really nice format for structured test logs as newline delimited JSON. That would be much easier to extend with all the additional information. |
|
yeah that's the thing you'd have to augment TAP, making it super ugly and not very useful to TAP consumers that aren't familiar with the format. I'd love the change but I don't think we should use TAP. It would just end up being json-infused TAP haha |
|
You could support TAP by essentially doing TAP + JSON comments. I don't think there would be any real point though as the reporters wouldn't be TAP compatible |
|
plus TAP isn't really all that popular, node is about it, most CIs don't consume TAP, so we're not losing much, they all have their own crazy-ass weird format. It's too bad we don't have Test-Anything-JSON because then you could augment it at will without making the test labels super ugly |
|
If anyone has a suggestion for the new line delimited JSON format I"ll use that. Otherwise I'll use whatever mocha does. I'll probably get the reporters to understand TAP as well since that will make them work with tape for my use case |
|
Only a fraction of the reporters can be made to understand TAP in any meaningful way. As for a format, I started formalizing some thoughts in https://github.com/test-everything/spec/blob/master/stream.md but they're pretty rubbish at the moment. I think we should specify something based loosely on mocha, but mocha's internal model (i.e. the event emitter) not the current JSON reporter (which doesn't report nearly enough information). Things I'd like to see from a format:
I'd love to collaborate with people on designing such a standard. |
|
One of the things that I would like to see is the inclusion of meta data for any failed test, not just the failure message/stack. Chai's assertion-error constructor already includes a .toJSON() method that allows for any user/plugin provided data (such as Side Note: I have also started a discussion about what might be included in future assertion-error enhancements. |
|
So i've come around to the idea that TAP is a terrible format and you need some kind of JSON object format. 👍 |
|
The other thing I would add to my list of requirements for reporting is that success or failure should be detected/indicated purely through the use of the exit code. i.e. it should be possible to have a failing test, but overall state that tests pass, and visa versa. All CIs should depend purely on the exit code. |
TAP is a human-readable format, JSON isn't. It's a good reason to use TAP. |
|
it doesn't need to be readable, it just needs to be an intermediate data format, which JSON is very good at, no reason to use TAP for this |
|
Well okay, I'm mostly concerned about interoperability with Anyway, what information does mocha provide that isn't supported by tap? Pending tests? Duration? Hmm... maybe you're right. starting to read TAP spec |
|
I just noticed that jenkins uses tap too, see node.js tests. I'd say it's pretty much a standard thing already. |
|
When in Node space, using JSON is the sane thing to do. |
|
Yes, tap is a standard. Yes, it's widely used. Yes, it has a fair amount of usage in node. No, it doesn't contain enough information for any of our reporters. No, not many systems are actually consuming TAP and doing anything useful with the fact that they can see how many tests passed/failed. |
|
I needed something like this for mocha-broken so I put together a quick prototype I'm using json-stream as a quick intermediate format until I put together a more comprehensive one that supports suites and fail errors. Actually if we update json-stream with suites and errors that would be ideal. |
|
btw there is:
There are 10 reporters that work today with
|
|
Hmm it may have been easier to adapt mocha-broken to use tap and then use a tap reporter. Thanks! I assume we can close this now that there are sufficient alternatives? On Wed, Jun 11, 2014 at 7:05 PM, Raynos (Jake Verbaten)
|
The mocha reporters are sweet. However they are hard coupled to the mocha interface.
It would be elegant if they were a seperate module.
Consider mocha just outputting TAP format for the tests to stdout
And then doing
mocha test | mocha-reporters --dotThe mocha-reporters module could just be a thing that takes TAP input and outputs to the reporter to stdout.
You could also consider using it programatically as
mocha.output.pipe(mochaReporter.spec).pipe(process.stdout)This allows people to re-use the reporters in any test framework that outputs TAP.
If you think this is a sensible idea we can draft out how the refactoring would go and I can try migrating the reporters into seperate modules. I've pretty much ported the HTML reporter to something that consumes TAP already so that feels like duplicated effort.
The text was updated successfully, but these errors were encountered: