Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests #10

Merged
merged 28 commits into from
Sep 23, 2016
Merged

Tests #10

merged 28 commits into from
Sep 23, 2016

Conversation

Apanatshka
Copy link
Contributor

@Apanatshka Apanatshka commented Aug 6, 2016 •
edited
Loading

Writing tests as described in #3.
Going straight for the bonus points: so far I've written quickcheck tests for a number of unit test situations.

TODO list

Unit tests

  • Test the benchmark parser, including the fact that throughput is optional.
  • Test that find overlap finds the correct pairs.
  • Test that find overlap reports correct missing benchmarks (for both old and new).
  • Test that the name determination for the two columns works correctly when given two file paths. (This may require refactoring the names method off of the Args struct.)
  • Test commafy.
  • Test that the name prefix feature works. (Refactor code to get a unit test.)

Integration tests

  • Test some basic example inputs and check that it produces the expected output.
  • Test that passing input on stdin works.

@Apanatshka Apanatshka mentioned this pull request Aug 10, 2016
Closed
@Apanatshka
Copy link
Contributor Author

@BurntSushi: I refactored some code to make it more testable, but didn't do the most minimal change because I saw room for improvement. The changes are in a2caf90.

@Apanatshka
Copy link
Contributor Author

The last two test can be implemented once uutils/coreutils#959 is resolved, which should provide a nice crate for integration testing of command line tools.

@Apanatshka
Copy link
Contributor Author

Looks like the appveyor build is failing because of rust-lang/rust#35991.

@Apanatshka Apanatshka changed the title Tests (WIP) Tests Aug 29, 2016
@Apanatshka
Copy link
Contributor Author

@BurntSushi: Ready for review/merge

@Apanatshka
Copy link
Contributor Author

@BurntSushi Still waiting on a review...

BurntSushi
BurntSushi reviewed Sep 22, 2016
View reviewed changes
Cargo.toml
@@ -15,6 +15,11 @@ license = "Unlicense/MIT"
[dependencies]
docopt = "0.6.81"
lazy_static = "0.2.1"
prettytable-rs = "0.6.2"
quickcheck = "0.3.1"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both quickcheck and second_law should be dev-dependencies, which means they'll only be required during tests.

@BurntSushi
Copy link
Owner

@Apanatshka Overall this looks really good. Could you say whether there are any behavior changes here?

Sorry about neglecting your PRs. I've been really into another project, and any leftover time I had after that has gone into wedding planning!

@Apanatshka
Copy link
Contributor Author

In reverse order:

I've been really into another project,

That doesn't surprise me at all, you seem to be quite the collector :P

and any leftover time I had after that has gone into wedding planning!

If that's not a legitimate "real life" excuse, I don't know what is :) Good luck with that, and remember to enjoy it rather than stressing out ^^

Sorry about neglecting your PRs

I don't mind nagging until you respond if you don't ;)

Could you say whether there are any behavior changes here?

Only one outside observable change that I'm aware of: Newlines in the output are now consistently unix \n. I found during testing that prettytable-rs actually did OS-specific line breaks, but in our own output code that wasn't done. The OS-specific stuff is now optional through a feature flag in that crate, so I turned it off.

@BurntSushi
Copy link
Owner

All right great! Since I merged your other color PR, this one now has a conflict. After a rebase I'll merge. :-)

@Apanatshka
Added quickcheck dep
3be1301
@Apanatshka
Using quickcheck tests on overlap
58fe145
@Apanatshka
Added quickcheck tests for commafy
ec52c3e
@Apanatshka
Added benchmark parsing quickcheck tests
8d19222
@Apanatshka
Using more idiomatic .parse() instead of ::from_str()
5aeabc8
@Apanatshka
checksums that the nightly toolchain adds (doesn't conflict with stable)
dbcef59
@Apanatshka
Turned Args.names into Args::names for testability and added quickche…
7d32715 
…ck tests
@Apanatshka
cargo fmt
88c3a10
@Apanatshka
Now checking string suffix with std function
91cdd64
@Apanatshka
Turned benchmark name prefix into more testable function
7a1cad9 
While doing so, removed reading whole files/stdin in favour of
buffered reading.
@Apanatshka
Removed unnecessary .ok() call
6f64d0b
@Apanatshka
Refactored split_benchmarks more, added quickcheck tests
2b1c187
@Apanatshka
Added integration tests with normal input/output
02ca7cb
@Apanatshka
Added stdin test
cf8d6c6
@Apanatshka
Attempted fix for integration test issue on windows
e18c767
@Apanatshka
Final fixes for windows integration tests
9da6441 
Now using windows line endings in expected output when on the windows
platform.
@Apanatshka
Removed quickcheck property that doesn't hold
f5d9c3d 
Counter-example found in
https://ci.appveyor.com/project/BurntSushi/cargo-benchcmp/build/1.0.23/job/fm2b0kwg993qeuuh
The from_original property should cover the same functionality anyway,
so this doesn't weaken the test suite.
@Apanatshka
Add integration test with non-overlapping benchmark names
5f979e7
@Apanatshka
Fix test: don't give output when no comparisons
346e51b
@Apanatshka
Copy link
Contributor Author

Hah, the test for the usage string just failed, because the --color option is now added.

Question: Is this even a good test? I basically duplicated the string in the code to a file and test that that's the output on -h/--help...

@BurntSushi
Copy link
Owner

@Apanatshka Oh, wow, I see. I missed that. I'd throw away that test. It's better to just write new tests for each new flag/option added. I think having to update a separate file every time the usage string changes is probably too annoying.

@Apanatshka
Update usage string test
2587c99
@Apanatshka
Remove static included strings from tests
6416d6e 
Accidentally left over from old workaround that was supposed to be completely removed in 3cb31d6
@Apanatshka
Remove integration test of usage string
5a6eea4
@Apanatshka
Copy link
Contributor Author

Ok then. The usage string test is removed now.
In a related category, there is still a test in there for the version flag. But it grabs the version from a cargo environment variable, so it shouldn't be fragile.

It's better to just write new tests for each new flag/option added.

Yeah, ideally I'd like to add integration tests for the coloured output.. But the planned "quickly rebase so it can be merged today" is already taking more time that I expected. I should really get back to work now ^^' In 6 hours I'll have time to capture the output with terminal colour codes in a fixture for a test.

@BurntSushi BurntSushi merged commit b021cab into BurntSushi:master Sep 23, 2016
@BurntSushi
Copy link
Owner

@Apanatshka This is great, thank you!

(Tests for coloring are strictly nice to have. Don't sweat it. :-))

@Apanatshka Apanatshka mentioned this pull request Sep 27, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BurntSushi BurntSushi BurntSushi left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Apanatshka @BurntSushi