Standard Library of Test Helpers

[ztombol]

Overview

This pull request introduces a Standard Library of Test Helpers that provides better feedback on failed tests to help debugging. It also argues that including this library in Bats greatly simplifies its usage.

Problem

When a test fails Bats displays a lot of useful information, e.g. the failed expression and its line number. However, in many cases additional information would be helpful. Providing relevant information to speed up debugging is an actively pursued feature. It has been requested and proposed multiple times without success. Implementations generally fall into one of the following two categories.

1. Modifying Bats to automatically print a static set of information, e.g. $status and $output, when a test fails.

   Advantages:

   • Just works - There is no need to modify tests to take advantage of this feature.

   Disadvantages:

   • Invasive - Prints regardless the user wants it or not (consequence of being automatic).
   • Inflexible - May print irrelevant information making debugging harder.

   Due to its disadvantages this approach has been rejected multiple times in the past (see report $output content when test fails #8, bats_print_failed_command: Include $output if available. #101, Dump ${output} from tests by default? #105).

2. Using test helpers to print only information relevant to the failure.

   Advantages:

   • Concise - Only displays the information relevant to the failure.
   • Opt-in - Produces output only if the developer wants it.

   Disadvantages:

   • Tests need a rewrite - Tests have to be rewritten to take advantage of this feature (consequence of being opt-in).

   This is the recommended approach (see Assertions #5).

Solution 2 is clearly the superior, and in fact the recommended approach. Two years have passed since that recommendation but a general purpose library has never emerged.

_Note:_ To be fair, @juanibiapina shared his library of assertions in #5. However, as of 2015-07-19 the link is dead and library cannot be found on his Github account.

Standard Library of Test Helpers

At the moment projects have to develop their own library by adapting another project's test helpers if the licences permit it or writing one from scratch. Six projects on the list of projects using Bats use a test helper library. All of which are a variation of the same library.

_Note:_ I know that the list mentioned above is not up to date, but users of Bats may take the projects on that list as references when building their own test suites.

• rbenv
• ruby-build
• nodenv
• shy
• envirius
• github-markdown-toc

The current situation has the following problems:

• Reinventing the wheel - Every project has to invest into developing their own, very similar, library.
• Licensing - If the licence of the source and your project are not compatible, you are back to writing it from scratch.
• No testing - Ironically none of the six projects mentioned above actually test their test helpers. Hence, bugs may propagate from one project to another.
• Non-standard API - It is harder for developers new to a project to contribute because they need to read the source of the test helpers on the off chance that these behave a little different than in another project where they encountered a similar library.
• No documentation - Total lack of documentation of test helpers in any of the mentioned projects further raises the barrier to entry.
• Isolated improvements - The lack of an upstream version and concentrated development efforts mean that improvements to the library of one project does not directly benefit another, and sharing improvements is difficult.

A properly licensed Standard Library of Test Helpers could mitigate these problems and provide a better experience for Bats users. Despite the advantages of a general purpose library, one has never emerged. This pull request introduces the Standard Library containing the most commonly sought functions.

Including the Standard Library in Bats

Although, distributing a library of test helpers with Bats has been rejected in the early days, I believe its time to re-evaluate the question in light of the current situation.

The dependency chain of the test suit for a typical project using the included Standard library would look like the following.

+-------+    +------------------+    +----------+    +------+
| Tests | -> | Project specific | -> | Standard | -> | Bats |
|       |    |  Test Libraries  |    | Library  |    |      |
+-------+    +------------------+    +----------+    +------+

\_______________________________/    \______________________/
               |                                |
            Project                           Bats

Including the Standard Library in Bats would greatly simplify its usage and would put less burden on individual projects.

• In sync with Bats - The Standard Library would always be kept compatible with Bats. This would eliminate the delay between a compatibility breaking update of Bats and the update of the Standard Library. Project specific test helper libraries would still need to be updated of course.
• No vendoring - The Standard Library would be available wherever Bats is installed without the project needing to vendor and periodically update it.

Additionally, having an official, universally available, easy to use Standard Library would encourage contribution.

Licensing

Regardless whether a general purpose library, like the one proposed in this pull request, is included in Bats, licensing remains an important question.

Test files are just input data to Bats. The licence of Bats does not restrict the licence of the input files or the licence of the software being tested.

However, when using test helper libraries, the licence of the libraries and the licence of the test files using them must be compatible.

Furthermore, if a test file sources a file of the project (for example to do unit tests on a library) the licences of the sourced file, the test helper library, and the test file must all be compatible.

This means that the licence of a general purpose test helper library can limit what projects may use it. A permissive licence or even the Public Domain may be necessary to allow many projects to use the library.

Summary

A Standard Library of Test Helpers helps projects to write better test suits faster. However, no such library has emerged during the past two years. This pull request introduces a library containing the most common test helpers.

Including this library in Bats, rather than hosting it in a separate repository, improves its usability, simplifies maintenance and encourages contribution.

The licence of the Standard Library has a great impact on its usability and should be carefully considered.

[sstephenson]

I'm in favor of including a standard library in Bats itself. I'll take a closer look at your patch tomorrow, but I think this makes a lot of sense.

Thank you for the very thoughtful, well-researched and reasoned pull request.

[sstephenson]

Some initial thoughts:

1. No need to split this up into multiple modules—one standard library file is sufficient. In other words, we don't need the separate "core" distinction.
2. Why shouldn't we automatically include these helpers for every test case? We can source the stdlib before the test file, so any existing functions with the same name just override the defaults.
3. If we decide we can't automatically include the helpers, we'll need a better name than load_std. I'd rather not load an STD into my tests ;-)
4. Let's move the documentation right into the main readme. These helpers should be front and center.
5. We ought to publish a separate man page for the helpers, too, as with bats(7). Giving the standard library a name like batslib might make this easier.

[sstephenson]

libexec is a systemwide namespace (/usr/libexec, /usr/local/libexec). Currently, we're safe installing directly into it because every filename is prefixed with bats. This patch introduces a std directory in that namespace, which I don't think we can rightfully claim ownership to.

I'd suggest restructuring things like this:

libexec/bats
libexec/bats-exec-suite
libexec/bats-exec-test
libexec/bats-format-tap-stream
libexec/bats-preprocess
lib/bats/batslib.bash
lib/bats/batslib/output.bash

[sstephenson]

What's the use case for piping a message to flunk?

[ztombol]

Piping simplifies the design and implementation of how output is generated by library functions.

When a test fails, output is generated in stages. Each stage has a well-defined job and pipes its output into the next stage.

1. Generate details: { ... } groups together the output of multiple commands that produce the formatted details relevant to the failed test, which is then all piped into the next stage at once.
2. Add header and footer: _decorate adds the header and footer around the input received from the previous stage.
3. Display and fail: flunk redirects input to STDERR and returns 1, effectively causing the test to fail and the output to be displayed.

This is cleaner than the alternative of separately printing a header, body and footer, all while having all function individually redirect output to STDERR.

See assert_success or assert_line for example.

[ztombol]

@sstephenson thank you for the detailed feedback!

• No need to split this up into multiple modules—one standard library file is sufficient. In other words, we don't need the separate "core" distinction.
• Why shouldn't we automatically include these helpers for every test case? We can source the stdlib before the test file, so any existing functions with the same name just override the defaults.

I agree with both. Simplifies usage quite a bit.

• If we decide we can't automatically include the helpers, we'll need a better name than load_std. I'd rather not load an STD into my tests ;-)

This cracked me up! Good one! 😄

• Let's move the documentation right into the main readme. These helpers should be front and center.
• We ought to publish a separate man page for the helpers, too, as with bats(7). Giving the standard library a name like batslib might make this easier.

Agreed. I'm also thinking of rewriting the documentation to be more concise (e.g. man page style), we'll see how it works out.

• libexec is a systemwide namespace (/usr/libexec, /usr/local/libexec). Currently, we're safe installing directly into it because every filename is prefixed with bats. This patch introduces a std directory in that namespace, which I don't think we can rightfully claim ownership to.

I completely overlooked that. Moved to lib/bats as requested.

Most of these changes are already done. Once I finish the man page I'll update the pull request. Hopefully by Wednesday.

P.S.: Sorry for the delay! Life happened. I'm back on working on this pull request. Replies should be quicker now. :)

_Edit:_ Add libexec issue.

[ztombol]

Pull Request v2

Overview

Thanks to the great feedback of @sstephenson and @aw-altiscale, I finished implementing the first round of changes and amended the pull request.

We have three different documentations now.

• developer: function comments in the source batslib.bash
• reference: concise description in man page batslib(7)
• how-to: detailed description with examples in README.md

I'm not quite happy with the current form of the man page. It's dry and hard to read. It's not much more friendly than the developer documentation. I will rewrite it as we work out how some of the
functions should change.

Changes

• Move library to lib/bats.
• Merge modules into a single library.
• Change name of variable storing the library's location to $BATS_LIB from $BATS_LIBSTD.
• Automatically load the library.
• Move documentation into README.md. Mention the library in the description of run.
• Add batslib(7). Update SEE ALSO section of other man pages.
• Prefix private functions with batslib_, e.g. batslib_err.
• Remove space between function names and parenthesis.
• Wrap source comments to 72 characters wide.
• Add notice about compound commands, e.g. [[, to assert.
• Replace sort -g with sort -n for portability.

Questions

• Is there a limit on what utilities the library can use? Can it use grep or wc? At the moment it uses sort.

TODO

• Implement partial and regex matching in assert_output.
• Test on environments other than linux with bash 4.3.

_Edit:_ Fix function name in TODO: partial and regex matching is for assert_output, not assert_failure and assert_success. - 2015-09-08 21:40 UTC

[ztombol]

Pull Request v3

Overview

I finished reorganising the output matching functionality. It took longer than I expected, sorry about that.

Now, the only functions that match output are assert_output and its new complement, refute_output. They implement the line matching features of the now removed assert_line and refute_line. They've also gained partial (substring) and regex (extended) matching.

The generic assert now displays $status and output too in addition to the expression being tested. These new bits of information can be helpful in debugging and without them the function didn't provide more detail than the stack trace.

To make reviewing easier, I made new commits instead of amending the previous ones. I will squash them before merging.

Changes

• Remove output matching from assert_success and assert_failure.
• Add status matching to assert_failure, e.g. fail with exit code 2.
• Move assert_line into assert_output.
• Add refute_output to complement assert_output, and move refute_line into it.
• Add partial and regex matching to assert_output and refute_output.
• Add $status and $output to the output of assert.

TODO

• Choose a suitable licence.
• Test on environments other than Linux + Bash 4.3.

Questions

• Are there any restrictions on the dependencies we can have from the point of view of portability and being lightweight? At the moment the library relies on sort.

[ztombol]

Now that we are mixing up the library and moving away from the model libraries (nodenv, rbenv, etc), I'm thinking that the standard input feature in assert_output and refute_output could be removed. It doesn't seem very useful and it would simplify usage.

The specific line matching (-l) and the line searching (-L) features could be implemented with a single option (-l), to which specifying an optional index argument would cause specific line matching to be performed, and line searching otherwise. Seems more logical and easier to remember to me.

We could move from the current...

• assert_output <expected> - entire output matching
• assert_output -l <index> <expected> - specific line matching
• assert_output -L <expected> - line searching

... to the simpler:

• assert_output <expected> - entire output matching
• assert_output -l <index> <expected> - specific line matching
• assert_output -l <expected> - line searching

The current implementation relies on the number of arguments to figure out where does the expected output/line come from. This conflicts with options having optional arguments.

@sstephenson what do you think? Can we remove STDIN from these functions?

[thomasleveil]

great PR, looking forward to use the stdlib :)

May I suggest you rename flunk to fail (unless fail is a reserved word).

As a non-native English speaker, flunk is confusing and other test libs are usually calling such a feature fail.

[ztombol]

Good point @thomasleveil! The name flunk is a not-so-old remnant from the the first version of this pull request in which I tried to stay close to the model libraries (the ones used in rbenv, nodenv, etc.).

fail is not a reserved word in Bats, but let's see what @sstephenson thinks about it.

I'm in favour of changing flunk to fail. 👍

As a non-native myself, I had to look up this word. :)

[ztombol]

By the way, writing style in documentation and comments, and some other conventions have gradually changed since the first commit. I will clean up everything once we have the features worked out.

[ztombol]

Pull Request v4

Overview

There was no feedback for a week, so I went ahead and pushed the modifications I mentioned above. I can revert them later if that becomes necessary.

Changes

• Renamed flunk to fail
• Removed STDIN support from assert_output and refute_output
• Changed the option of line searching (formerly -L) to -l in assert_output and refute_output

TODO

• Clean up to have consistent coding and documentation style
• Choose a suitable licence.
• Test on environments other than Linux + Bash 4.3.

Questions

• Are there any restrictions on the dependencies we can have from the point of view of portability and being lightweight? At the moment the library relies on sort.

[ztombol]

Pull Request v5

I cleaned up all tests, documentation and code. Style should be consistent now.

@thomasleveil proposed some changes, but due to their scope they should be in their own separate PRs.

I think the following two questions left to answer before the PR can be merged.

• What licence should cover the standard library?
• What dependencies can the standard library use? It's using sort at the moment.

@sstephenson what do you think?

Changes

• Clean up documentation, comments and tests.
• Add documentation on errors from assert_output and refute_output

[danutchindris]

Great PR, looking forward to having this library included in Bats!

[mislav]

Thanks for working on this impressive pull request. Ultimately it's @sstephenson's call whether to merge this in, and he might not be able to get back to you for 2 more weeks since he's on vacation. However, if it was up to me I wouldn't merge this in for size alone. Before this, bats was 760 LoC in total. This PR alone adds 3,400 new lines. I understand that a big chunk of that is comprehensive documentation, which is really great, but still.

I much more like the prospect of a separate project as an assertion library, e.g. rbenv/ruby-build#817

[ztombol]

@jasonkarns At this point it wouldn't make much sense trying to merge this library into bats-assert. The pull request would replace most, if not all lines in the repository. Feature-wise bats-assert is a subset of this library. This library has a more mature API, extensive documentation and tests. Don't misunderstand, bats-assert is good. But it is what this this pull request was ~5 month ago. Since then it picked up long options and improved the API (both suggested by you).

Also, the name bats-assert suggests a narrower scope. From the beginning I imagined the standard library (or whatever we'll end up calling it) to be a collection of various general purpose helpers, not just assertions. We already have a solid set of basic assertions, and I hope that helpers of other kind (e.g. handling temporary directories) will be added in the future. So I don't think bats-assert would be a descriptive name.

I hope to find a fitting name that properly captures the scope. bats-helpers is an obvious candidate. Though, bats-sloth is tempting... I'm open to suggestions. 😄

Of course, the repository I'll make will be moved into a Bats organisation when created.

P.S. Sorry for the long hiatus. I'm still determined to make this library a reality.

_Edit:_ Add the note on Bats organisation. -- 2016-01-29 02:41 UTC

[ztombol]

@schrepfler Bats will not be dependent on anything. The library is opt-in. Only when you decide to use it, your test suite becomes dependent on one of the many planned installation methods (e.g. git-submodule, vendoring, package managers).

And, as @RomanSaveljev said, since @sstephenson has not been active in bats development recently, we cannot move forward with a battery included approach anyway. Perhaps after the library has matured and become widely popular (i.e. the advantages outweigh the disadvantages), we can revisit the question of inclusion.

[jasonkarns]

I'm excited to begin using the new lib and ideally replacing bats-assert. How soon do you think it will be standalone in its own repo?

As for the scope, there is also bats-mock that extracts the mocking utilities from ruby-build. I'm not excited about the idea of a helper lib that is so large to include assertions, helpers and mocks. But on the other hand, given bash's lack of package management, having it all brought in at once might be appealing. Regardless, there's a whole host of discussion related to mocking that would need to occur anyway (mocks vs stubs vs spies, etc)

I'd like to see this lib get its own repo so we can start using it and providing feedback.

[ztombol]

@jasonkarns I understand what you mean. I don't want a monolithic library either, but as you said the lack of package management makes things difficult and there's a trade-off between modularity and ease of use. Here are three possible alternatives from the slightly monolithic to the modular. Starting with the monolithic side.

Alternative 1: Common helpers as core

Since Bash package management is hard, there is a core library (bats-helpers) containing common helpers (e.g. basic assertions, other popular helpers) and internal functions used by all helpers (e.g. output formatting). People who only need the common features need only to install this library. Using other libraries requires the helper library to be installed as they depend on the internal functions it provides. It's not an ideal solution, but it's simple.

This is what I proposed.

Usage

To use the assertions (also installs potentially unnecessary helpers):

install bats-helpers

To use another library (also installs potentially unnecessary helpers):

install bats-helpers
install bats-other-library

Alternative 2: Assertions as core

Same as alternative 1, except that the core library (bats-assert) contains only assertions and internal functions. Other helpers go into separate libraries and depend on the assertion library to be installed. This is a more modular implementation, but it's counter intuitive why the assertion library has to be installed for using other libraries.

This what @jasonkarns suggested. (Am I correct?)

Usage

To use the assertions:

install bats-assert

To use another library (also installs potentially unnecessary helpers):

install bats-assert
install bats-other-library

Alternative 3: Fully modular architecture

Same as alternative 2, except that assertions (bats-assert) and internal functions (bats-core) are separated into their own libraries. The internal library must be installed in order to use any user facing library, e.g. assertions, mocking. This is a completely modular implementation. It ensures that no unnecessary helpers are installed, but at the cost of slightly more complicated usage.

Usage

To use the assertions:

install bats-core
install bats-assert

To use another library:

install bats-core
install bats-other-library

Conclusion

There's a trade-off between ease of use and modularity. We have an internal library that needs to be shared between libraries, which complicates the situation.

To be honest I would go with no 3, but I thought people wouldn't like the added complexity.

_Edit:_ Grammar, wording and formatting. -- 2016-01-29 20:44 UTC
_Edit:_ More grammar and spelling. -- 2016-01-30 23:19 UTC

[jasonkarns]

TBH, I was thinking more along the lines of option 3, but not thinking about any core/internal helpers other than standard assertions.

I think 3 may be the way to go for organization, but it increases the complexity of managing all the projects. (Especially considering there won't be any simple way to handle versioning and compatibility between the libs.) Personally, my projects will need to use npm for these libs. They won't need published to npm itself; rather, I would expect they simply include a minimal package.json and I would reference them directly from github. (I'm hoping that we can actually use the same package.json to support both npm and bpkg.) But for anyone who doesn't use npm, resolving version compatibility between these various libs would be a manual effort. This theoretically makes option 1 more appealing, but it doesn't actually avoid the issue, it just reduces the number of libs to deal with.

OTOH, since I don't expect a lot of churn, there probably won't be many version compatibility issues in the first place. So I'm still voting for 3.

[ztombol]

@jasonkarns Same here. While the handling of versioning and compatibility will not be a dream, I don't expect it to be a too heavy burden later on. Like you said, consolidating libraries would not help all that much.

I have a busy week ahead, but the standalone version should be ready by Friday (hopefully earlier). Most of the changes will be in documentation. Removing man pages, rewriting the section about loading libraries, and adding installation notes. I'll add git-submodule and git-clone as installation methods. The rest can be added by people more knowledgeable about npm, bpkg, etc.

It won't be super polished at first, but I want to release early and often. I will comment here when the repos are up.

[ztombol]

There are more changes than I expected, but the standalone version is coming along nicely.

There's however one question that didn't get enough attention before. What licence should we use? I can't imagine anyone wanting to invest in using software that may later turn out to be incompatible with his/her project. Especially that many would want to use these libraries at work to test commercial software.

I think a suitable licence should be selected before releasing these libraries.

Things that would help:

• comments from people knowledgeable on the topic
• asking on Stack Overflow
• writing to the FSF

Help is much appreciated. If nobody has the time, I'll go the Stack Overflow and FSF routes after finishing the standalone versions.

[juanibiapina]

MIT ?

[jasonkarns]

Bats itself is under MIT, so consistency would be nice. Though I believe @mislav generally prefers CC0

[ztombol]

I was thinking about the licence compatibility issue.

However, when using test helper libraries, the licence of the libraries and the licence of the test files using them must be compatible.

Furthermore, if a test file sources a file of the project (for example to do unit tests on a library) the licences of the sourced file, the test helper library, and the test file must all be compatible.

Is this correct? If yes, the licence we use could forbid certain projects of using the test libraries. The more permissive the licence, the larger the selection of compatible licences. But my reasoning could be incorrect.

[jasonkarns]

To my knowledge, CC0 is literally the most permissive license possible.
It's essentially public domain.

On Wed, Feb 3, 2016 at 1:00 PM, Zoltán Tömböl notifications@github.com
wrote:

I was thinking about the licence compatibility issue.

However, when using test helper libraries, the licence of the libraries
and the licence of the test files using them must be compatible.

Furthermore, if a test file sources a file of the project (for example to
do unit tests on a library) the licences of the sourced file, the test
helper library, and the test file must all be compatible.

Is this correct? If yes, the licence we use could forbid certain projects
of using the test libraries. The more permissive the licence, the larger
the selection of compatible licences. But my reasoning could be incorrect.

—
Reply to this email directly or view it on GitHub
#110 (comment).

[ztombol]

Consistency would be nice. Users of Bats would have to be familiar with only one licence instead of two. But we don't know whether the MIT licence would forbid testing software that is covered by a non-compatible licence, e.g. CC0. To be on the safe side, I vote for CC0.

The standalone version is not ready yet unfortunately. Hopefully I can finish it during the weekend. Until then, let's see if anybody else wants to chime in on the licence question.

[ztombol]

Release

After more than 9 month of development and nearly 7 month of discussion the test helper libraries have finally been released!

The initially proposed Standard Library of Test Helpers have been broken up into two external libraries and an additional repository storing shared documentation.

• bats-core - Supporting library for Bats test helpers. Limited to output formatting at the moment.
• bats-assert - Common assertions for Bats. Functions that were the main focus of this discussion.
• bats-docs - Shared documentation covering installation, loading and testing of libraries.

Features:

• Unified and simple library loading that shields the user from the internal implementation and layout of the library.
• Manual dependency loading to keep libraries simple.
• Detailed documentation.
• Thorough testing.
• Licensed under CC0 for compatibility with tested software.

Moving forward all discussion should happen on the issue tracker of the respective repository.

@thomasleveil You opened some very interesting pull requests over at my fork of Bats. If you are still interested, would you please reopen them on the new repositories?

Thanks

Thanks to everyone who contributed to this discussion.

Special thanks to @sstephenson @mislav @thomasleveil @jasonkarns for their invaluable contribution. Without you guys, we wouldn't have the quality libraries we have now.

[juanibiapina]

Awesome!! Looking forward to test them with basher when I'm back from the
holidays!
On Feb 9, 2016 11:20 PM, "Zoltán Tömböl" notifications@github.com wrote:

Release

After more than 9 month of development and nearly 7 month of discussion
the test helper libraries have finally been released!

The initially proposed Standard Library of Test Helpers have been broken
up into two external libraries and an additional repository storing shared
documentation.

• bats-core https://github.com/ztombol/bats-core - Supporting
  library for Bats test helpers. Limited to output formatting at the moment.
• bats-assert https://github.com/ztombol/bats-assert - Common
  assertions for Bats. Functions that were the main focus of this discussion.
• bats-docs https://github.com/ztombol/bats-docs - Shared
  documentation covering installation, loading and testing of libraries.

Features:

• Unified and simple library loading that shields the user from the
  internal implementation and layout of the library.
• Manual dependency loading to keep libraries simple.
• Detailed documentation.
• Thorough testing.
• Licensed under CC0 for compatibility
  Standard Library of Test Helpers #110 (comment)
  with tested software.

Moving forward all discussion should happen on the issue tracker of the
respective repository.

@thomasleveil https://github.com/thomasleveil You opened some very
interesting pull requests https://github.com/ztombol/bats/pulls over at
my fork of Bats. If you are still interested, would you please reopen them
on the new repositories?
Thanks

Thanks to everyone who contributed to the discussion.

Special thanks to @sstephenson https://github.com/sstephenson @mislav
https://github.com/mislav @thomasleveil https://github.com/thomasleveil
@jasonkarns https://github.com/jasonkarns for their invaluable
contribution. Without you guys, we wouldn't have the quality libraries we
have now.

—
Reply to this email directly or view it on GitHub
#110 (comment).

[ztombol]

Notifications

@X1011 @nskforward @ivans @gwen21 @dmp1ce @jwilder @marcondesdddi @mudler

I noticed that a while ago you started using the Bats test helper library we've been developing in this pull request. Good news! The library's been finally released! Check out the announcement above to learn more!

[thomasleveil]

@ztombol I will take a look at the pull requests later this week