Tiny Unified Runner N' Tester (Turnt)

Turnt is a simple snapshot testing tool inspired by Cram and LLVM's lit. It's good for testing things that translate text files to other text files, like compilers. The idea is that each test is one input file, and you want to run a command and check that it still matches the saved output file.

To use it:

1. Configure. Decide what command you want to test. Make a turnt.toml config file and put command = "mycmd {filename}" in it to pass each test file as an argument to mycmd.
2. Create a test. Just write an input file next to your turnt.toml. We'll call it foo.t.
3. Take a snapshot. Run turnt --save foo.t to execute mycmd foo.t and save the standard output into foo.out. You might want to take a look at this output to make sure it's what you expect. Then you check both the input foo.t and output foo.out into version control.
4. Test your work. Now that you have a test in place, keep working. Use turnt *.t to run all your tests and confirm that the output still matches. If there's a mismatch, you can do turnt --diff to see the changes. (Or if you're confident, try turnt --save followed by git diff.)

Turnt's philosophy is to minimize the effort it takes to write new tests so you can quickly build up lots of them. You don't write any custom logic to check results; you just record the complete "golden" output for each test.

Compared to ordinary unit testing, "snapshot" tests incur the mental effort of manually inspecting diffs when things change. In return, it's easier to expand test coverage. Snapshots also act as a crude form of documentation because every test is a complete, valid input to your program.

Install

This is a Python 3 tool. Install it with pip:

$ pip install --user turnt

Or, if you want to work on Turnt, you can install Flit, clone this repository, and type this to get a "live" installation with a symlink:

$ flit install --symlink --user

Configuring

Turnt looks for a configuration file called turnt.toml in any of the ancestor directories of your test file. It can be alongside the test file or in any containing directory. It's a TOML file that looks like something this:

command = "mycmd {args} < {filename}"
return_code = 42
output.txt = "result.txt"

command

Set command to a shell command to run on a given test file. This is the only setting that is truly required.

The command is a template; Turnt will fill in these values:

• {filename}: The path to the test file, relative to the working directory.
• {base}: The basename of the test file, with the extension removed.
• {args}: Some extra arguments that the test or user provides. (See the ARGS: override and the --args command-line option below.)

The working directory for the command is the location of the turnt.toml configuration file, if any. If there's no configuration file, then it's the location of the test file itself.

return_code

By default, Turnt expects the test command to succeed, i.e., exit with status code 0. Set return_code to a different status if you expect failure.

output

By default, Turnt captures the standard output stream from your test command. If your command produces other output files "on the side" or you want to capture the standard error stream, you can configure the output table.

output is a mapping from snapshot extensions to collected filenames. For example, this TOML configuration:

output.txt = "result.txt"

means that running the command will produce a file called result.txt, and we want to save that file in a snapshot called <test-name>.txt.

In place of a filename, use - to indicate the command's standard output and 2 to indicate its standard error. The default behaves like this configuration:

output.out = "-"

which captures stdout and saves it in <test-name>.out. Defining output in turnt.toml disables this default behavior; you can include it explicitly if you want it alongside other outputs.

binary

By default, Turnt looks inside test files for overrides (see below). This won't work if your test inputs are binary (non-text) files (Turnt will warn you and proceed with no overrides). Set binary = true to suppress this search for overrides altogether.

out_dir

Optionally put expected test outputs in a different directory. By default, this is ., i.e., expected output files go "next to" test files. Set this to a different directory, relative to the test file, to keep them somewhere else.

todo

Mark a test as an allowed failure. You might add a test that exposes a bug that you haven't fixed yet, for example, but you want your CI to stay "green." Set todo = true to allow the test to fail; Turnt will mark it with # todo in the TAP output.

Per-Test Overrides

Sometimes you need to alter the setup for a specific test file. Turnt looks for some overrides embedded in the test file itself: for example, you might put them in a comment at the top of a test program.

Put these things into your test file to override the configuration:

• CMD: <command> overrides command from the configuration.
• ARGS: <arguments> adds arguments to a configured command. Turnt puts this string in where the command uses {args}.
• OUT: <ext> <filename> overrides output from the configuration. You can specify multiple files this way: one line per file.
• RETURN: <code> overrides the expected exit status.
• TODO: <true|false> can mark a test as a "todo" (allowed failure).

Multiple Environments

Turnt is mostly about running one command on many input files. Sometimes, however, you need to run several commands on each file. This can be especially useful for differential testing: when you want to check that multiple things behave the same way by checking that they produce the same input when you give them the same input.

You can create multiple environments in your configuration file under the envs table. The table maps environment names to sets of configuration options that are just like the top-level configuration described above. For example:

[envs.baseline]
command = "interp -g {filename}"

[envs.optimized]
command = "compile -O3 {filename} ; ./a.out"

Each environment can have the full complement of configuration options described above: for example, command, output, and return_code. When you run turnt on some files, it will run all the environments on all the files. There is also a Boolean default flag to turn off an environment by default; that way, you can only use it by asking for it with the -e flag (see below).

This example is a differential testing setup because both environments share the same (default) snapshot file: a test foo.t will look for its stdout snapshot in foo.out. If the two environments don't match, at least one will fail. If you want a more standard (non-differential) setup, just set the output configuration differently for the two environments, like this:

[envs.interp]
command = "interp -g {filename}"
output.res = "-"

[envs.profile]
command = "profile {filename}"
output.prof = "-"

Directory Tests

A Turnt test is usually just a single input file, but you can also organize multiple related files into a directory. Use the directory the same way as you would a single file: pass its path to the turnt command, and the path will appear as the {filename} for the configured command. So you might configure your test command like this:

command = "mycmd {filename}/test.c"

if you want each test directory to contain a file called test.c.

Turnt puts snapshots inside the test directory instead of adjacent to it. It names them out.<extension> in that directory.

There are two configuration options just for dealing with directory tests:

• out_base. The basename for output files in directory tests: by default, out.
• opts_file. The filename to read inside of a directory test to search for embedded overrides. In our above example, you could set this to test.c to look for ARGS: and friends in that file, or opts.txt to look for them in a separate file on the side.

Command-Line Interface

The most common turnt command-line options you'll need while running and updating tests are:

• --save: Bless the current output from each test as the "correct" output, saving it to the output file that you'll want to check into version control.
• --diff: Show diffs between the actual and expected output for each test. (The diff goes to stderr while the TAP results remain on stdout.)

You also might enjoy:

• --parallel or -j: Run your tests faster using parallel threads.

These options are useful when working with one specific test file:

• --verbose or -v: Disable Turnt's default behavior where it will suppress test commands' stderr output. The result is more helpful but harder to read.
• --print or -p: Instead of checking test results, just run the command and show the output directly. This can be useful (especially in combination with -v) when iterating on a test interactively.
• --args or -a: Override the {args} string in the test command.

These options lets you switch between different test environments:

• --env or -e: Give the name of a configured environment to run. Use this multiple times to run multiple environments. By default, Turnt runs all the configured environments for every test. Use the special name default for the top-level, unnamed test environment.
• --config or -c: Look for this config filename instead of the default turnt.toml.

TAP

Turnt outputs results in the machine-readable TAP format. To make the output more pleasant to read, you can pipe it into a tool like tap-difflet, tap-dot, or faucet:

$ npm install -g tap-difflet
$ turnt *.t | tap-difflet

Credits

Turnt is by Adrian Sampson and Alexa VanHattum. We made it to test various research compilers in Capra. The license is MIT.