Testing your CLI

The cmdtest package simplifies testing of command-line interfaces. It provides a simple, cross-platform, shell-like language to express command execution. It can compare actual output with the expected output, and can also update a file with new "golden" output that is deemed correct.

Test files

Start using cmdtest by writing a test file with the extension .ct. The test file will consist of commands prefixed by $ and expected output following each command. Lines starting with # are comments. Example:

# Testing for my-cli.

# The "help" command.
$ my-cli help
my-cli is a CLI, and this is its help.

# Verify that an invalid command fails and prints a useful error.
$ my-cli invalidcmd --> FAIL
Error: unknown command "invalidcmd".

You can leave the expected output out and let cmdtest fill it in for you using update mode (see below).

More details on test file format:

• Before the first line starting with a $, empty lines and lines beginning with "#" are ignored.
• A sequence of consecutive lines starting with $ begin a test case. These lines are commands to execute. See below for the valid commands.
• Lines following the $ lines are command output (merged stdout and stderr). Output is always treated literally.
• After the command output there should be a blank line. Between that blank line and the next $ line, empty lines and lines beginning with # are ignored. (Because of these rules, cmdtest cannot distinguish trailing blank lines in the output.)
• Syntax of a line beginning with $:
  • A sequence of space-separated words (no quoting is supported). The first word is the command, the rest are its args. If the next-to-last word is <, the last word is interpreted as a file and becomes the standard input to the command. None of the built-in commands (see below) support input redirection, but commands defined with Program do.
• By default, commands are expected to succeed, and the test will fail otherwise. However, commands that are expected to fail can be marked with a --> FAIL suffix.

All test files in the same directory make up a test suite. See the TestSuite documentation for the syntax of test files, and the testdata/ directory for examples.

Commands

cmdtest comes with the following built-in commands:

• cd DIR
• cat FILE
• mkdir DIR
• setenv VAR VALUE
• echo ARG1 ARG2 ...
• fecho FILE ARG1 ARG2 ...

These all have their usual Unix shell meaning, except for fecho, which writes its arguments to a file (output redirection is not supported). All file and directory arguments must refer to the current directory; that is, they cannot contain slashes.

You can add your own custom commands by adding them to the TestSuite.Commands map; keep reading for an example.

Variable substitution

cmdtest does its own environment variable substitution, using the syntax ${VAR}. Test execution inherits the full environment of the test binary caller (typically, your shell). The environment variable ROOTDIR is set to the temporary directory created to run the test file (except in parallel mode; see below).

Running the tests

To test, first read the suite:

ts, err := cmdtest.Read("testdata")

Next, configure the resulting TestSuite by adding a Setup function and/or adding commands to the Commands map. In particular, you will want to add a command for your CLI. There are two ways to do this: you can run your CLI binary directly from from inside the test binary process, or you can build the CLI binary and have the test binary run it as a sub-process.

Invoking your CLI in-process

To run your CLI from inside the test binary, you will have to prevent it from calling os.Exit. You may be able to refactor your main function like this:

func main() {
        os.Exit(run())
}

func run() int {
    // Your previous main here, returning 0 for success.
}

Then, add the command for your CLI to the TestSuite:

ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run)

Invoking your CLI out-of-process

You can also run your CLI as an ordinary program, if you build it first. You can do this outside of your test, or inside with code like

if err := exec.Command("go", "build", ".").Run(); err != nil {
        t.Fatal(err)
}
defer os.Remove("my-cli")

Then add the command for your CLI to the TestSuite:

ts.Commands["my-cli"] = cmdtest.Program("my-cli")

Running the test

Finally, call TestSuite.Run with false to compare the expected output to the actual output, or true to update the expected output. Typically, this boolean will be the value of a flag. So, your final test code will look something like:

var update = flag.Bool("update", false, "update test files with results")

func TestCLI(t *testing.T) {
    ts, err := cmdtest.Read("testdata")
    if err != nil {
        t.Fatal(err)
    }
    ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run)
    ts.Run(t, *update)
}

Parallel mode

If you call ts.RunParallel instead of ts.Run, each file in the suite is run in parallel with the others. (The cases in a single file are still run sequentially, however.) In this mode, no temporary directories are created and ROOTDIR is not set.