Describe how to write and run functional tests with Cram

Also places one sentence per line in the testing section and clarifies how to
run a subset of unit tests.

DEV_DOCS.md

@@ -59,54 +59,93 @@ output of `augur` installed with pip and `./bin/augur` from your local source co
 ### Testing
 
 Writing good tests and running tests helps maintain code quality and eases future refactoring.
-We use the [pytest](https://docs.pytest.org) package for running our tests.
+We use [pytest](https://docs.pytest.org) and [Cram](https://bitheap.org/cram/) to test augur.
 This section will describe briefly:
 
 - Writing tests
   - Unit tests
   - Doctests
-  - End-to-end tests
+  - Functional tests
 - Running tests
   - Locally
   - Continuous Integration
 
 #### Writing Tests
 
-It's good practice to write **unit tests** for any code contribution. The
-[pytest documentation](https://docs.pytest.org) and [Python documentation](https://docs.python.org)
-are good references for unit tests. Augur's unit tests are located in the `tests` directory and
-there is generally one test file for each code file.
+It's good practice to write **unit tests** for any code contribution.
+The [pytest documentation](https://docs.pytest.org) and [Python documentation](https://docs.python.org) are good references for unit tests.
+Augur's unit tests are located in the `tests` directory and there is generally one test file for each code file.
 
-On the other hand, [**doctests**](https://docs.python.org/3/library/doctest.html) are a type of
-tests that are written within a module's docstrings. They can be helpful for testing a real-world
-example and determining if a regression is introduced in a particular module.
+On the other hand, [**doctests**](https://docs.python.org/3/library/doctest.html) are a type of tests that are written within a module's docstrings.
+They can be helpful for testing a real-world example and determining if a regression is introduced in a particular module.
 
-A pull request **should always contain unit tests**. Optionally, a pull request may also contain
-doctests if the contributor believes a doctest would improve the documentation and execution
-of a real world example.
+A pull request that contributes new code **should always contain unit tests**.
+Optionally, a pull request may also contain doctests if the contributor believes a doctest would improve the documentation and execution of a real world example.
 
-End-to-end tests confirm that augur behaves as expected in real-world pipelines.
-These domain-specific tests are implemented as Snakemake pipelines in `tests/builds`.
+We test augur's command line interface with functional tests implemented with the [Cram framework](https://bitheap.org/cram/).
+These tests complement existing unit tests of individual augur Python functions by running augur commands on the shell and confirming that these commands:
+
+1. execute without any errors
+2. produce exactly the expected outputs for the given inputs
+
+These tests can reveal bugs resulting from untested internal functions or untested combinations fo internal functions.
+
+Functional tests should either:
+
+* suitably test a single augur command with an eponymously named Cram file in `tests/functional/` (e.g., `mask.t` for augur mask)
+
+OR
+
+* test a complete build with augur commands with an appropriately named Cram file in `tests/builds/` (e.g., `zika.t` for the example Zika build)
+
+##### Functional tests of specific commands
+
+Functional tests of specific commands consist of a single Cram file per test and a corresponding directory of expected inputs and outputs to use for comparison of test results.
+
+The Cram file should test most reasonable combinations of command arguments and flags.
+
+##### Functional tests of example builds
+
+Functional tests of example builds use output from a real Snakemake workflow as expected inputs and outputs.
+These tests should confirm that all steps of a workflow can execute and produce the expected output.
+These tests reflect actual augur usage in workflows and are not intended to comprehensively test interfaces for specific augur commands.
+
+The Cram file should replicate the example workflow from start to end.
+These tests should use the output of the Snakemake workflow (e.g., files in `zika/results/` for the Zika build test) as the expected inputs and outputs.
+
+##### Comparing outputs of augur commands
+
+Compare deterministic outputs of augur commands with a `diff` between the expected and observed output files.
+For extremely simple deterministic outputs, use the expected text written to standard output instead of creating a separate expected output file.
+
+To compare trees with stochastic branch lengths:
+
+1. provide a fixed random seed to the tree builder executable (e.g., `--tree-builder-args "-seed 314159"` for the “iqtree” method of augur tree)
+2. use `scripts/diff_trees.py` instead of `diff` and optionally provide a specific number to  `--significant-digits` to limit the precision that should be considered in the diff
+
+To compare JSON outputs with stochastic numerical values, use `scripts/diff_jsons.py` with the appropriate `--significant-digits` argument.
+
+Both tree and JSON comparison scripts rely on [deepdiff](https://deepdiff.readthedocs.io/en/latest/) for underlying comparisons.
 
 #### Running Tests
 
 You've written tests and now you want to run them to see if they are passing.
-First, install augur with the development dependencies as described above.
-Next, run augur's tests (unit tests and doctests) with the following command from the root, top-level of the augur repository:
+First, you will need to [install the complete Nextstrain environment](https://nextstrain.org/docs/getting-started/local-installation) and augur dev dependencies as described above.
+Next, run all augur tests with the following command from the root, top-level of the augur repository:
 
 ```bash
 ./run_tests.sh
 ```
 
-Troubleshooting tip: As tests run on the development code in the augur repository, your environment should not have an existing augur installation that could cause a conflict in pytest.
-
-To run end-to-end tests, use the following command from the root, top-level directory.
-You will need to [install the complete Nextstrain environment](https://nextstrain.org/docs/getting-started/local-installation) to run these tests.
+For rapid execution of a subset of unit tests (as during test-driven development), the `-k` argument will disable code coverage and functional tests and pass directly to pytest to limit the tests that are run.
+For example, the following command only runs unit tests related to augur mask.
 
 ```bash
-bash tests/builds/runner.sh
+./run_tests.sh -k test_mask
 ```
 
+Troubleshooting tip: As tests run on the development code in the augur repository, your environment should not have an existing augur installation that could cause a conflict in pytest.
+
 We use continuous integration with Travis CI to run tests on every pull request submitted to the project.
 We use [codecov](https://codecov.io/) to automatically produce test coverage for new contributions and the project as a whole.