-
Notifications
You must be signed in to change notification settings - Fork 115
Writing test cases
There are a number of good reasons why it's worth writing a test case that checks typical behaviour for the kinds of problems you are interested in:
- if you have a test case checked into fluidity, it's easy for you, or anyone else, to run it, hence checking for problems in fluidity in the most convenient way;
- in particular, if your test case runs quickly enough, it can be included in the suite of tests that are run automatically by the buildbot every time a change is made to fluidity trunk - you will know very quickly if someone has broken fluidity in such a way that will affect the kinds of problem that you are interested in
A test case consists of two things:
- A problem to run (a flml file, any generic functions, etc)
- A way to check if the problem ran correctly
For example you might wish to consider:
- A parallel problem and test it doesn't crash
- Performance and test that the problem ran in a reasonable time
- Diagnostic values and test that the answer is correct
The test case will be run on buildbot by the test harness and can also be run locally.
To examine how to write a test case, we're going to take a look at the cdisk_2d_dg test case from the fluidity trunk (tests/cdisk_2d_dg). It is a very simple example that sets up a concentration of tracer and advects and diffuses it around and then tests that the tracer is conserved.
Let's examine the contents of the tests/cdisk_2d_dg directory.
[FLUIDITY_PATH]$ ls -l tests/cdisk_2d_dg/
total 228
-rw-r--r-- 1 skramer skramer 777 2009-01-29 12:07 cdisk_2d_dg.xml
-rw-r--r-- 1 skramer skramer 213383 2008-09-23 11:05 cdisk.flml
-rw-r--r-- 1 skramer skramer 166 2009-01-29 12:02 Makefile
drwxr-xr-x 3 skramer skramer 4096 2009-01-29 12:02 src
[FLUIDITY_PATH]$ ls -l tests/cdisk_2d_dg/src
total 196
-rw-r--r-- 1 skramer skramer 412 2008-09-23 11:05 cdisk.geo
Let's take each of these in turn:
- the src/ subdirectory of tests/cdisk_2d_dg: this contains the mesh file, in this case just the .geo file that serves as input for gmsh, and any extra files that may be needed to setup the problem.
- cdisk.flml this is the fluidity options file to run the problem with fluidity.
- Makefile: A Makefile is a way of grouping together commands that do a particular thing. The test harness needs to know how to mesh the problem from scratch therefore the Makefile contains instructions on how to mesh the problem.
Let's take a look at the example:
[FLUIDITY_PATH]$ cat tests/cdisk_2d_dg/Makefile
input: clean
gmsh -2 -o cdisk.msh src/cdisk.geo
../../tools/gmsh2triangle --2d cdisk.msh
clean:
rm -f *.ele *.edge *.node *.poly *.vtu *.s *.d.1 *.stat *.msh
This defines an input 'target' that runs gmsh to create a mesh from src/cdisk.geo and output it as cdisk.msh and then runs gmsh2triangle to make the triangle mesh (in 2d: cdisk.node, cdisk.ele and cdisk.edge) that is specified in the cdisk.flml. It also defines a clean target that removes all output from a previous run. The fact that clean is also behind the colon of input means that clean always gets run before 'making the input'. You can try it out by running 'make input' on the command line.
- cdisk_2d_dg.xml, this tells the testharness how to run your problem and how to perform the tests. It is described in the section below. It has to have the same name as the test directory!
In order to run your problem, the test harness has to know a few things:
- Exactly how to run your problem -- is it serial or parallel? If parallel, how many processors? What command line do you use? Are there any other special options?
- What do you want to test? What variables do you want to look at? What about them do you want to check? What consists of a pass and what consists of a failure?
Test case XML files can be created and edited using a text editor or the Local:Diamond GUI:
diamond -s [FLUIDITY_PATH]/schemas/test_options.rng [XML_FILE]
The XML file (in this case, cdisk_2d_dg.xml) contains the following information, dissected section-by-section to see how it works, (don't let the syntax throw you: it's very easy):
[FLUIDITY_PATH]$ cat tests/cdisk_2d_dg/cdisk_2d_dg.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE testproblem SYSTEM "regressiontest.dtd">
The first two funny-looking lines are a necessary evil that you don't ever need to understand or change.
<testproblem>
<name>Concentration Disk</name>
<owner userid="dham"/>
<tags>flml</tags>
<problem_definition length="short" nprocs="1">
<command_line>fluidity cdisk.flml </command_line>
<!-- Trivial 2D test case of uniform flow through a square. -->
</problem_definition>
(Note that you must use >, < and & instead of the >, < and & characters! You must use these since >, < and & are special characters in XML.)
The above section tells the test harness what the name of the problem is (Concentration Disk), who the owner is, how long it takes to run, how many processors it uses, and the command line to execute. Let's take them in turn:
- The name: used for printing out the results of the test case. There are no particular requirements for it.
- The owner: username of the person who wrote and will maintain the test
- The tags: you can associate arbitrary labels with tests, so that you can group them later. All tests should have at least one tag, e.g. "flml", depending on how the problem is set up.
- The length: can be one of "short", "medium", "long" and "special". This is used in the buildbot to decide how and when to run your problem automatically. Note that "long" tests require that a script called 'genpbs' is called. This means you cannot run them on your local system. Change to "special" to run locally, remembering to change back to "long" before committing.
Short test cases (less than 30 seconds) are run everywhere (32- and 64-bit, intel and gcc compilers) for every commit. Medium test cases (30 seconds - 20 minutes) are run on beefier machines (the 64-bit ones). Long ones (more than 20 minutes) will be run continuously and you will also need to read #Notes on long test cases. (Times are runtime with debugging enabled.) Special test cases are never automatically invoked (this is useful for test cases that you know currently fail, etc.).
- The number of processors: here this is a simple serial job, so the number of processors (nprocs) is 1. For all short or medium tests this should be 1. For long test, this should be the number of processors to use.
- The command line: what command to execute to run your problem. (Previous to this the test harness has made the input files as described in the Makefile). The test harness executes this command and waits for it to finish before moving on. You can '''cheat''' buildbot into running a parallel test as a short or medium test by adding 'mpiexec -n 2' to the command line. For long tests this is not necessary as the correct mpiexec is added automatically.
The next section defines variables that you want to look at. Each variable has a name. Here we want to check the maximum concentration of tracer left after it's been diffused to check it's diffusing correctly. This is the part of the XML file that describes how to find that:
<variables>
<variable name="conservation" language="python">
from fluidity_tools import stat_parser
s = stat_parser("cdisk.stat")
conservation=s["Water"]["Tracer"]["integral"][-1]
</variable>
</variables>
This snippet of code tells the test harness how to get the "conservation" variable. The first line of code imports the stat_parser that is very useful for writing test cases. It's used in the second line to read all information contained in the .stat file into a variable s. The third line means take the last, denoted by [-1], (of all time steps) of the "integral" of the field "Tracer" of the material_phase "Water" and assign that value to variable conservation.
You can have many variables, just make a separate <variable>..</variable> block for each one.
Now that we have the variables we want to check, we need to check that they have suitable values. This is the function of the next snippet from tests/cdisk_2d_dg/cdisk_2d_dg.xml:
<pass_tests>
<test name="Tracer is conserved." language="python">
assert abs(conservation-0.195925) &lt; 1e-4
</test>
</pass_tests>
This defines a test, called "Tracer is conserved", that checks that the integral of the concentration is close enough to 0.195925. The test passes if no assertion is raised, and it fails if the assertion is not true.
<warn_tests>
</warn_tests>
</testproblem>
Finally we finish up. This problem has no tests that issue warnings, so <warn_tests></warn_tests> is empty.
Again, note that if you use > or < in your tests, you have to replace them with > or < respectively, because > or < are part of the XML syntax.
Now that you have the Makefile (specifying how to mesh your problem) and your .xml file (specifying what to run and what to check) written, run your test case with:
[FLUIDITY_PATH]$ tools/testharness.py -f TESTCASE.xml
This will make the input, run the command line, fetch the variables and check they meet all the conditions you specify.
If you have already run the test case and so have dump files etc available, but want to play with what is checked in the xml file, you can run the tests without having to run the simulation with Fluidity all over again. To do this type:
[FLUIDITY_PATH]$ tools/testharness.py -f TESTCASE.xml --just-test
- Long test cases have a separate git repository: https://github.com/FluidityProject/longtests
If you wish to use the test harness then this needs to be checked out to a directory called longtests i.e.:
bzr co lp:~fluidity-core/fluidity/longtests longtests
The running of the test harness is otherwise the same.
-
Long test cases are NOT scheduled automatically. After committing a long test case, contact the buildbot administrator to have it scheduled.
-
Long test cases must have mpiexec before any executables in <command_line>. This is to make it play nice with PBS.