UnitTests

The first unit tests developed for the toolbox were created in late 2.5 versions (more specifically 2.5.32). From version 2.6, the idea to use unit tests continuously for development started. Currently, the coverage is continuously increasing and one of our objectives is to have as much coverage as possible in the long term.

If you are developing for the toolbox, It is highly recommended that you write tests as you code. This way, the task at hand transforms into writing code that passes the behaviour you are expecting and reducing side-effects. This is not bullet-proof but ensures minimal testing is available.

Triggering all tests in the repository.

To trigger all tests in the imos toolbox repository you need:

1. Obtain all testfiles
2. If on windows, run the runalltests.bat script
3. If on Linux, run the runalltests.sh script.

Triggering all tests within matlab

Both sh/bat scripts are just shell wrappers to call batchTesting.m within matlab. The batchTesting.m script will execute all the tests in the repository and report a test coverage summary at the end, with all tests that pass,fail or files that were skipped for some reason.

The unit tests in the imos-toolbox uses two different "frameworks" to do test evaluation. Each framework is triggered by a particular function, so check our Frameworks section for more details.

How to obtain the test-files

The get_testfiles.py python script requires a python3 environment with two external packages - boto3 and docopt. They are easily installed with pip:

pip install boto3 docopt

Most test scripts assumes all tests are found within the data/test_files folder in the toolbox_root_folder. Since tests are not distributed with the code, you may need to create this folder before running the get_testfiles.py script:

cd <toolbox_root_folder>
mkdir data/testfiles/
./get_testfiles.py 

The python script will download all files from the AWS imos-toolbox bucket and you will be ready to run all tests. Please note that you don't need any special permissions to download the files.

The script will also sync the test folder within the current s3 bucket if further execution is made, avoiding repeated downloads. This is so because we store the md5sum of each file in the s3 bucket, fetch these metadata entries, and compare them with local files. The script will only download data that is missing or with mismatched md5sum (corrupted/incomplete).

Available Tests for instruments ingestion

At the moment, we currently hold test files for the following manufacturers/instruments:

1. Sea Bird Scientific

   • SBE
     • 37plus
     • 39plus
     • 9plus
     • 16plus
     • 25plus
   • ECOTriplet
     • FLSB
   • WQM
     • DAT files
     • RAW files

2. RBR (dat/txt files)

   • TR-1060
   • TDR-1060
   • TDR-2050
   • TWR-2050
   • XR420

3. JFE

   • infinitySD

4. Star Oddi

   • mini
   • DST Tilt
   • DST CTD

5. FSI

   • NXIC CTD

6. Nortek

   • aquadopp profiler
   • aquadopp velocity
   • Awac wave
   • continental ADCPs
   • Signature (1000,500,250) ADCPs

7. Teledyne RDI

   • workhorse

8. matlab

• Nortek OceanContour

11. Netcdf

• IMOS Netcdf convention files

GUI tests

Most of the toolbox GUI features cannot be tested automatically and will require some sort of manual interactivity. Features implemented before version 2.6 are only testable manually - no unit tests were written (yet) for them. Thus, manual GUI testing is, most of the time, always required. Some side effects in the GUI are quite common so be aware that different or chained callbacks can be triggered when in you within a GUI window.

I recommend testing GUI behaviour with the stand-alone binary app. The reason why is because the Matlab runtime is a reduced set of libraries, including graphical ones, and some behaviour/functions may not be completely available in the stand-alone app. I also strongly recommend battle testing new development against unhandled inputs (argument sanitisers/type checking).

See also https://github.com/aodn/imos-toolbox/wiki/GUIDevelopment if you plan to develop/extend the GUI functionality.

Frameworks

The unit tests here are written in two different "frameworks" - xunit and as docstrings. XUnit is a MathWorks framework based on a popular java library. testDocstring is our own take on the task of writing tests within the documentation body of a function.

The idea is the same, implementation and behaviour is slightly different.

Xunit

As said, Xunit is a test framework native to matlab. It allows one to write tests in several ways, including an object-oriented style when testing classes or simple functions. Xunit requires you to write specific functions to evaluate your own code. You can write them as functions or classes. You also need to, in general, use the testSuite class for test execution, debugging, and collection of outputs. Moreover, you may need to configure plugins for a more sane test reporting.

All tests using xUnit are located in the <imos_toolbox_root_folder>/test folder. Individual tests tests can be run with the runtests matlab function, or with the testSuite class. We provide a simple wrapper to run all xunit tests in one go - test/runAllTests.m. This function is also used by batchTesting.m, which is the interface to run all toolbox tests.

testDocstring

Another possibility to evaluate tests is to use Util/testUtils/testDocstring.m( and Util/testUtils/checkDocstring.m to evaluate all files in a given folder). The testDocstring.m function evaluates the Example block of our docstring standard. Hence, this functionality enables one to write code snippets in the documentation section of a function and evaluate them.

This has the advantage that a good (bad) docstring triggers a pass (fail) test, which promotes good documentation & tests practices. It also reduces the overhead that for every function file, there is a respective test function. You can find examples in some functions in the codebase, particularly with new functionality (e.g. Util/+IMOS toolbox functions).

Thus, testDocstring abstracts the requirement of writing another file for testing. The inspiration behaviour is from rust and doctest.

The usage is also quite flexible:

testDocstring <name_of_function>
testDocstring('<name_of_function>')

As for xUnit, the batchTesting.m function will also inspect all matlab source files (via checkDocstrings) in the repository and report the results of the docstring evaluations (if any). Files that do not follow the docstring standard will be skipped.

For example,

cd <imos_toolbox_root>
[ok,msgs]=checkDocstrings('PreProcessing')

will evaluate all docstring tests in all functions within the <imos_toolbox_root>/PreProcessing folder, and capture any error msgs.

Should I use Xunit or docstrings tests?

Individual functions, utilities and toolboxes benefit quite well from testDocstring, and as such, most of the low-level functionality is tested using tests as docstrings. One limitation of testDocstring is that only functions following our docstring template are supported. This is so because the docstring parser is quite simple. If no example block can be found, the evaluation is skipped and the function will return a pass (True).

On the other hand, xunit tests have advantages for complex input and requirements. Hence, it is more suited to parsers and top-level functionality, since they usually require dealing with instrument files or fully loaded datasets. In some cases, some tests are multisteps and as such, not suitable or practical for tests as docstrings. For example, declaring several files or specific behaviour given individual files would create a docstring that is too long.

How to write functions that testDocstring will evaluate.

To use testDocstring you need to make sure you follow our documentation template. By using this template you promote good documentation & tests practices. You can find examples in some functions in the codebase (e.g. inCell). I recommend that new functionality follows this template (e.g. see Util/+IMOS toolbox functions).

File Requirements for xunit tests.

Some tests - particularly parsers - require instrument files. These files are usually binary or text files. Previously, some instrument files were included in the codebase. This was not scalable since it would increase the codebase size, slow down git operations, and would force all users to store test files that are not used when operating the toolbox.

Now, all instrument files are now kept outside of the codebase within an AWS bucket. This s3 bucket - imos-toolbox - is maintained by AODN toolbox developers.

The developers are responsible to maintain and keep files in the bucket so they can be fetched and further used for testing. Developers (and users) that wish to run all unit tests will need to obtain these files first, via the python script get_testfiles.py. See how to obtain the test files below.

The testing code that uses hard files assumes the developer/user will first fetch/store these files in the <toolbox_root_path>/data/testfiles folder.

Again, To obtain all files, use the get_testfiles.py Python script. The folder hierarchy follows a Instrument_Maker/Instrument_Model/InstrumentVersion/filename.extension style. This style was created given the typical testing workflow. For example, after a long-time, the Star_Oddi manufacturer updated their processing software and Mini model text files were affected. Previous files for testing are stored in Star_Oddi/mini/v000/ while new files are in Star_Oddi/mini/v001. See also <toolbox_root_path>/test/Parser/testStartoddi.m.

Finally, please be aware that some functionality requires one to compare ingestion results of an instrument file against a previous version of the toolbox. This is useful when refactoring an instrument parser and ensure they are correct and following some expectations. You may also leverage our treeDiff function to test content comparison.

treeDiff will evaluate, recursively, the content of structures or cells, including the toolbox datasets. Although recursively comparison is available from isequal (e.g. structs/cells), one usually fall short about the reason two entities are not equal. treeDiff tries to solve this problem by returning "reasons" of why some items are not equivalent, and as such, help on debugging why the fields are different and if those are significant (e.g. order of assignment, missing variables, different attributes, etc).