Testing with GoogleTest

Introduction

For testing the correctness of our functions in t8code, we use MPI extensions of the GoogleTest framework. This framework allows us to test code sections independently for several parameters and properties of t8code. The task is finding code errors in the early development of new t8code sections. You get more information and guidelines about the implementation of GoogleTests at GoogleTest User’s Guide.

Guidelines

A test should only check one property of the code. A common test is the Parameterized Test. These tests allow you to test code sections over a range of variables. In our case, this is the go-to when testing element-functions for all available elements in a scheme. The tests should not test specific cases, but should also be applicable to code extensions. Keep the code simple and understandable. For this you should follow our Coding Guideline · DLR-AMR/t8code.

Output

The test should not print an output apart from error-messages of failing tests. Other messages should only print a message in debug configuration. Use the command t8_debugf to print such messages.

Assertions and expectations

Various checks can be applied to the code. There are assertions and expectations. The expectations are called by EXPECT_*. If an expectation fails, the current test (or function) will not be aborted. By using an assertion, which is called by ASSERT_*, a test (or a function) will abort directly. Note that if a function will be aborted, that the test will not be aborted. It is not possible to use assertions in functions, that have a return value.

The rule of thumb should be: Use EXPECT_* whenever possible. Use ASSERT_* only if the test cannot continue if the asserted part fails.

The most common assertions and expectations are:

• EXPECT/ ASSERT_TRUE
• EXPECT/ ASSERT_FALSE
• EXPECT/ ASSERT_EQ (equal)
• EXPECT/ ASSERT_NE (not equal)
• EXPECT/ ASSERT_ LT (less than)
• EXPECT/ ASSERT_LE (less or equal)
• EXPECT/ ASSERT_GT (greater than)
• EXPECT/ ASSERT_GE (greater or equal)

More checks on: Assertions Reference | GoogleTest.

Choose the most fitting check for your test. Use ASSERT_TRUE, ASSERT_FALSE, EXPECT_TRUE or EXPECT_FALSE only if no other check can be applied.

Customized Assertions for t8code

In many test we check the equality for two elements. We implemented a customized EXPECT/ASSERT for that case. It also handles the printing of the elements (coordinates, level, ...). If you want to check if two elements are equal use EXPECT_ELEM_EQ. To use this function in your test you have to include the header test/t8_gtest_custom_assertion.hxx in your testfile. If you find another case, where a customized assertion can be benefitial, have a look at Advanced Google Test and Pretty formatted Google Test Assertions to contribute your customized assertion.

Failure message

If a test fails we would like to know more about the reason for aborting. A failure message, which is connected with an assertion (or expectation) by using << “my_string” throws an information message if the test fails at this point.

ASSERT_TRUE(t8_cmesh_is_commited(cmesh)) << "Cmesh commit failed.";
ASSERT_EQ(dual_face, checkface) << “Wrong dual face. Expected “ << checkface << “ got ” << dual_face;

MPI extensions of Google Test

Since t8code is designed as an MPI parallel library, the tests are executed in parallel as well.

However, the original GoogleTest is not designed for MPI parallel use. Hence, we use the MPI extension.

This extension communicates the test result automatically among all processes in MPI_COMM_WORLD. Hence, if the test fails on one process it fails on all processes. This ensures that the failure is definitely reported.

Additionally, the extension introduces the new EXPECT_*_MPI and ASSERT_*_MPI macros. These are MPI collective variants of EXPECT_* and `ASSERT_*. Thus, if the specific check fails on one process, then it fails on all processes.

Whenever you need to ASSERT something and carry out parallel communication afterwards, you must use one of the ASSERT_*_MPI macros. Otherwise, the test could run in a deadlock.

For more information, see https://github.com/DLR-SC/googletest_mpi/blob/feature/mpi_nonblocking_expect-1.10.0/googletest/docs/MPIGuide.md

GoogleTest and Cmeshes

Some of our tests require us to test over multiple cmeshes. In the following we explain how to do that and how to add a new example to our test-suite.

How to write a test that uses cmeshes as parameter

You can use the parameterized tests of GoogleTest to do so. The parameter passed to the test has the type cmesh_example_base * that is defined in test/t8_cmesh_generator/t8_cmesh_example_sets.hxx. To create the cmesh from the parameter you use the member-function cmesh_create(). The class does not take ownership of the cmesh, so you are responsible for its memory. Therefore the class your test-suite uses should look like this:

#include <test/t8_cmesh_generator/t8_cmesh_example_sets.hxx>

class my_cmesh_test: public testing::TestWithParam<cmesh_example_base *> {
 protected:
  void
  SetUp () override
    cmesh = GetParam ()->cmesh_create ();
    /* Initialise the rest of your test */
  }

  void
  TearDown () override
  {
    t8_cmesh_unref (&cmesh);
    /* Free the rest of your variables if necessary */
  }

  t8_cmesh_t cmesh;
  /* other members*/
};

Use AllCmeshsParam to iterate over all parameter combinations for all cmeshes example that are already transfered to the new parametrized tests (Take care, not all are already transfered! ). To print the full name of your example and the parameters used to create the cmesh you can pass the lambda pretty_print_base_example to the instantiation of your test suite:

INSTANTIATE_TEST_SUITE_P (Name, my_cmesh_test, AllCmeshsParam, pretty_print_base_example);

How to add a new cmesh to our test-suite

The files in t8_cmesh_generator/ enable us to produce the union of sets of parametrized tests. One can use the class cmesh_cartesian_product_paramsto create a new set of cmesh-examples that use all combinations of parameters passed to the constructor. The class 'cmesh_sum_of_sets' is able to merge multiple different sets of cmesh-examples into one.

The following step-by-step solution will help you too add a new example to the parametrized tests.

1. Create a new file in test/t8_cmesh_generator/t8_cmesh_parametrized_examples/
2. Create an std::function wrapper around the cmesh example that you want to use
3. (Optional) If not already there, create the vector of values of your paramter-type you want to test over in t8_cmesh_params.hxx
4. Create a function that prints the parameters of your example
5. Create a std::function wrapper around your print-function
6. Use the cmesh_cartestion_product_params to generate all combinations of your parameters
7. Add your new cmesh_cartesian_product_params object to the cart_prod_vec in t8_cmesh_example_sets.hxx