Testing

Testing Garden

In Garden we have always taken testing quite seriously. The whole code base had been developed in a test driven manner, which basically means that there are tons of tests. We never really measured our code coverage but we suspect it is pretty darn high. Due to the low level nature of the project and its use of linux kernel primitives some tests can only run on linux. Some of our tests are being run on both linux and windows to check cross platform compatibility. Since we are all working on Mac OS machines all our tests rely heavily on Concourse - a continuous integration system that can run both linux and windows nodes. We also like Ginkgo - a golang BDD testing framework powering all our test suites.

Running the tests in a Garden repo

You can run all tests in most of the garden repositories by executing the following command in the repo root:

scripts/test

You can think about the scripts dir as a Makefile, each script being a target.

Note: Some scripts/test scripts will attempt to run the tests on concourse, so make sure you have fly - the concourse cli - installed and that you are logged onto concourse. Here is how you login in garden's concourse server

fly -t garden-ci login -c https://garden.ci.cf-app.com/

Garden has the following types of test suites:

• unit tests: these live next to the code and use fakes generated by counterfeiter
• integration tests: these usually live in the integration subdirectory in the repo they are testing and involve running the actual binary with different options and configurations. There is one exception - the GQTs - the guardian integration tests.
• acceptance tests: these live in their own repo and depend on a full deployment of the product they are testing. Since they are run against a given deployment, they cannot alter its configurations.
• performance tests: There are two performance suites:
  • GPATS
  • GATS Performance

Given that all our test scripts are a bit different you may want to read them to fully understand their behaviour, here are a few examples of our most important test scripts.

Running garden-runc-release tests

• ./scripts/test -a: runs all tests in parallel in tmux split windows. This is how we usually run these tests to save time. However you have to have tmux installed and configured.
• ./scripts/test : runs all tests serially, no tmux needed, but takes more time

Included tests

'All tests' includes:

• all garden tests

• all guardian tests, run in the following modes:

  • runc mode
  • containerd for containers, runc for processes
  • containerd for containers, runc for processes with CPU throttling enabled
  • containerd for containers and processes
  • winc mode on Windows worker VMs

• garden acceptance tests (GATs), in the following modes

  • runc mode
  • containerd for containers
  • garden in standalone mode

Running grootfs tests

• ./script/test -i: runs all integration tests
• ./script/test -u: runs all unit tests
• ./script/test <path>: runs all unit tests at
• ./script/test -a: runs all tests in parallel in tmux split windows.

Included tests

'All tests' includes:

• unit tests
• integration tests in following modes:
  • privileged (grootfs running as root)
  • unprivileged (grootfs running as uid 5000, gid 5000)