Testing

Testing

webKnossos supports unit tests, integration/end-to-end (E2E) tests, and screenshot tests.

Requirements

• Google Chrome
• node v0.12+
• A running wK installation

Running Tests

yarn test  # For frontend unit tests
sbt test  # For E2E tests only (using a dedicated test DB)

E2E can also be run on directly, as long as a wK instance is running in background

yarn test-e2e  # For E2E tests

Ava

Both unit tests and E2E tests are run using the Ava test runner and are written in ES6.

import test from 'ava';

test(t => {
	t.deepEqual([1, 2], [1, 2]);
});

Unit Tests

All unit tests live in the frontend/javascripts/test directory and end with .spec.ts.

End-to-End (E2E) tests

All E2E tests live in the public-test/test-bundle/test/backend-snapshot-tests/ directory and end with .e2e.js. These files are automatically generated by the yarn test-e2e command. These tests are executed against a running server with a testing database. E2E tests may use ava's snapshot testing functionality, which has an interface similar to snapshot(key, value). If no value was saved for a key, the given value is saved. Otherwise, the given value is compared to the saved value and the test passes if the values are equal.

For example, the test/enzyme folder contains tests which render react components including data from the server, while using snapshots.

Run E2E Tests

Run docker compose up e2e-tests

Alternatively:

1. Switch the default mongo DB in conf/application.conf from play-oxalis to webknossos-testing
2. Run sbt test.

Handling failed snapshot tests

Check the diff which ava shows. The snapshot change is likely due to code change which may be either right or wrong. If it is wrong, fix the corresponding code. Otherwise, update the snapshots by removing them (yarn remove-snapshots) and rerunning the e2e tests.

Troubleshooting

The tests fail.

Ensure that ...

• ... you have not another server and/or mongo instance running.
• ... dependencies are installed (yarn install) and the JS files are transpiled (yarn test-prepare-watch).
• ... that docker is executed with the necessary rights (sudo might be necessary)
• ... that the testing DB is up-to-date. Did your PR introduce a new evolution? Apply it manually to the testing DB and export the DB using tools/import_export/export.sh webknossos-testing test/db/.

Docker messed up the file permissions

sudo chown -R $(whoami) webknossosRepository

Replacing / Creating New Snapshots

Whenever there are no snapshot files present, the E2E will generate new snapshots on the first run.

1. Delete the existing the snapshots yarn remove-snapshots
2. Run the E2E tests once. (see instructions above)
3. Commit new snapshots.

Test DB

For the E2E tests, the DB dump stored in testdb/ is imported into the oxalis-testing DB. This happens automatically.

In order to change the test db, perform the following steps:

# Drop the play-oxalis db
tools/dropDB.sh

# Import the testdb into play-oxalis
tools/import_export/import.sh play-oxalis test/db/

# Do appropriate changes to the db...

# Export play-oxalis, overwriting the existing testdb dump
tools/import_export/export.sh play-oxalis test/db/

Screenshot tests

We're running screenshot tests using puppeteer (headless chrome), which allows us to do pixel perfect comparison tests of our rendering output. All screenshot tests live in the app/assets/javascripts/test directory and end with .screenshot.js.

Run screenshot tests

Run docker compose up screenshot-tests.

The tests are run against a live instance (local or deployed), and for authentication purposes they assume that an X-Auth-Token of secretScmBoyToken is present in the database and valid (this is automatically inserted for our dev instances). The tests will use the instance at master.webknossos.xyz by default. A specific other instance can be used by specifying the url as follows: URL=yourdeployedbranch.webknossos.xyz docker compose up screenshot-tests.

To renew the screenshots, simply delete them, they are then newly created when running the tests for the next time.

Handling failed screenshot tests

Whenever a screenshot test fails, a diff screenshot as well es the new screenshot are saved to disk, to allow for visual inspection. See the command line output for the screenshot directory.