Exercism Test Runner Template

This repository is a template repository for creating test runners for Exercism tracks.

Using the Test Runner Template

1. Ensure that your track has not already implemented a test runner. If there is, there will be a https://github.com/exercism/<track>-test-runner repository (i.e. if your track's slug is python, the test runner repo would be https://github.com/exercism/python-test-runner)
2. Follow GitHub's documentation for creating a repository from a template repository
   • Name your new repository based on your language track's slug (i.e. if your track is for Python, your test runner repo name is python-test-runner)
3. Remove this Exercism Test Runner Template section from the README.md file
4. Build the test runner, conforming to the Test Runner interface specification.
   • Update the files to match your track's needs. At the very least, you'll need to update bin/run.sh, Dockerfile and the test solutions in the tests directory
   • Tip: look for TODO: comments to point you towards code that need updating
   • Tip: look for OPTIONAL: comments to point you towards code that could be useful

Once you're happy with your test runner, open an issue on the exercism/exercism to request an official test runner repository for your track.

Exercism TRACK_NAME_HERE Test Runner

The Docker image to automatically run tests on TRACK_NAME_HERE solutions submitted to Exercism.

Run the test runner

To run the tests of an arbitrary exercise, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the test runner on an exercise using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests of an arbitrary exercise using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-in-docker.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the tests

To run the tests to verify the behavior of the test runner, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.

Run the tests using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests to verify the behavior of the test runner using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests-in-docker.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.