Skip to content

Commit

Permalink
[docs] Improve testing Readme
Browse files Browse the repository at this point in the history
  • Loading branch information
salkinium committed Apr 21, 2021
1 parent f6f0fd6 commit 4c80ba1
Show file tree
Hide file tree
Showing 2 changed files with 51 additions and 60 deletions.
2 changes: 1 addition & 1 deletion docs/src/guide/custom-project.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ want [to choose one at this point](../../reference/build-systems). Add the
```xml
<library>
<repositories>
<repository><path>../modm/repo.lb</path></repository>
<repository><path>../../ext/modm/repo.lb</path></repository>
</repositories>
<options>
<option name="modm:build:project.name">custom_name</option>
Expand Down
109 changes: 50 additions & 59 deletions test/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Testing modm

modm includes a large number of example as well as an extensive unit test suite
which can be compiled and execute on hosted and embedded targets. Weve written
which can be compiled and execute on hosted and embedded targets. We've written
our own small test harness to be able to fit the tests onto very small devices
like AVRs. As part of our Continuous Integration checks, all examples are
compiled and hundreds of unit tests are executed to help us catch regressions.
Expand All @@ -19,7 +19,7 @@ abilities we have available today.

## Manual Tests

There is plenty of opportunity to customize modm, either using lbuilds module
There is plenty of opportunity to customize modm, either using lbuild's module
options or simply by using the HAL on your specific hardware. This means the
more examples we have of different configurations and use-cases the better we
can debug existing problems and prevent new ones.
Expand All @@ -29,58 +29,70 @@ so that others have an idea of the environment in which it was developed.
Test your new code on real hardware with the compiler toolchain we use and make
sure to document any caveats you discover.

You can compile all examples locally using our helper script.
You can compile all or parts of the examples locally using our helper script.

```sh
$ cd examples
$ ../tools/scripts/examples_compile.py .
cd examples
../tools/scripts/examples_compile.py .
# Or only a subset of the examples
../tools/scripts/examples_compile.py generic/blinky
```

!!! tip "Add your example to our CI"
If you created a new folder inside `modm/examples`, you need to add it to
one of the jobs in `.circleci/config.yml`:
```sh
- run:
name: Examples YOUR TARGET
command: |
(cd examples && ../tools/scripts/examples_compile.py new_folder)
```

## Continuous Integration

modm uses [GitHub Actions for CI][gha] to compile all the examples and to run
the unit tests on Linux. It also generates a bunch of artifacts that you can
download to check what the CI has been doing.

If you create a new folder inside `modm/examples`, you need to add it to the
appropriate job in in `.github/workflows/linux.yml`:

```yml
- name: Examples for new or existing families
if: always()
run: |
(cd examples && ../tools/scripts/examples_compile.py your_example_folder)
```
[gha]: https://github.com/modm-io/modm/actions
## Unit Tests
Our unit tests are located in the `modm/test` directory and are generated using
lbuild as well. They are all submodules of `modm:test` and are all platform
independent. Weve written a small Makefile for the most commonly used commands.
independent. We've written a small Makefile for the most commonly used commands.

```sh
$ cd tests
# generates, compiles and executes the unit tests for hosted targets
$ make run-hosted-linux
$ make run-hosted-darwin
# generates and compiles the unit tests for embedded targets
$ make compile-nucleo-f411
$ make compile-nucleo-f103
$ make compile-al-avreb-can
# executes the unit tests on the embedded targets
$ make run-nucleo-f411
$ make run-nucleo-f103
$ make run-al-avreb-can
cd tests
# generates, compiles and executes the unit tests for hosted targets
make run-hosted-linux
make run-hosted-darwin
# generates and compiles the unit tests for embedded targets
make compile-nucleo-f411
make compile-nucleo-f103_A
make compile-nucleo-f103_B
make compile-arduino-nano_A # to _H
# executes the unit tests on the embedded targets
make run-nucleo-f411
make run-nucleo-f103_A
make run-nucleo-f103_B
make run-arduino-nano_A # to _H
```

!!! info "Monitor the serial output"
The embedded test targets all use the `modm::Board` interface to initialize
the targets and output unit tests results via the default serial connection.
The embedded test targets all use the `modm::Board` interface to initialize the
targets and output unit tests results via the default serial connection.

The unit test library we use is located in `modm/src/unittest` which is the
`modm:unittest` modules. See the existing unit tests for example on how to write
your own.
The unit test library we use is located in `modm/src/unittest` which corresponds
to the `modm:unittest` modules. See the existing unit tests for examples on how
to write your own.

!!! bug "Running unit tests on small targets"
Fitting all unit tests into one executable image is not possible on smaller
AVR and STM32 targets. For these targets only a subset of unit tests must be
selected in the `project.xml` file and multiple images must be executed
manually. We would like to automate this in the future.
Fitting all unit tests into one executable image is not possible on smaller AVR
and STM32 targets. For these targets multiple compile targets generate multiple
images with partial unit tests that must be executed manually. In the future we
would like to execute the unit tests as well as the examples automatically on
the target hardware as well.


## Test all Targets
Expand All @@ -90,7 +102,6 @@ for ALL targets we support and compile it with a simple GPIO example.
This doesn't catch everything, but it does make sure that these modules are at
least compilable, which is particularly interesting when writing new peripheral
drivers which may have different implementations on other targets.
You may call

```sh
$ cd test/all
Expand All @@ -109,23 +120,3 @@ FAIL 0
Total: 209
$ make run-failed # only run those that failed the last test
```

!!! info "Target Compilation Logs"
The individual compilation logs are all saved as artifacts, so that when
something fails, it‘s easy to understand why.
You can access them in [the Artifacts tab of a build][logs].


## Continuous Integration

modm uses [CircleCI as a Continuous Integration service][circleci]. It compiles
all the examples and executes the unit tests on Linux, then generates and
compiles the full library for a blinky example for all targets that we support.

In the future we would like to execute the unit tests as well as the examples
automatically on the target hardware as well. However, there isn‘t any
third-party framework to make that easy, so we probably have to write our own.


[logs]: https://circleci.com/gh/modm-io/modm/971#artifacts/containers/0
[circleci]: https://circleci.com/gh/modm-io/workflows/modm/tree/develop

0 comments on commit 4c80ba1

Please sign in to comment.