[RFC/WIP]: Use pytest to run CoAP tasks

[kb2ma]

Contribution description

This PR refines the experimental 09-CoAP tasks in #77 by using pytest to automate running the tasks. This PR also is a kind of RFC for use of pytest in RIOT. pytest provides a number of advantages for composing and running tests, and it provides an evolutionary path forward from RIOT's pexpect-based testrunner.

If you're not familiar with pytest, review the documentation or some of the presentations about it. The content of the tests in 09-coap are introduced in coap.md. I am a newbie to pytest myself, but we do use it at my employer.

01_test.py

Runs an aiocoap-based Resource Directory server as a pytest fixture in rd_server(). Also runs the gcoap CORD client as a fixture. These fixtures include a generic class to run pexpect, ExpectHost, found in conftest.py. They also use the capability for a fixture to include teardown code, which neatly packages this functionality with the setup code.

Finally, the test itself in test_register() is a single function call. Notice that the fixtures are included in the test by passing them to the test function.

02_test.py

Runs an soscoap server for its capability to short circuit responses to a confirmable request. The CoAP server is started in retry_server(), and the client is started in gcoap_example() in conftest.py, which allows for reuse in other tests.

The test functions also include an ignores parameter to vary the number of confirmable requests to ignore. The value of the parameter is specified in the 'parametrize' decorator. pytest includes a few mechanisms for parameterization, as described in the fixture documentation.

Finally, notice that the test_timeout function is able to simply modify the pexect timeout for this test. It also uses the pytest.raises() context manager function to mark that send_recv() is expected to timeout.

Other pytest features

pytest includes an extensive collection of plugins. Below you can see the output of the pytest-html plugin for a test run of 09-coap.

Thanks to @jia200x for his testutils PR #79, which inspired some of this work. Also thanks to @smlng and @MrKevinWeiss for RIOT #10241 and RIOT #10095 for Robot Framework.

Issues/PRs references

This PR improves on #77 by automating the tasks.

[aabadie]

Very interesting PR! cc'ing @cladmi who could also be interested in this work.

[cladmi]

I am really in favor of using pytest for wrapping our tests.

Right now, it could already be used without modifying our tests, by hackily reading the timeout value, and currently duplicating what is inside testrunner to not modify the original one.

I have a poc branch for it, was just a friday evening work so not refined enough yet. I modified one test to replace term by child to show multiple tests detection.

I did not have time to prepare a demo PR during the release though, I should now.

EDIT: better with the branch RIOT-OS/RIOT@master...cladmi:pr/wip/pytest

[cladmi]

Only nitpick about pytest-html for the moment, from my quick tests, it only generates the output for the current test and cannot really aggregate different test runs if executed for different tasks. So would need another tool to display an aggregated result of all junitXML outputs.

[kb2ma]

I now have added automated tests to cover all of the experimental tasks in #77. However, I am adding the tests to my riot-coap-pytest repository until the community decides on the next steps for functional test automation.

This PR really is more of an RFC/POC, and in that context, others now can review more pytest examples in the repository referenced above.

[kb2ma]

Closing for now as this PR has gotten no traction. However, I continue to push automated tests to my riot-coap-pytest repository. I am happy to reopen if others want to pursue this idea.