Enable launch test discovery in pytest #312

hidmic · 2019-08-16T15:22:10Z

Closes #237. This pull request allows for launch tests discovery and execution via pytest. It also sets up the scene for downstream packages i.e. launch_testing_ros to customize this.

Note this is not a tight integration between launch_testing and pytest but simply a mechanism for the latter to be able to execute launch tests.

Depends on #308.

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-20T17:57:26Z

CI up to launch_testing, launch_testing_ros and test_communication for extra safety:

Linux
Linux-aarch64
macOS
Windows

ivanpauno

I left some minimal comments.
I surfed in pytest plugin documentation and I don't fully understand the patch. But I don't have much more to add if it's working.

ivanpauno · 2019-08-21T12:55:56Z

launch_testing/launch_testing/pytest/hooks.py

+def find_launch_test_entrypoint(path):
+    try:
+        return getattr(path.pyimport(), 'generate_test_description', None)
+    except SyntaxError:


I'm not sure if that's the only possible exception, I would catch everything.

A SyntaxError is the only error I'm expecting. I'd rather not catch (and hide) unexpected errors e.g. some exception as a result of code being executed at the module level.

This is old code, but just as reference:
https://github.com/ros2/launch_ros/blob/bf3d800a53ea53d9465da9298c104998a7461dcf/ros2launch/ros2launch/command/launch.py#L125-L141

Couldn't value error happen here too?

launch_testing/test/launch_testing/examples/README.md

launch_testing/test/launch_testing/test_examples.py

ivanpauno · 2019-08-21T13:43:23Z

Extra comment, instead of rostest files needing a pytest mark and launchtest files just needing a generate_test_description function in the file, why not requiring a mark in them too? Something like, @pytest.mark.launchtest.
I prefer the difference being more explicit (than one explicit and the other implicit).

Also, aren't files with @pytest.mark.rostest mark being collected by both hooks now?

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

…unch_testing plugin. Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-21T18:21:25Z

Extra comment, instead of rostest files needing a pytest mark and launchtest files just needing a generate_test_description function in the file, why not requiring a mark in them too? Something like, @pytest.mark.launchtest.
I prefer the difference being more explicit (than one explicit and the other implicit).

Right now, any module in the search path that has a generate_test_description() is taken as a launch test and thus any mark was redundant in the pure launch test case. But I see the value in being explicit in both cases. Check 4491759.

Also, aren't files with @pytest.mark.rostest mark being collected by both hooks now?

Nope, this hook collects them appropriately once.

hidmic · 2019-08-21T18:22:20Z

I had to fix other failing tests, due to examples not being installed and pytest compulsively re-writing asserts within launch_testing. @ivanpauno PTAL.

ivanpauno

LGTM with green CI

ivanpauno · 2019-08-21T20:34:42Z

launch_testing/launch_testing/parametrize.py

-        def __iter__(self):
-            return iter(self.__calls)
+                yield partial, partial_args
+        setattr(_wrapped, '__parametrized__', True)


Couldn't be:

_wrapped.__parametrized__ = True

?

Yes, it can be. Done in 0705777.

ivanpauno · 2019-08-21T20:48:23Z

launch_testing/launch_testing/asserts/assert_exit_codes.py

+"""
+A module providing exit code assertions.
+
+PYTEST_DONT_REWRITE


Maybe, add a comment/link explaining why this is needed (e.g.: https://docs.pytest.org/en/latest/assert.html#assertion-introspection-details).

Sure, done in c7e6f80.

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-08-23T16:24:16Z

Alright, all green! Going in.

launch_testing/setup.py

hidmic mentioned this pull request Aug 16, 2019

Support launch_ros test runner in pytest ros2/launch_ros#54

Merged

hidmic added 3 commits August 20, 2019 14:53

Enable launch test discovery in pytest.

c177332

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Test examples using pytest.

67d53fe

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Enable downstream customization of launch tests execution.

94922bc

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic force-pushed the hidmic/launch-tests-in-python branch from 1af785b to 94922bc Compare August 20, 2019 17:54

hidmic requested review from dirk-thomas and ivanpauno August 20, 2019 17:57

hidmic mentioned this pull request Aug 20, 2019

Unit tests for composition actions ros2/launch_ros#15

Open

ivanpauno reviewed Aug 21, 2019

View reviewed changes

hidmic added 7 commits August 21, 2019 12:29

Move launch testing examples README content to the root README.

8014b4c

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Mark launch tests explicitly.

4491759

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Refactor parameterization support to play nice with pytest markers.

5be3345

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Improve pytest output on launch test failure.

e7cc2b9

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Please flake8

b658591

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Fix failing launch_testing tests.

d15a47c

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Prevent pytest from performing any assertion rewriting outside the la…

b4b56a2

…unch_testing plugin. Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

ivanpauno approved these changes Aug 21, 2019

View reviewed changes

hidmic added 2 commits August 22, 2019 15:14

Change explicit setattr for regular assignment.

0705777

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Better document PYTEST_DONT_REWRITE docstring.

c7e6f80

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

ivanpauno approved these changes Aug 22, 2019

View reviewed changes

Please flake8

dfa47c4

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic merged commit 5d6a87b into master Aug 23, 2019

delete-merged-branch bot deleted the hidmic/launch-tests-in-python branch August 23, 2019 16:24

dirk-thomas reviewed Aug 23, 2019

View reviewed changes

launch_testing/setup.py Show resolved Hide resolved

hidmic mentioned this pull request Aug 26, 2019

Update launch_testing example name in launch_testing_ament_cmake #320

Merged

hidmic mentioned this pull request Nov 19, 2019

Release Testing Automation - Iteration #1 ros2/ros2#739

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable launch test discovery in pytest #312

Enable launch test discovery in pytest #312

hidmic commented Aug 16, 2019 •

edited

Loading

hidmic commented Aug 20, 2019 •

edited

Loading

ivanpauno left a comment

ivanpauno Aug 21, 2019

hidmic Aug 21, 2019

ivanpauno Aug 21, 2019

ivanpauno commented Aug 21, 2019

hidmic commented Aug 21, 2019

hidmic commented Aug 21, 2019

ivanpauno left a comment

ivanpauno Aug 21, 2019

hidmic Aug 22, 2019

ivanpauno Aug 21, 2019

hidmic Aug 22, 2019

hidmic commented Aug 23, 2019

Enable launch test discovery in pytest #312

Enable launch test discovery in pytest #312

Conversation

hidmic commented Aug 16, 2019 • edited Loading

hidmic commented Aug 20, 2019 • edited Loading

ivanpauno left a comment

Choose a reason for hiding this comment

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

hidmic Aug 21, 2019

Choose a reason for hiding this comment

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

ivanpauno commented Aug 21, 2019

hidmic commented Aug 21, 2019

hidmic commented Aug 21, 2019

ivanpauno left a comment

Choose a reason for hiding this comment

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

hidmic Aug 22, 2019

Choose a reason for hiding this comment

ivanpauno Aug 21, 2019

Choose a reason for hiding this comment

hidmic Aug 22, 2019

Choose a reason for hiding this comment

hidmic commented Aug 23, 2019

hidmic commented Aug 16, 2019 •

edited

Loading

hidmic commented Aug 20, 2019 •

edited

Loading