2018-06-14 ci.ros2.org failures

[clalancette]

Last night, there were quite a few failures on the build farm, including some builds that failed outright. Since there are some old problems that are still around, and I need to get a handle on the situation, this ticket is an attempt to collect the problems all in one place since I'll need help to getting it back into some semblance of order. The below list is in rough priority order. @ros2/team

• OSX builds failed building rviz_default_plugins (https://ci.ros2.org/view/nightly/job/nightly_osx_ospl_release/16/ is one example). @wjwwood , thoughts? (maybe Fix failing tests rviz#299 fixes?)
• Launch/launch_testing failed on Linux Xenial (https://ci.ros2.org/view/nightly/job/nightly_xenial_linux_release/9): tracking ticket at Typing annotations of variables used in launch only available for python 3.6+ launch#78
• rcl tests segfaulting on OSX (https://ci.ros2.org/view/nightly/job/nightly_osx_release/890) - Presumably fixed by Allocate during initialization. osrf/osrf_testing_tools_cpp#10, needs more testing.
• Windows debug failed tests for test_params_yaml (https://ci.ros2.org/view/nightly/job/nightly_win_deb/911) @sloretz @mikaelarguedas (Use debug python executable on windows system_tests#281 in review)
• Windows debug failed tests for cli_remapping (https://ci.ros2.org/view/nightly/job/nightly_win_deb/911) @sloretz @mikaelarguedas (Use debug python executable on windows system_tests#281 in review)
• Linux launch tests seem to fail on repeated tests (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/449) @wjwwood
• Linux Opensplice release failed memory tools test (Make sure that my_first_function does not get optimized away. osrf/osrf_testing_tools_cpp#8 was thought to have fixed it, but did not) (https://ci.ros2.org/view/nightly/job/nightly_linux_ospl_release/15/) @clalancette - Should be fixed by Make sure my_first_function does not get optimized away. osrf/osrf_testing_tools_cpp#9
• Nightly Win Debug had a build warning on rviz_default_plugins (https://ci.ros2.org/view/nightly/job/nightly_win_deb/912/) - @wjwwood
• All linux packaging jobs failed trying to build the ros1_bridge, with No package 'builtin_interfaces' found (https://ci.ros2.org/view/packaging/job/packaging_linux/1107/): tracking ticket at Packaging jobs failing to find 'builtin_interfaces' ros1_bridge#120
• Nightly Linux aarch64 Opensplice started failing test_serialize, with bad buffer_length (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_ospl_release/19/) only test serialization on fastrtps and connext system_tests#284
• Nightly macOS/linux jobs failed with a colcon command-line error (https://ci.ros2.org/view/nightly/job/nightly_osx_ospl_release/20/console) - @dirk-thomas - rename build-base to test-result-base ci#196
  * [ ] Nightly Windows opensplice release failed a few different tests (https://ci.ros2.org/view/nightly/job/nightly_win_ospl_rel/19) duplicate of test_serialize above
• Nightly Windows release job failed timeout_subscriber test for rmw_connext_cpp (https://ci.ros2.org/view/nightly/job/nightly_win_rel/850/)

Not new:

• Linux composition tests are more flaky (known to be flaky since last build-cop handoff) (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/449)
• Linux Opensplice jobs are flaky for test_publisher_subscriber_{Builtins,Primitives,Empty} - (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_ospl_release/15/)
• Linux Opensplice jobs fails pendulum tests - not new, but continuing problem (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_ospl_release/15/)
• Nightly Windows rep failed rclpy test_timer tests (known to be flaky, still there) (https://ci.ros2.org/view/nightly/job/nightly_win_rep/1110/)
• Linux aarch64 Fast-RTPS get_node_names failed (https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/449)

[mikaelarguedas]

• RCL tests have been segfaulting on MacOS for 3 days: https://ci.ros2.org/view/nightly/job/nightly_osx_release/890/ @wjwwood this seem to have appeared when orsf_testing_tools has been introduced

• Fast-RTPS <=> Opensplice don't talk in debug mode on Windows: https://ci.ros2.org/job/ci_windows/4721

• launch / launch_testing tests crash on Xenial because of an invalid syntax error: https://ci.ros2.org/view/nightly/job/nightly_xenial_linux_release/9

E     File "/home/rosbuild/ci_scripts/ws/install/launch/lib/python3.5/site-packages/launch/launch_context.py", line 45
E       self._event_queue: asyncio.Queue = asyncio.Queue()
E                        ^
E   SyntaxError: invalid syntax

[Martin-Idel-SI]

rviz_default_plugins OSX failures should indeed be fixed by ros2/rviz#299 . That happened with merging a few branches at once...

[clalancette]

@wjwwood , @ros2/team , the situation in CI changed last night, but there are still a large number of failing tests and some failing builds. Everything in the original comment needs to be looked into, with a much lower priority to the "Not New" section. Tagging people in a separate comment here because I'm not sure an edit to a comment sends email.

[mikaelarguedas]

There's a new warning on windows

Coming from rviz_default_plugins, @wjwwood @Martin-Idel-SI FYI

[dirk-thomas]

• Failing packaging jobs on Linux, likely due to *_raw function rclcpp#388

[dhood]

failing turtlebot demo also fixed by ros2/realtime_support#64

[clalancette]

@ros2/team @dirk-thomas , some new failures over the weekend.

[clalancette]

@ros2/team It looks like most of these failures are now fixed, so thanks to everyone for that! I'm going to open up individual tickets for the remaining failures.