Increase in test failures on windows debug with actions tests

[sloretz]

Windows debug has a lot of failing tests, but suddenly there are a few more in https://ci.ros2.org/view/nightly/job/nightly_win_deb/1174/#showFailuresLink

New failures

projectroot.test_action_client_server__Fibonacci__rclcpp__rmw_connext_cpp
projectroot.test_action_client_server__Fibonacci__rclcpp__rmw_fastrtps_cpp
projectroot.test_action_client_server__NestedMessage__rclcpp__rmw_connext_cpp
projectroot.test_action_client_server__NestedMessage__rclcpp__rmw_fastrtps_cpp
projectroot.test_client_scope_consistency_cpp__rmw_connext_cpp
projectroot.test_client_scope_consistency_cpp__rmw_fastrtps_cpp
projectroot.test_client_scope_cpp__rmw_connext_cpp
projectroot.test_client_scope_cpp__rmw_fastrtps_cpp
projectroot.test_parameter_server_cpp__rmw_connext_cpp
projectroot.test_parameter_server_cpp__rmw_fastrtps_cpp
projectroot.test_services_cpp__rmw_connext_cpp
projectroot.test_services_cpp__rmw_fastrtps_cpp
test_communication.build.test_communication.test_action_client_server__Fibonacci__rclcpp__rmw_connext_cpp_Debug.test_action_client_server
test_communication.build.test_communication.test_action_client_server__Fibonacci__rclcpp__rmw_fastrtps_cpp_Debug.test_action_client_server
test_communication.build.test_communication.test_action_client_server__NestedMessage__rclcpp__rmw_connext_cpp_Debug.test_action_client_server
test_communication.build.test_communication.test_action_client_server__NestedMessage__rclcpp__rmw_fastrtps_cpp_Debug.test_action_client_server
test_rclcpp.build.test_rclcpp.test_client_scope_consistency_cpp__rmw_connext_cpp_Debug.test_client_scope_consistency_cpp
test_rclcpp.build.test_rclcpp.test_client_scope_consistency_cpp__rmw_fastrtps_cpp_Debug.test_client_scope_consistency_cpp
test_rclcpp.build.test_rclcpp.test_client_scope_cpp__rmw_connext_cpp_Debug.test_client_scope_cpp
test_rclcpp.build.test_rclcpp.test_client_scope_cpp__rmw_fastrtps_cpp_Debug.test_client_scope_cpp
test_rclcpp.build.test_rclcpp.test_parameter_server_cpp__rmw_connext_cpp_Debug.test_parameter_server_cpp
test_rclcpp.build.test_rclcpp.test_parameter_server_cpp__rmw_fastrtps_cpp_Debug.test_parameter_server_cpp
test_rclcpp.build.test_rclcpp.test_services_cpp__rmw_connext_cpp_Debug.test_services_cpp
test_rclcpp.build.test_rclcpp.test_services_cpp__rmw_fastrtps_cpp_Debug.test_services_cpp

[sloretz]

Not sure what the cause is. First guess is the legacy launch API removal.

CI windows debug with lastest code on all repos testing just test_communication and test_rclcpp
(23 test failures)

CI windows debug excluding:

• Drops legacy launch API usage. ros1_bridge#163
• Drops legacy launch API usage. demos#311
• Drops legacy launch API usage. demos#311
• Drops legacy launch API usage. rcl#387
• Split test_token to avoid compiler note rcl#392
• Drops legacy launch API usage. rcutils#134
• Drops legacy launch API usage. system_tests#328
• Migrates legacy launch API tests launch#167
  (47 test failures)

[sloretz]

Looks like there are 24 more test failures on windows debug after the removal of the launch legacy API. Since the test falures are test_communication it seems like ros2/system_tests#328 might be the cause.

@hidmic may I ask you to investigate if these new test failures?

[hidmic]

I'll take a look, though I'd bet that it's not the launch API migration itself that broke the tests, but the fact that the legacy launch API based tests did not check process exit codes on Windows. So if it was failing before, we couldn't have noticed.

I'm amazed though, I ran CI many many times with those PRs.

[sloretz]

I'm amazed though, I ran CI many many times with those PRs.

It only fails on windows with CMAKE_BUILD_TYPE=Debug

[hidmic]

@sloretz as I thought, the executables built for debug on Windows don't even run but fail in various ways depending on the RMW_IMPLEMENTATION chosen. What changed after the launch API migration is that now launch tests fail when the launched executable tests exit with a non-zero returncode.

BTW I get a lot of test failures on my local setup, and many have nothing to do with launch. Is this normal?

[sloretz]

BTW I get a lot of test failures on my local setup, and many have nothing to do with launch. Is this normal?

Unfortunately yeah. There are a lot of failures on windows debug.

[dirk-thomas]

Seems to be resolved after #176 and ros2/rclcpp#682.

---

Resolved during #168.