Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buildfarmer log 2024/03/25 - 2024/04/01 #36

Closed
github-actions bot opened this issue Mar 25, 2024 · 13 comments
Closed

Buildfarmer log 2024/03/25 - 2024/04/01 #36

github-actions bot opened this issue Mar 25, 2024 · 13 comments
Assignees
Labels
buildfarmer-log Log of issues in the buildfarm

Comments

@github-actions
Copy link
Contributor

Previous log #34

@github-actions github-actions bot added the buildfarmer-log Log of issues in the buildfarm label Mar 25, 2024
@Blast545 Blast545 self-assigned this Mar 25, 2024
@Blast545
Copy link
Contributor

Monday

Checking buildfarm healthiness
Overall looks OK, gazebo farm is getting higher demand on a specific node tagged for debbuilders given that debbuilders in noble are processed now as well, checking with Gazebo team if this is expected (I think it is)

image

Stopped hanging job, 3 days and 9 hours: https://ci.ros2.org/job/nightly_win_rel/2992/
Parallel executor does not give us relevant info about the hanged job in the logs :/ it seems it only reports once the test suite ends.

@Blast545
Copy link
Contributor

[INVESTIGATION]

Linter errors related to point_cloud_transport:
Reference builds: https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_debug/2715/
https://build.ros2.org/view/Rci/job/Rci__nightly-release_ubuntu_jammy_amd64/783/

Reported to the ROS 2 team to check if someone is looking at it, should be an easy fix. Not sure if it's related to flake8 transitions.

@Blast545
Copy link
Contributor

Blast545 commented Mar 25, 2024

[INVESTIGATION]
Internal tools failing, see: https://github.com/osrf/buildfarm-tools/actions/runs/8401493877/job/23009932079

Of course, Cristobal is on PTO so some things gotta break 😆 Taking a look to the problem, it seems it's related to an annotation not being recognized by the script for some reason.

Ugh, is not related to Python, is related to Ruby.
This ruby script:

tr_flakiness = %x{./sql_run.sh calculate_flakiness_jobs.sql "#{tr['error_name']}" "15 days"}
to be specific.
This error broke it: point_cloud_transport.flake8.E261 (./doc/conf.py:37:21)
I think the script doesn't like the spaces, probably requires modifying: https://github.com/osrf/buildfarm-tools-private/blob/c69a4256413713d9bc7faf8e00d972cabe504738/databaseFetcher.py#L89

Debugging:
puts %x{echo calculate_flakiness_jobs.sql '#{tr['error_name']}' "15 days"}
And this prints
calculate_flakiness_jobs.sql point_cloud_transport.flake8.E261 (.\doc

Copy link
Contributor Author

Mar 26

Build regressions
nightly_win_deb|3044|2024-03-26 04:00:01.665000|Number of failed test exceeds the threshold|test
Ici__nightly-performance_ubuntu_jammy_amd64|91|2024-03-26 00:38:27.926000|Packages performance_test Failed|build workspace in isolation and install
gz_sensors-7-win|34|2024-03-25 18:06:23.343000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-sensors7
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_win_rep
- Failing 93.33% in 14 out of 15 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 81.25% in 13 out of 16 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors
	rclcpp.TestExecutorNotifyWaitable.wait

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 81.25% in 13 out of 16 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)

Issue #4 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 81.25% in 13 out of 16 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #5 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 81.25% in 13 out of 16 builds
	projectroot.test.rclcpp.test_executor_notify_waitable
	  (also in: "nightly_win_rep" job)

Issue #6 in nightly_linux_humble_coverage
- Failing 92.86% in 13 out of 14 builds
	launch_ros.test.test_flake8.test_flake8
	  (also in: "nightly_linux_iron_coverage" job)
	  (also in: "nightly_linux_humble_coverage" job)

Issue #7 in nightly_linux_iron_coverage
- Failing 92.86% in 13 out of 14 builds
	launch.test.test_flake8.test_flake8

Issue #8 in nightly_win_rel
- Failing 86.67% in 13 out of 15 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #9 in nightly_win_rep
- Failing 60.0% in 9 out of 15 builds
	test_rclcpp.test_subscription.subscription_and_spinning
	  (also in: "nightly_win_deb" job)

Issue #10 in nightly_win_rep
- Failing 46.67% in 7 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	test_rclcpp.TestNNodes.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp

Issue #11 in nightly_win_rep
- Failing 33.33% in 5 out of 15 builds
	rclpy.rclpy.test.test_timer.test_zero_callback[0.1]

Issue #12 in nightly_linux_iron_coverage
- Failing 28.57% in 4 out of 14 builds
	test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #13 in nightly_win_rep
- Failing 26.67% in 4 out of 15 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #14 in nightly_linux_iron_coverage
- Failing 28.57% in 4 out of 14 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_iron_coverage" job)
	  (also in: "nightly_linux_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #15 in Rci__nightly-fastrtps_ubuntu_jammy_amd64
- Failing 20.0% in 3 out of 15 builds
	projectroot.flake8
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-release_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-fastrtps_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-debug_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #16 in Rci__nightly-fastrtps_ubuntu_jammy_amd64
- Failing 20.0% in 3 out of 15 builds
	point_cloud_transport.flake8.Q000 (./doc/conf.py:65:5)
	point_cloud_transport.flake8.Q000 (./doc/conf.py:38:21)
	point_cloud_transport.flake8.I100 (./doc/conf.py:36:1)
	point_cloud_transport.flake8.E402 (./doc/conf.py:36:1)
	point_cloud_transport.flake8.E261 (./doc/conf.py:37:21)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)
	  (also in: "Rci__nightly-release_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-fastrtps_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-debug_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #17 in gz_rendering-7-win
- Failing 100.0% in 3 out of 3 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #18 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #19 in gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #20 in gz_sensors-ci-ign-sensors6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	CameraSensor/TriggeredCameraTest.ImagesWithBuiltinSDF/ogre

Issue #21 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 28.57% in 2 out of 7 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates

Issue #22 in gz_sim-ci-ign-gazebo3-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	ServerRepeat/AckermannSteeringTest.TfPublishes/0 (known issue in: https://github.com/gazebosim/gz-sim/issues/1993 - CLOSED)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/1993 - CLOSED)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/1993 - CLOSED)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/1993 - CLOSED)

Issue #23 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	ShaderParamTest.ShaderParam

Issue #24 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)

Issue #25 in gz_sim-ci-gz-sim8-homebrew-amd64
- Failing 40.0% in 2 out of 5 builds
	TriggeredPublisherTest.InputMessagesTriggerServices
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-main-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #26 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #27 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_multiple_servers.test_ran

Issue #28 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.src.core.ddsc.tests.CUnit_ddsc_discstress_create_writer

Issue #29 in nightly_linux_humble_coverage
- Failing 14.29% in 2 out of 14 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #30 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 14.29% in 1 out of 7 builds
	ThrusterTest.VelocityControl
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)

Issue #31 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent

Issue #32 in gz_transport-ign-transport8-win
- Failing 25.0% in 1 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-ign-transport11-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #33 in nightly_linux-rhel_debug
- Failing 6.67% in 1 out of 15 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #34 in nightly_linux-rhel_debug
- Failing 6.67% in 1 out of 15 builds
	launch_testing_examples.launch_testing_examples.check_multiple_nodes_launch_test.launch_testing_examples.check_multiple_nodes_launch_test (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux_release" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)

Issue #35 in nightly_linux_coverage
- Failing 7.14% in 1 out of 14 builds
	ros2action.ros2action.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

@claraberendsen
Copy link
Contributor

claraberendsen commented Mar 26, 2024

🔴 Build regressions 26th March 🔴

nightly_win_deb (#3044)

Some PRs were merged yesterday that broke it. Chris is addressing it today.
The combination of these PRs should fix the issue:

Ici__nightly-performance_ubuntu_jammy_amd64 (#91)

Seems like an issue with Fastdds it has been failing since build #83 on Feb 16th.

/tmp/ws2/build_isolated/performance_test/src/idlgen/fast_rtps/gen/fast_rtps/Header_PubSubTypes.cxx:23:10: fatal error: fastdds/rtps/common/CdrSerialization.hpp: No such file or directory

Might need the fix implemented for humble, Chris will take a look at it.

Copy link
Contributor Author

Mar 27

Build regressions
nightly_win_deb|3045|2024-03-27 04:00:01.204000|Number of failed test exceeds the threshold|test
nightly_win_rep|3306|2024-03-27 04:30:15.748000|Number of failed test exceeds the threshold|test
Rci__benchmark_ubuntu_jammy_amd64|652|2024-03-26 14:03:13.175000|Packages rosidl_typesupport_fastrtps_cpp Failed|build workspace in isolation and install
Rci__nightly-release_ubuntu_jammy_amd64|785|2024-03-26 13:00:41.911000|Unknown Reason|Enumerating packages needed to build
gz_cmake-ign-cmake2-win|30|2024-03-26 22:06:43.125000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for ignition-cmake2
gz_sim-ci-ign-gazebo3-focal-amd64|30|2024-03-26 19:11:08.524000|Jenkins channel closing down (Agent disconnected)|compiling
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in nightly_linux_humble_coverage
- Failing 100.0% in 14 out of 14 builds
	launch_ros.test.test_flake8.test_flake8
	  (also in: "nightly_linux_iron_coverage" job)
	  (also in: "nightly_linux_humble_coverage" job)

Issue #2 in nightly_linux_iron_coverage
- Failing 100.0% in 14 out of 14 builds
	launch.test.test_flake8.test_flake8

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 86.67% in 13 out of 15 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors
	rclcpp.TestExecutorNotifyWaitable.wait

Issue #4 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 86.67% in 13 out of 15 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)

Issue #5 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 86.67% in 13 out of 15 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #6 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 86.67% in 13 out of 15 builds
	projectroot.test.rclcpp.test_executor_notify_waitable
	  (also in: "nightly_win_rep" job)

Issue #7 in nightly_win_rel
- Failing 86.67% in 13 out of 15 builds
	launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #8 in nightly_linux-aarch64_repeated
- Failing 31.25% in 5 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #9 in nightly_linux-aarch64_repeated
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #10 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #11 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	test_rclcpp.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp.xunit.missing_result
	projectroot.test_node_name__rmw_cyclonedds_cpp__rmw_fastrtps_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #12 in gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #13 in gz_sim-ci-main-homebrew-amd64
- Failing 100.0% in 3 out of 3 builds
	TriggeredPublisherTest.InputMessagesTriggerServices
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-main-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)

Issue #14 in gz_sensors-ci-ign-sensors6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	CameraSensor/TriggeredCameraTest.ImagesWithBuiltinSDF/ogre

Issue #15 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 28.57% in 2 out of 7 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates

Issue #16 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	ShaderParamTest.ShaderParam

Issue #17 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #18 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #19 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/SingleThreadedExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)

Issue #20 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #21 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	ros2doctor.ros2doctor.test.test_qos_compatibility.test_qos_compatibility

Issue #22 in nightly_linux_repeated
- Failing 13.33% in 2 out of 15 builds
	ros2action.ros2action.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_coverage" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #23 in gz_physics-ci-main-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DetachableJointTest/0.CorrectAttachmentPoints

Issue #24 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 14.29% in 1 out of 7 builds
	ThrusterTest.VelocityControl
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-homebrew-amd64" job)
	  (also in: "gz_sim-ci-gz-sim7-homebrew-amd64" job)

Issue #25 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent

Issue #26 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-main-win" job)
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #27 in gz_transport-ign-transport8-win
- Failing 25.0% in 1 out of 4 builds
	playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-ign-transport11-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)
	  (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN)

Issue #28 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

@Blast545
Copy link
Contributor

Blast545 commented Mar 27, 2024

WEDNESDAY

SERVERS and CAPACITY OK

Odd build regression on Rci_nightly_Release yesterday: https://build.ros2.org/view/Rci/job/Rci__nightly-release_ubuntu_jammy_amd64/785/ -> Keeping track to see if it fails again

Odd build regression on doc_rosindex yesterday: https://build.ros.org/job/doc_rosindex/1937/ -> Keeping track to see if it fails again

Chris fixing Windows_debug: https://osrf.slack.com/archives/C04R65U5CNA/p1711542127941529 (Keeping track)

@Blast545
Copy link
Contributor

Blast545 commented Mar 27, 2024

INVESTIGATION Flake8 errors on coverage jobs, older distros

Reference jobs: https://ci.ros2.org/job/nightly_linux_iron_coverage/378/
https://ci.ros2.org/job/nightly_linux_humble_coverage/751/

Probably should be solved with backports, raised to ROS 2 team on: https://osrf.slack.com/archives/C04R65U5CNA/p1711550109739249

@Blast545
Copy link
Contributor

Keeping track: https://build.osrfoundation.org/job/gz_rendering-7-win/ has never been green?

Copy link
Contributor Author

Mar 28

Build regressions
nightly_linux_iron_coverage|379|2024-03-28 05:57:56.204000|Packages test_rclcpp  Failed|build
packaging_linux|3382|2024-03-28 06:56:11.635000|Early fail|No last section
packaging_linux-rhel|1878|2024-03-28 06:56:32.249000|Early fail|No last section
nightly_win_rep|3307|2024-03-28 04:00:01.485000|Number of failed test exceeds the threshold|test
Rci__nightly-performance_ubuntu_jammy_amd64|664|2024-03-27 18:30:44.756000|Packages performance_test Failed|build workspace in isolation and install
Rci__nightly-release_ubuntu_jammy_amd64|786|2024-03-27 17:49:13.008000|Build timed out after 300 minutes|run tests
gz_cmake-main-win|36|2024-03-28 01:06:38.008000|Jenkins channel closing down (Agent disconnected)|vcpkg: install all dependencies
gz_sensors-ci-gz-sensors8-jammy-amd64|63|2024-03-28 08:06:25.396000|Unknown Reason|NVIDIA MISMATCH RECOVERY
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 93.33% in 14 out of 15 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors
	rclcpp.TestExecutorNotifyWaitable.wait

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 93.33% in 14 out of 15 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 93.33% in 14 out of 15 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #4 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 93.33% in 14 out of 15 builds
	projectroot.test.rclcpp.test_executor_notify_waitable
	  (also in: "nightly_win_rep" job)

Issue #5 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	rclcpp.test_publisher_with_type_adapter.gtest.missing_result

Issue #6 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	projectroot.test.rclcpp.test_publisher_with_type_adapter
	  (also in: "nightly_win_rep" job)

Issue #7 in nightly_win_deb
- Failing 50.0% in 7 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy

Issue #8 in nightly_linux-aarch64_repeated
- Failing 37.5% in 6 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #9 in nightly_linux-aarch64_repeated
- Failing 37.5% in 6 out of 16 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #10 in nightly_win_deb
- Failing 35.71% in 5 out of 14 builds
	rviz_default_plugins.cpplint.xunit.missing_result

Issue #11 in nightly_win_deb
- Failing 35.71% in 5 out of 14 builds
	projectroot.cpplint (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)
	  (also in: "Rci__nightly-performance_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)
	  (also in: "Hci__nightly-performance_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)

Issue #12 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #13 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #14 in gz_fuel_tools-8-win
- Failing 75.0% in 3 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #15 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 37.5% in 3 out of 8 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #16 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[/home/jenkins-agent/workspace/nightly_linux-rhel_repeated/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]

Issue #17 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #18 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/SingleThreadedExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_linux_repeated" job)

Issue #19 in nightly_win_deb
- Failing 21.43% in 3 out of 14 builds
	test_rclcpp.test_subscription.subscription_and_spinning
	  (also in: "nightly_win_rep" job)

Issue #20 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 33.33% in 2 out of 6 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #21 in gz_sensors-ci-ign-sensors6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	CameraSensor/TriggeredCameraTest.ImagesWithBuiltinSDF/ogre

Issue #22 in gz_sim-8-win
- Failing 33.33% in 2 out of 6 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

Issue #23 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	ShaderParamTest.ShaderParam

Issue #24 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 50.0% in 2 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #25 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #26 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 9.09% in 1 out of 11 builds
	projectroot.benchmark_string_conversions

Issue #27 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 6.67% in 1 out of 15 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/SingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some
	rclcpp.TestExecutors/MultiThreadedExecutor.spinAll

Issue #28 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 6.67% in 1 out of 15 builds
	projectroot.test.rclcpp.test_executors
	  (also in: "nightly_win_rep" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #29 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED)
	UNIT_ShutdownButton_TEST.test_ran
	UNIT_Screenshot_TEST.test_ran
	UNIT_Publisher_TEST.test_ran

Issue #30 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DetachableJointTest/0.CorrectAttachmentPoints
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #31 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent

Issue #32 in gz_sim-main-win
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #33 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-main-win" job)
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #34 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[2-3]
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #35 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes
	  (also in: "nightly_win_rep" job)

Issue #36 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	rclcpp_action.TestServer.publish_status_succeeded
	rclcpp_action.TestServer.publish_status_canceling
	rclcpp_action.TestServer.publish_status_canceled
	rclcpp_action.TestServer.publish_status_accepted
	rclcpp_action.TestServer.publish_status_aborted
	rclcpp_action.TestServer.publish_feedback
	rclcpp_action.TestServer.handle_goal_called
	rclcpp_action.TestServer.handle_cancel_unknown_goal
	rclcpp_action.TestServer.handle_cancel_terminated_goal
	rclcpp_action.TestServer.handle_cancel_reject
	rclcpp_action.TestServer.handle_cancel_called
	rclcpp_action.TestServer.handle_accepted_called
	rclcpp_action.TestServer.get_result
	rclcpp_action.TestServer.construction_and_destruction_wait_set_error
	rclcpp_action.TestServer.construction_and_destruction_sub_node
	rclcpp_action.TestServer.construction_and_destruction_server_init_error
	rclcpp_action.TestServer.construction_and_destruction_callback_group
	rclcpp_action.TestServer.construction_and_destruction_after_node
	rclcpp_action.TestServer.construction_and_destruction
	  (also in: "nightly_win_rep" job)

Issue #37 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	rclcpp_action.TestServer.get_result_deferred
	rclcpp_action.TestServer.deferred_execution
	projectroot.test_server
	  (also in: "nightly_win_rep" job)

Issue #38 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rclcpp.cppcheck.xunit.missing_result
	projectroot.cppcheck

Copy link
Contributor Author

Mar 29

Build regressions
nightly_linux-aarch64_debug|2719|2024-03-29 04:00:01.219000|Number of failed test exceeds the threshold|ccache stats (after)
Ici__nightly-performance_ubuntu_jammy_amd64|92|2024-03-29 01:30:21.282000|Packages performance_test Failed|build workspace in isolation and install
Rci__nightly-performance_ubuntu_jammy_amd64|665|2024-03-28 18:25:41.681000|Packages performance_test Failed|build workspace in isolation and install
gz-fuel-tools10-debbuilder|217|2024-03-28 12:26:15.767000|Failed to compile and install build|install build dependencies from debian/control
gz-gui9-debbuilder|217|2024-03-28 12:56:24.544000|Failed to compile and install build|install build dependencies from debian/control
gz-launch8-debbuilder|208|2024-03-28 14:51:50.529000|Failed to compile and install build|install build dependencies from debian/control
gz-physics8-debbuilder|221|2024-03-28 13:51:36.044000|Failed to compile and install build|install build dependencies from debian/control
gz-rendering9-debbuilder|215|2024-03-28 11:45:23.467000|Failed to compile and install build|install build dependencies from debian/control
gz-sensors9-debbuilder|213|2024-03-28 13:10:11.247000|Failed to compile and install build|install build dependencies from debian/control
gz-sim9-debbuilder|215|2024-03-28 14:41:21.522000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1549|2024-03-28 22:12:29.826000|Failed to compile and install build|install build dependencies from debian/control
gz-tools2-debbuilder|1548|2024-03-28 22:12:21.609000|Unknown Reason|see Dockerfile
gz-tools2-debbuilder|1544|2024-03-28 21:55:10.391000|Failed to compile and install build|install build dependencies from debian/control
gz-transport14-debbuilder|218|2024-03-28 12:40:44.043000|Failed to compile and install build|install build dependencies from debian/control
sdformat15-debbuilder|216|2024-03-28 12:17:42.354000|Failed to compile and install build|install build dependencies from debian/control
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 16 out of 16 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 16 out of 16 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 16 out of 16 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #4 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	rclcpp.test_publisher_with_type_adapter.gtest.missing_result

Issue #5 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	projectroot.test.rclcpp.test_publisher_with_type_adapter
	  (also in: "nightly_win_rep" job)

Issue #6 in nightly_win_deb
- Failing 57.14% in 8 out of 14 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #7 in nightly_win_deb
- Failing 50.0% in 7 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy

Issue #8 in nightly_win_rep
- Failing 42.86% in 6 out of 14 builds
	test_rclcpp.test_timeout_subscriber.timeout_subscriber

Issue #9 in nightly_win_rep
- Failing 42.86% in 6 out of 14 builds
	projectroot.test_action_server

Issue #10 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #11 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 50.0% in 4 out of 8 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #12 in nightly_linux-rhel_repeated
- Failing 25.0% in 4 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #13 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node

Issue #14 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - OPEN)

Issue #15 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 18.75% in 3 out of 16 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some

Issue #16 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 18.75% in 3 out of 16 builds
	projectroot.test.rclcpp.test_executors
	  (also in: "nightly_win_rep" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #17 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #18 in gz_fuel_tools-8-win
- Failing 75.0% in 3 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #19 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 40.0% in 2 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #20 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 18.18% in 2 out of 11 builds
	projectroot.benchmark_string_conversions

Issue #21 in gz_sensors-ci-ign-sensors6-homebrew-amd64
- Failing 50.0% in 2 out of 4 builds
	CameraSensor/TriggeredCameraTest.ImagesWithBuiltinSDF/ogre

Issue #22 in gz_sim-8-win
- Failing 33.33% in 2 out of 6 builds
	INTEGRATION_mesh_inertia_calculation.test_ran

Issue #23 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	ShaderParamTest.ShaderParam

Issue #24 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #25 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #26 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED)

Issue #27 in nightly_win_rel
- Failing 13.33% in 2 out of 15 builds
	rclcpp.cppcheck.xunit.missing_result
	projectroot.cppcheck

Issue #28 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)

Issue #29 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED)
	UNIT_ShutdownButton_TEST.test_ran
	UNIT_Screenshot_TEST.test_ran
	UNIT_Publisher_TEST.test_ran

Issue #30 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DetachableJointTest/0.CorrectAttachmentPoints
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #31 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 12.5% in 1 out of 8 builds
	INTEGRATION_sensors_system.test_ran

Issue #32 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	ThermalTest.ThermalSensorSystem
	ThermalTest.TemperatureComponent

Issue #33 in gz_sim-ign-gazebo6-win
- Failing 25.0% in 1 out of 4 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-main-win" job)

Issue #34 in nightly_linux-aarch64_release
- Failing 6.67% in 1 out of 15 builds
	test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_coverage" job)

Issue #35 in nightly_linux-aarch64_release
- Failing 6.67% in 1 out of 15 builds
	test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_0
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_3
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_2
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_1
	projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_0
	  (also in: "nightly_linux_coverage" job)

Issue #36 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testTimer2CancelledWithExecutorSpin

Issue #37 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #38 in nightly_linux-rhel_release
- Failing 6.25% in 1 out of 16 builds
	rosbag2_transport.RecordIntegrationTestFixture.published_messages_from_multiple_services_are_recorded

Issue #39 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.gtest_multithreaded__rmw_fastrtps_cpp.gtest.missing_result
	projectroot.gtest_multithreaded__rmw_fastrtps_cpp

Issue #40 in nightly_linux_iron_coverage
- Failing 7.69% in 1 out of 13 builds
	test_tracetools.test_tracetools.test.test_lifecycle_node.TestLifecycleNode.test_all
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #41 in nightly_linux_iron_coverage
- Failing 7.69% in 1 out of 13 builds
	projectroot.test_lifecycle_node
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #42 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	test_rclcpp.TestNNodes.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)

Issue #43 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[C:/ci/ws/src/ros2/system_tests/test_cli_remapping/test/name_maker.py]
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[C:/ci/ws/build/test_cli_remapping/Debug/name_maker_rclcpp.exe]

Issue #44 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #45 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.test_subscription.subscription_and_spinning
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #46 in nightly_linux_coverage
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.test_multithreaded.multi_access_publisher
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_coverage" job)

Issue #47 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	ros2cli.ros2cli.test.test_strategy.test_with_no_daemon_running[2-3]
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[3-3]
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testTimer1CancelledWithExecutorSpin
	rclcpp.PerTimerType/TestTimer.test_simple_cancel/generic_timer (4-byte object <01-00 00-00>)                     
	rcl_action.TestActionServer.test_action_server_is_valid
	projectroot.test_tutorial_content_filtering_publisher_content_filtering_subscriber__rmw_fastrtps_cpp

Issue #48 in nightly_linux-rhel_release
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_record_all__rmw_cyclonedds_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Copy link
Contributor Author

Mar 30

Build regressions
Hci__nightly-cyclonedds_ubuntu_jammy_amd64|234|2024-03-30 06:15:00.830000|Jenkins channel closing down (Agent disconnected)|build workspace in isolation and install
Rci__nightly-performance_ubuntu_jammy_amd64|666|2024-03-29 14:14:46.829000|Packages performance_test Failed|build workspace in isolation and install
gz-cmake4-debbuilder|227|2024-03-30 09:10:12.665000|Unknown Reason|see Dockerfile
gz-common6-debbuilder|217|2024-03-30 09:17:20.317000|Unknown Reason|see Dockerfile
gz-fuel-tools10-debbuilder|220|2024-03-30 09:20:05.116000|Unknown Reason|see Dockerfile
gz-gui9-debbuilder|220|2024-03-30 09:21:31.036000|Unknown Reason|see Dockerfile
gz-launch8-debbuilder|211|2024-03-30 09:24:15.430000|Unknown Reason|see Dockerfile
gz-math8-debbuilder|223|2024-03-30 09:15:56.138000|Unknown Reason|see Dockerfile
gz-msgs11-debbuilder|224|2024-03-30 09:18:00.520000|Unknown Reason|see Dockerfile
gz-physics8-debbuilder|224|2024-03-30 09:22:54.997000|Unknown Reason|see Dockerfile
gz-plugin3-debbuilder|218|2024-03-30 09:16:37.751000|Unknown Reason|see Dockerfile
gz-rendering9-debbuilder|218|2024-03-30 09:18:40.703000|Unknown Reason|see Dockerfile
gz-sensors9-debbuilder|216|2024-03-30 09:22:13.802000|Unknown Reason|see Dockerfile
gz-sim9-debbuilder|218|2024-03-30 09:23:36.557000|Unknown Reason|see Dockerfile
gz-tools2-debbuilder|1555|2024-03-30 09:14:29.575000|Unknown Reason|see Dockerfile
gz-transport14-debbuilder|221|2024-03-30 09:20:47.282000|Unknown Reason|see Dockerfile
gz-utils3-debbuilder|223|2024-03-30 09:15:15.686000|Unknown Reason|see Dockerfile
sdformat15-debbuilder|219|2024-03-30 09:19:23.056000|Unknown Reason|see Dockerfile
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #4 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	rclcpp.test_publisher_with_type_adapter.gtest.missing_result
	projectroot.test.rclcpp.test_publisher_with_type_adapter

Issue #5 in nightly_win_deb
- Failing 57.14% in 8 out of 14 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #6 in nightly_win_deb
- Failing 35.71% in 5 out of 14 builds
	rviz_default_plugins.cpplint.xunit.missing_result

Issue #7 in nightly_win_deb
- Failing 35.71% in 5 out of 14 builds
	projectroot.cpplint (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)
	  (also in: "Rci__nightly-performance_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)
	  (also in: "Hci__nightly-performance_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rosidl/issues/790 - OPEN)

Issue #8 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node

Issue #9 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
	projectroot.test_multi_nested_message_introspection
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #10 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 57.14% in 4 out of 7 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #11 in nightly_linux-rhel_repeated
- Failing 25.0% in 4 out of 16 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]

Issue #12 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - OPEN)

Issue #13 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 27.27% in 3 out of 11 builds
	projectroot.benchmark_string_conversions

Issue #14 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #15 in gz_fuel_tools-8-win
- Failing 75.0% in 3 out of 4 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #16 in gz_rendering-7-win
- Failing 100.0% in 3 out of 3 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #17 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)
	  (also in: "gz_sim-ci-ign-gazebo6-focal-amd64" job)

Issue #18 in nightly_linux_repeated
- Failing 20.0% in 3 out of 15 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/SingleThreadedExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #19 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 20.0% in 3 out of 15 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #20 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	projectroot.test_cli_remapping
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #21 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 20.0% in 3 out of 15 builds
	projectroot.test.rclcpp.test_executors
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #22 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 40.0% in 2 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #23 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #24 in nightly_win_deb
- Failing 14.29% in 2 out of 14 builds
	rclcpp_lifecycle.test_lifecycle_service_client.gtest.missing_result

Issue #25 in nightly_win_rel
- Failing 13.33% in 2 out of 15 builds
	test_rclcpp.test_subscription.subscription_and_spinning
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #26 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.detachUnattached
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callbacks
	rclcpp.TestTimeSource.callback_handler_erasure
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_wait_for_message
	projectroot.test.rclcpp.test_time_source
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits

Issue #27 in nightly_linux-rhel_repeated
- Failing 12.5% in 2 out of 16 builds
	projectroot.gtest_multithreaded__rmw_fastrtps_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #28 in Rci__nightly-release_ubuntu_jammy_amd64
- Failing 7.69% in 1 out of 13 builds
	test_rclcpp.test_time.timer_fire_regularly
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	rclcpp.TestTimerCancelBehavior/StaticSingleThreadedExecutor.testBackTimerCancelThenResetBehavior
	projectroot.gtest_timer__rmw_fastrtps_cpp
	projectroot.gtest_timer__rmw_cyclonedds_cpp

Issue #29 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED)
	UNIT_ShutdownButton_TEST.test_ran
	UNIT_Screenshot_TEST.test_ran
	UNIT_Publisher_TEST.test_ran

Issue #30 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 25.0% in 1 out of 4 builds
	DetachableJointTest/0.CorrectAttachmentPoints
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #31 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 14.29% in 1 out of 7 builds
	INTEGRATION_sensors_system.test_ran

Issue #32 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #33 in gz_sim-ign-gazebo6-win
- Failing 33.33% in 1 out of 3 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-main-win" job)

Issue #34 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testBothTimerCancelThenResetT2Behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #35 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	projectroot.test_record__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #36 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	rclcpp.TestAddCallbackGroupsToExecutorStable/MultiThreadedExecutor.subscriber_triggered_to_receive_message
	  (also in: "nightly_linux_repeated" job)

Issue #37 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_timer
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux_release" job)
	  (also in: "nightly_linux_iron_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "nightly_linux-aarch64_debug" job)

Issue #38 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_subscription
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #39 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rclpy.test_waitable.xunit.missing_result
	rclpy.test_timer.xunit.missing_result
	rclpy.test_time_source.xunit.missing_result
	rclpy.test_subscription.xunit.missing_result
	rclpy.test_service_introspection.xunit.missing_result
	rclpy.test_service.xunit.missing_result
	projectroot.test_waitable
	projectroot.test_time_source
	projectroot.test_service_introspection
	  (also in: "nightly_win_rep" job)

Issue #40 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_service
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)

Issue #41 in nightly_linux-rhel_repeated
- Failing 6.25% in 1 out of 16 builds
	test_rclcpp.test_multithreaded.multi_access_publisher
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_coverage" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #42 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spinAll
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #43 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_cli_remapping.TestCLIRemapping.test_node_name_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe]
	rclcpp.TimeSource.valid_clock_type_for_sim_time
	rclcpp.TimeSource.invalid_sim_time_parameter_override
	rclcpp.TimeSource.invalid_clock_type_for_sim_time
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestLoggerService.test_set_logging_level_with_partial_invalid_param
	rclcpp.TestLoggerService.test_set_logging_level_with_invalid_param
	rclcpp.TestLoggerService.test_set_and_get_one_logging_level
	rclcpp.TestLoggerService.test_set_and_get_multi_logging_level
	rclcpp.TestLoggerService.check_connect_set_logger_service
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/SingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/SingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/SingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/MultiThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/MultiThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/MultiThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/MultiThreadedExecutor.addNodeTwoExecutors
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_failures_with_exceptions/generic_timer (4-byte object <01-00 00-00>)                     

Issue #44 in nightly_win_rel
- Failing 6.67% in 1 out of 15 builds
	rclpy.test_wait_for_message.xunit.missing_result
	projectroot.test_wait_for_message
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)

Issue #45 in Rci__nightly-release_ubuntu_jammy_amd64
- Failing 7.69% in 1 out of 13 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Rci__nightly-release_ubuntu_jammy_amd64" job)

Copy link
Contributor Author

Mar 31

Build regressions
packaging_linux|3385|2024-03-31 06:51:23.144000|Jenkins channel closing down (Agent disconnected)|build underlay packages
packaging_linux-rhel|1881|2024-03-31 07:27:33.269000|Jenkins channel closing down (Agent disconnected)|build underlay packages
Ici__nightly-cyclonedds_ubuntu_jammy_amd64|110|2024-03-31 06:15:00.930000|Unknown Reason|run tests
Rci__nightly-performance_ubuntu_jammy_amd64|667|2024-03-30 22:46:20.613000|Packages performance_test Failed|build workspace in isolation and install
gz-cmake4-debbuilder|229|2024-03-31 09:12:44.994000|Unknown Reason|see Dockerfile
gz-common6-debbuilder|219|2024-03-31 09:19:42.750000|Unknown Reason|see Dockerfile
gz-fuel-tools10-debbuilder|222|2024-03-31 09:22:36.547000|Unknown Reason|see Dockerfile
gz-gui9-debbuilder|222|2024-03-31 09:24:08.488000|Unknown Reason|see Dockerfile
gz-launch8-debbuilder|213|2024-03-31 09:27:08.368000|Unknown Reason|see Dockerfile
gz-math8-debbuilder|225|2024-03-31 09:18:10.912000|Unknown Reason|see Dockerfile
gz-msgs11-debbuilder|226|2024-03-31 09:20:24.410000|Unknown Reason|see Dockerfile
gz-physics8-debbuilder|226|2024-03-31 09:25:40.355000|Unknown Reason|see Dockerfile
gz-plugin3-debbuilder|220|2024-03-31 09:18:57.785000|Unknown Reason|see Dockerfile
gz-rendering9-debbuilder|220|2024-03-31 09:21:09.223000|Unknown Reason|see Dockerfile
gz-sensors9-debbuilder|218|2024-03-31 09:24:55.743000|Unknown Reason|see Dockerfile
gz-sim9-debbuilder|220|2024-03-31 09:26:23.462000|Unknown Reason|see Dockerfile
gz-tools2-debbuilder|1558|2024-03-31 09:16:41.426000|Unknown Reason|see Dockerfile
gz-transport14-debbuilder|223|2024-03-31 09:23:20.738000|Unknown Reason|see Dockerfile
gz-utils3-debbuilder|225|2024-03-31 09:17:27.501000|Unknown Reason|see Dockerfile
sdformat15-debbuilder|221|2024-03-31 09:21:54.323000|Unknown Reason|see Dockerfile
Test regressions
Excluding 'performance, connext, fastrtps-dynamic' jobs
Issue #1 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger
	rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger
	rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors

Issue #2 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	projectroot.test_lifecycle_service_client
	  (also in: "nightly_win_deb" job)

Issue #3 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 100.0% in 15 out of 15 builds
	projectroot.test_guard_condition
	  (also in: "nightly_win_rep" job)

Issue #4 in nightly_win_deb
- Failing 100.0% in 14 out of 14 builds
	rclcpp.test_publisher_with_type_adapter.gtest.missing_result
	projectroot.test.rclcpp.test_publisher_with_type_adapter

Issue #5 in nightly_win_deb
- Failing 50.0% in 7 out of 14 builds
	sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy

Issue #6 in nightly_win_rep
- Failing 50.0% in 7 out of 14 builds
	test_rclcpp.TestNNodesAfterShutdown.test_10_nodes
	projectroot.test_n_nodes__rmw_fastrtps_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #7 in nightly_win_rep
- Failing 42.86% in 6 out of 14 builds
	rclpy.test_node.xunit.missing_result
	projectroot.test_node

Issue #8 in nightly_linux-rhel_repeated
- Failing 31.25% in 5 out of 16 builds
	projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp
	  (also in: "nightly_linux_repeated" job)

Issue #9 in nightly_win_rep
- Failing 35.71% in 5 out of 14 builds
	projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - OPEN)

Issue #10 in Rci__benchmark_ubuntu_jammy_amd64
- Failing 36.36% in 4 out of 11 builds
	projectroot.benchmark_string_conversions

Issue #11 in gz_rendering-7-win
- Failing 100.0% in 4 out of 4 builds
	MeshTest.NormalMapWithoutTexCoord
	  (also in: "gz_rendering-main-win" job)
	  (also in: "gz_rendering-8-win" job)
	  (also in: "gz_rendering-7-win" job)

Issue #12 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 57.14% in 4 out of 7 builds
	ServerRepeat/SceneBroadcasterTest.StateStatic/0
	NetworkHandshake.Updates
	  (also in: "gz_sim-ci-main-jammy-amd64" job)
	  (also in: "gz_sim-ci-gz-sim8-jammy-amd64" job)

Issue #13 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 26.67% in 4 out of 15 builds
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #14 in nightly_win_rep
- Failing 28.57% in 4 out of 14 builds
	projectroot.test_wait

Issue #15 in nightly_linux-aarch64_repeated
- Failing 25.0% in 4 out of 16 builds
	projectroot.test.rclcpp.test_executors_timer_cancel_behavior
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "nightly_linux-aarch64_release" job)
	  (also in: "Rci__nightly-release_ubuntu_jammy_amd64" job)

Issue #16 in Rci__nightly-cyclonedds_ubuntu_jammy_amd64
- Failing 26.67% in 4 out of 15 builds
	projectroot.test.rclcpp.test_executors
	  (also in: "nightly_win_rep" job)
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)

Issue #17 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 75.0% in 3 out of 4 builds
	SdfGeneratorFixture.ModelWithNestedIncludes
	  (also in: "gz_sim-ign-gazebo6-win" job)
	  (also in: "gz_sim-ci-ign-gazebo6-homebrew-amd64" job)

Issue #18 in nightly_linux-rhel_repeated
- Failing 18.75% in 3 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_debug" job)
	  (also in: "nightly_linux-aarch64_repeated" job)
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #19 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	rclpy.test_parameter_client.xunit.missing_result
	projectroot.test_parameter_client

Issue #20 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	rclcpp.TestTimeSource.callback_handler_erasure
	projectroot.test.rclcpp.test_time_source
	  (also in: "Rci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #21 in nightly_win_rep
- Failing 21.43% in 3 out of 14 builds
	rclcpp.TestUtilities.wait_for_message_twice_one_sub
	rclcpp.TestUtilities.wait_for_message_indefinitely
	rclcpp.TestUtilities.wait_for_message
	rclcpp.TestTimeSource.reattach
	rclcpp.TestTimeSource.parameter_activation
	rclcpp.TestTimeSource.no_pre_jump_callback
	rclcpp.TestTimeSource.detachUnattached
	rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic
	rclcpp.TestTimeSource.clock
	rclcpp.TestTimeSource.check_use_clock_thread_value
	rclcpp.TestTimeSource.check_clock_thread_status
	rclcpp.TestTimeSource.callbacks
	rclcpp.TestTimeSource.ROS_time_valid_wall_time
	rclcpp.TestTimeSource.ROS_time_valid_sim_time
	rclcpp.TestTimeSource.ROS_time_valid_attach_detach
	rclcpp.TestTimeSource.ROS_invalid_sim_time
	rclcpp.TestLoggerService.check_connect_get_logger_service
	rclcpp.TestInterfaceTraits.has_node_base_interface
	rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction
	projectroot.test.rclcpp.test_wait_for_message
	projectroot.test.rclcpp.test_logger_service
	projectroot.test.rclcpp.test_interface_traits

Issue #22 in Ici__nightly-debug_ubuntu_jammy_amd64
- Failing 40.0% in 2 out of 5 builds
	launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)
	  (also in: "nightly_linux_humble_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED)

Issue #23 in gz_fuel_tools-8-win
- Failing 66.67% in 2 out of 3 builds
	FuelClientTest/FuelClientDownloadTest.DownloadModel/1
	  (also in: "gz_fuel_tools-ign-fuel-tools7-win" job)
	  (also in: "gz_fuel_tools-ign-fuel-tools4-win" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools7-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-ign-fuel-tools4-focal-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-homebrew-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)
	  (also in: "gz_fuel_tools-8-win" job)

Issue #24 in gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64
- Failing 66.67% in 2 out of 3 builds
	UNIT_gz_TEST.test_ran
	  (also in: "gz_fuel_tools-ci-main-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools9-jammy-amd64" job)
	  (also in: "gz_fuel_tools-ci-gz-fuel-tools8-focal-amd64" job)

Issue #25 in gz_sim-main-win
- Failing 50.0% in 2 out of 4 builds
	INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED)

Issue #26 in nightly_linux-aarch64_repeated
- Failing 12.5% in 2 out of 16 builds
	launch_testing_examples.launch_testing_examples.check_multiple_nodes_launch_test.launch_testing_examples.check_multiple_nodes_launch_test (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux_release" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)
	  (also in: "nightly_linux-rhel_debug" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED)

Issue #27 in nightly_linux_coverage
- Failing 14.29% in 2 out of 14 builds
	test_rclcpp.test_multithreaded.multi_access_publisher
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #28 in nightly_win_deb
- Failing 14.29% in 2 out of 14 builds
	rcl.test_node_type_cache__rmw_fastrtps_cpp.gtest.missing_result
	rcl.test_get_type_description_service__rmw_fastrtps_cpp.gtest.missing_result
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_nominal
	projectroot.test.test_service_event_publisher__rmw_fastrtps_cpp
	projectroot.test.test_node_type_cache__rmw_fastrtps_cpp
	projectroot.test.test_get_type_description_service__rmw_fastrtps_cpp

Issue #29 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/SingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/MultiThreadedExecutor.spinAll
	  (also in: "Rci__nightly-cyclonedds_ubuntu_jammy_amd64" job)

Issue #30 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.PerTimerType/TestTimer.test_failures_with_exceptions/wall_timer (4-byte object <00-00 00-00>)                     

Issue #31 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	rclcpp.TimeSource.valid_clock_type_for_sim_time
	rclcpp.TimeSource.invalid_sim_time_parameter_override
	rclcpp.TimeSource.invalid_clock_type_for_sim_time
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testHeadTimerCancelThenResetBehavior
	rclcpp.TestLoggerService.test_set_logging_level_with_partial_invalid_param
	rclcpp.TestLoggerService.test_set_logging_level_with_invalid_param
	rclcpp.TestLoggerService.test_set_and_get_one_logging_level
	rclcpp.TestLoggerService.test_set_and_get_multi_logging_level
	rclcpp.TestLoggerService.check_connect_set_logger_service
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spinAll
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/SingleThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/SingleThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/SingleThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/SingleThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/SingleThreadedExecutor.addNodeTwoExecutors
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilSharedFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteWithTimeout
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteNoTimeout
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinUntilFutureComplete
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/MultiThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWithTimer
	rclcpp.TestExecutors/MultiThreadedExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/MultiThreadedExecutor.emptyExecutor
	rclcpp.TestExecutors/MultiThreadedExecutor.detachOnDestruction
	rclcpp.TestExecutors/MultiThreadedExecutor.addTemporaryNode
	rclcpp.TestExecutors/MultiThreadedExecutor.addNodeTwoExecutors
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_timer_without_autostart/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_failures_with_exceptions/generic_timer (4-byte object <01-00 00-00>)                     

Issue #32 in nightly_win_rep
- Failing 14.29% in 2 out of 14 builds
	projectroot.test_arrays_service_introspection

Issue #33 in gz_gui-ci-ign-gui6-focal-amd64
- Failing 33.33% in 1 out of 3 builds
	UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED)
	UNIT_ShutdownButton_TEST.test_ran
	UNIT_Screenshot_TEST.test_ran
	UNIT_Publisher_TEST.test_ran

Issue #34 in gz_physics-ci-gz-physics6-homebrew-amd64
- Failing 33.33% in 1 out of 3 builds
	DetachableJointTest/0.CorrectAttachmentPoints
	  (also in: "gz_physics-ci-main-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics7-homebrew-amd64" job)
	  (also in: "gz_physics-ci-gz-physics6-homebrew-amd64" job)

Issue #35 in gz_sim-7-win
- Failing 20.0% in 1 out of 5 builds
	INTEGRATION_multiple_servers.test_ran
	  (also in: "gz_sim-main-win" job)
	  (also in: "gz_sim-ign-gazebo6-win" job)

Issue #36 in gz_sim-ci-gz-sim8-jammy-amd64
- Failing 14.29% in 1 out of 7 builds
	INTEGRATION_sensors_system.test_ran

Issue #37 in gz_sim-ci-ign-gazebo6-focal-amd64
- Failing 25.0% in 1 out of 4 builds
	ServerRepeat/FollowActorTest.PublishCmd/0
	ModelPhotoShootTest.ModelPhotoShootRandomJoints
	ModelPhotoShootTest.ModelPhotoShootDefaultJoints

Issue #38 in nightly_linux-aarch64_release
- Failing 6.67% in 1 out of 15 builds
	rosbag2_transport.RecordIntegrationTestFixture.published_messages_from_multiple_services_are_recorded
	  (also in: "nightly_linux-rhel_release" job)

Issue #39 in nightly_linux-aarch64_release
- Failing 6.67% in 1 out of 15 builds
	projectroot.test_record_all__rmw_cyclonedds_cpp
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_linux-rhel_release" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Issue #40 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testTimer1CancelledWithExecutorSpin
	  (also in: "nightly_win_rep" job)

Issue #41 in nightly_linux-aarch64_repeated
- Failing 6.25% in 1 out of 16 builds
	rosbag2_transport.RecordIntegrationTestFixture.can_record_again_after_stop
	projectroot.test_record__rmw_fastrtps_cpp

Issue #42 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	projectroot.test.rclcpp.test_executors_intraprocess
	  (also in: "nightly_win_rep" job)

Issue #43 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	projectroot.message_filters_test_latest_time_policy
	  (also in: "nightly_win_rep" job)

Issue #44 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.gtest_multithreaded__rmw_cyclonedds_cpp.gtest.missing_result
	rclcpp.test_executors_intraprocess.gtest.missing_result
	rclcpp.TestTimerCancelBehavior/MultiThreadedExecutor.testBothTimerCancelThenResetT1Behavior
	projectroot.gtest_multithreaded__rmw_cyclonedds_cpp
	message_filters.LatestTimePolicy.Leading

Issue #45 in nightly_win_deb
- Failing 7.14% in 1 out of 14 builds
	rcl.TestServiceEventPublisherWithServicesAndClientsFixture.test_service_event_publisher_with_subscriber_disable_service_events
	rcl.TestServiceEventPublisherWithServicesAndClientsFixture.test_service_event_publisher_with_subscriber
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_utils
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_send_message_return_codes
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_send_message_nominal
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_init_and_fini
	rcl.TestServiceEventPublisherFixture.test_service_event_publisher_enable_and_disable_return_codes

Issue #46 in nightly_linux_repeated
- Failing 6.67% in 1 out of 15 builds
	test_rclcpp.test_subscription.subscription_and_spinning
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_rel" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_repeated" job)
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #47 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3]
	  (also in: "nightly_linux-rhel_repeated" job)

Issue #48 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_tf2.BufferClientTestAfterShutdown.test_exit_code
	projectroot.test_buffer_client_tester.launch.py
	  (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job)
	  (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job)

Issue #49 in nightly_win_rep
- Failing 7.14% in 1 out of 14 builds
	test_rclcpp.test_local_parameters.local_async_with_callback
	test_rclcpp.test_local_parameters.helpers
	test_rclcpp.test_local_parameters.get_from_node_variant_type
	test_rclcpp.test_local_parameters.get_from_node_primitive_type
	rosidl_typesupport_introspection_tests.ArraysServiceIntrospectionTest/0.CanWriteTypeErasedRequestMessage
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.testRaceConditionAddNode
	rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some_max_duration
	rclcpp.TestExecutors/EventsExecutor.spinWithTimer
	rclcpp.TestExecutors/EventsExecutor.spinWhileAlreadySpinning
	rclcpp.TestExecutors/EventsExecutor.emptyExecutor
	rclcpp.TestExecutors/EventsExecutor.detachOnDestruction
	rclcpp.TestExecutors/EventsExecutor.addTemporaryNode
	rclcpp.TestExecutors/EventsExecutor.addNodeTwoExecutors
	rclcpp.PerTimerType/TestTimer.test_run_cancel_timer/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_run_cancel_timer/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_bad_arguments/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.test_bad_arguments/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.callback_with_timer/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.callback_with_timer/generic_timer (4-byte object <01-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.callback_with_period_zero/wall_timer (4-byte object <00-00 00-00>)                     
	rclcpp.PerTimerType/TestTimer.callback_with_period_zero/generic_timer (4-byte object <01-00 00-00>)                     
	rcl_action.TestActionServerWait.test_wait_set_add_action_server
	projectroot.gtest_local_parameters__rmw_fastrtps_cpp

Issue #50 in nightly_linux_debug
- Failing 6.25% in 1 out of 16 builds
	demo_nodes_cpp.TestExecutablesTutorial.test_processes_output
	  (also in: "nightly_win_rep" job)
	  (also in: "nightly_win_deb" job)
	  (also in: "nightly_linux_debug" job)
	  (also in: "nightly_linux-rhel_repeated" job)
	  (also in: "nightly_linux-aarch64_repeated" job)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
buildfarmer-log Log of issues in the buildfarm
Projects
None yet
Development

No branches or pull requests

2 participants