CI ROS test timeout

[mcsauder]

Describe the bug
The CI ROS tests frequently timeout. Sometimes after a rebase a previously passing PR will fail, and other times after a rebase against current master the same tests will pass.

At the time of this writing, the following PR(s) exhibit this failure mode:
http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/PR-11194/20/pipeline
http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/PR-11260/11/pipeline

To Reproduce
Rebase a current PR against master and force push to remote. This will re-run/restart the CI tests, and sometimes everything will run smoothly, however, sometimes the tests will timeout.

Expected behavior
@dagar, if a timeout occurs, is there a way to automatically restart the test, (perhaps a finite number of times), or allow users to manually restart failing tests through the web interface?

[dagar]

Duplicate of #11257

[lamping7]

This isn't a load-based timeout. There is a failure of the software env not monitored by the test itself.

[mcsauder]

Another occurrence of the issue reported in this issue is in PR #11314 at the time of this writing: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/PR-11314/1/pipeline

(I don't believe that the LOC in this PR could impact the ROS tests.)

[mcsauder]

PR #11314 is passing now after a rebase with current PX4/Firmware master.

[mcsauder]

PR #11308 currently displays this behavior: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/modular-attitude-control/2/pipeline

[mcsauder]

So does PR #1305: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/windows-exiting-hotfix/5/pipeline

[lamping7]

#11308 and #11189 are different from the original issue you posted about. Those are preflight fails for accels.

[mcsauder]

Thanks @lamping7 , this is what I see at the bottom of #11308:

ERROR: max time [300.0s] allotted for test [VTOL_mission_1] of type [px4/mission_test.py]

But you're right about #11189,

WARN  [commander] Preflight Fail: Accels inconsistent - Check Cal


Exiting...

I'll delete the reference to #11189, I think I autocompleted an incorrect PR. I'll look again. Thanks!

[lamping7]

Scroll up on #11308. The accel problem is not associated with these PRs. It seems something crept into master.

#11316 is doing it too

[mcsauder]

@Lamping, #11305 still shows the same behaviors, so at least that instance is still relevant. I watched #11189 automagically reload the page about 20 times and keep scrolling my page to the bottom shortly after your post, so I honestly can't say if it was the same at the point you saw it as when I saw it... if the CI tools run again it definitely yields different results, I've seen this many times in the past few months with rebasing my own PRs. Any ideas? How can I help out?

[mcsauder]

PR #11224 is back to the same failure mode after rebase with current master: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/PR-11224/21/pipeline

[mcsauder]

PR #11192 is back to the same failure mode after rebase against current master: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-SITL_tests/detail/PR-11192/36/pipeline

Updated: An additional rebase against current master is passing again now: #11192

[mcsauder]

I haven't seen this issue in roughly the past week, so I think recent changes might have fixed it. Thanks for squashing this bug!