CI stack check timeout

[mcsauder]

Describe the bug
The CI stck check very frequently times out, so CI fails even though there might be no issue with a PR.

To reproduce, rebase a previously passing PR against current master, the CI tools will re-run, and frequently the stack check will fail with a timeout.

At the time of this writing, these three PR's can be observed exhibiting this behavior:
http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/PR-11260/11/pipeline
http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/PR-11192/33/pipeline
http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/PR-11194/20/pipeline

Expected behavior
@dagar, if a timeout has been exceeded, is there a way to make the CI test retry/restart again after a wait period? Currently I wait for the master branch to update, then rebase and force push again in hopes of getting through... (As a bonus, when it does pass after a rebase, the experience is a lot like winning the lottery! :) Unfortunately, rebasing again often leads to a new failure.)

Let me know if I can help provide additional information. Thanks for your help with this!

-Mark

[dagar]

I believe it's probably an issue with the python scripts that monitor the serial console.
https://github.com/PX4/Firmware/tree/master/Tools/HIL

Serial possibly getting stuck in a bad state from a previous run.

[mcsauder]

PR #11265 currently shows this behavior: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/PR-11265/12/pipeline

[mcsauder]

PR #11214 currently shows this behavior: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-fmu-v4_spi_ext/10/pipeline

[mcsauder]

PR #11167 currently shows this behavior: http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-lndmc-takeoff/4/pipeline

[mcsauder]

I haven't seen this issue in about 8 days, so I think recent changes might have fixed it. I can go ahead and close this issue early next week after it makes it to the 10 day point. Thanks for squashing this bug!