PreFlightCheck: add checks for CPU load

[MaEtUgR]

Describe problem solved by this pull request
We had cases where people were accidentally taking of with a too highly loaded system even already when it was disarmed. There can be a lot of reasons like custom WIP changes or a combination of board and configuration that exceeds the expected load.

Describe your solution
I add a preflight check that will prevent arming depending on CPU load because it's a low hanging fruit and would have saved crashes before.

Test data / coverage
Untested so far, SITL doesn't provide cpuload, not even fake so I'll test it on a pixhawk 4.

[dagar]

comment

[dagar]

A few notes that immediately come to mind.

1. For cpuload I think it should be an average to prevent any false positives from a small temporary cpu spike).
2. Similarly we should add something that prevents arming if the logger watchdog had triggered (boosting the logger priority to max). If that happens prematurely it can really drag the rest of the system down. This is actually why I introduced the logger_status message.
3. We should add a simple cpuload publisher for linux.
4. Instead of the check silently succeeding when the message isn't present I think it would be better to have it fail. On platforms where it's not available (or not wanted) we can have a parameter to explicitly disable it.

[MaEtUgR]

@dagar Good points, thanks for your ideas. I'll try to cover them.
For 2. I'm not clear how that would look like since logger_status doesn't contain a "watchdog_triggered" flag and I don't know the details.

[dagar]

For 2. I'm not clear how that would look like since logger_status doesn't contain a "watchdog_triggered" flag and I don't know the details.

Right, it needs to be added. Similarly something could be added here to require an sd card for operation (optionally) and possibly the offboard logging scenario as well (#13200).

[MaEtUgR]

1. For cpuload I think it should be an average to prevent any false positives from a small temporary cpu spike).

I just wanted to add a filter and checked the sampling of the cpu load in load_mon: It gets ampled with 1Hz https://github.com/PX4/Firmware/blob/74e99faedff29b863bb374f4e046e894dc6e4784/src/modules/load_mon/load_mon.cpp#L68 and already takes the average during that time: https://github.com/PX4/Firmware/blob/74e99faedff29b863bb374f4e046e894dc6e4784/src/modules/load_mon/load_mon.cpp#L192 I would have configured the filter with ~1 second time constant so it wouldn't make sense.

[dagar]

1. For cpuload I think it should be an average to prevent any false positives from a small temporary cpu spike).

I just wanted to add a filter and checked the sampling of the cpu load in load_mon: It gets ampled with 1Hz

https://github.com/PX4/Firmware/blob/74e99faedff29b863bb374f4e046e894dc6e4784/src/modules/load_mon/load_mon.cpp#L68

and already takes the average during that time:
https://github.com/PX4/Firmware/blob/74e99faedff29b863bb374f4e046e894dc6e4784/src/modules/load_mon/load_mon.cpp#L192

I would have configured the filter with ~1 second time constant so it wouldn't make sense.

Ok, thanks for looking into it. I might look into getting a higher rate cpuload published (maybe with average included) later. Those occasional cpu spikes are now somewhat more concerning...

[MaEtUgR]

3. We should add a simple cpuload publisher for linux.
4. Instead of the check silently succeeding when the message isn't present I think it would be better to have it fail. On platforms where it's not available (or not wanted) we can have a parameter to explicitly disable it.

I added a parameter that lets you set the threshold and disable the check. So it doesn't silently pass but has to be disabled for SITL.

Can we iterate in a separate pr on adding the logger_status and SITL CPU load publishing?

[dagar]

Can we iterate in a separate pr on adding the logger_status and SITL CPU load publishing?

Yes.

[MaEtUgR]

I rebased because there was a conflict with #14628 and added an esccaped % to the error message. I tested it on a pixhawk 4 and it works like expected when I start hogging the CPU too much with unnecessary operations.

@dagar Can we get this in as a first iteration?

[MaEtUgR]

I rebased again after a conflict with #14633. Still ready in my eyes.

[julianoes]

       _~
    _~ )_)_~
    )_))_))_)
    _!__!__!_
    \______t/
  ~~~~~~~~~~~~~
  Ship it!