Leftover healthcheck processes from engine crash loops

[cywang117]

When the engine is not allowed time to gracefully stop, its leftover processes become orphaned from what I can tell and do not get cleaned up. A possible fix for this is to have the engine attempt to clean up leftover processes on start.

Description

Steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of balena-engine version:

(paste your output here)

Output of balena-engine info:

(paste your output here)

Additional environment details (device type, OS, etc.):

[jellyfish-bot]

[cywang117] This issue has attached support thread https://jel.ly.fish/aa791470-bc0b-4694-b4a6-07998b249870

[lmbarros]

In our experience, many cases (most of them, I dare to say) of the Engine crash-looping were ultimately triggered by Systemd's watchdog killing the Engine. We have vastly improved Engine health checks with this PR: balena-os/meta-balena#2734 , so for this common case, upgrading to balenaOS v2.101.7 shall be a good solution.

I am not closing this issue because it is still relevant for other, rarer cases of Engine crashloops.