shoring up the guardian watchers...

[jumpsiegel]

Below are some example changes to the hedera watcher to make it not crash the entire guardian on a bug

(https://github.com/wormhole-foundation/wormhole/compare/hedera...hedeta-comment?expand=1#diff-67e6ea5614f7abd7ad39660937fe8f6b0c3b3c359323d48df97e5f1394607b0a) Pr examples

Key thing here to remember is, if you create a go func/thread, you have to add a defer/recover to deal with any error and correctly propagate it to the supervisor. You also need to add a defer/recover for the main thread because the supervisor currently logs the panic nicely and THEN kills the guardian

Possibly we should add a common function to the supervisor (supervisor.GoFunc()?) that takes the err channel and the thread name...

This would work for everything other then the top level thread since we have to directly modify the return type in that case

[jumpsiegel]

Switch to a utility function...

eb29fcf

[jumpsiegel]

https://github.com/wormhole-foundation/wormhole/tree/hedeta-comment/testing/scissors

[aadam-10]

since then the watchers have been updated with the pattern.