Logging intermittent failure fix and enhancement by capturing memory stats

[cpelley]

• Record plugin step memory footprint to the log (+sqlite.db).
  • New tools made available under utils for this purpose (get_proc_mem_stat, get_sys_mem_stat, CaptureProcMemory, CaptureSysMemory).
  • This is a must for run-time and offline analysis of graph execution.
  • This is a pre-requisite if we have to pick-up INV: Real-time resource recording, usage and feedback mechanism #5
• Expose host and port number of client and server side components of the logger (pre-requisite for multi-node execution).
• logger server now exposed as an executable script: dagrunner-logger (on pip install). Pre-requisite capability for a single logger server shared amongst any number independent graphs being executed.
• Fix logging integration test intermittent failures (mostly seen during CI previously) and re-enable the tests run under CI.
  • See comment for context.

Issues

• https://github.com/MetOffice/improver_suite/issues/2253
• https://github.com/MetOffice/improver_suite/issues/2079

Note

Ignore the CI failure. See comment.

[cpelley]

This is the necessary change to avoid intermittent failures (i.e. why logging was disabled in CI).
This ensures that we write everything from the log queue before closing things down.

[cpelley]

The CI failure is to do with the reference branch (master) which still contains the bug that this PR fixes. Note that the pytest run for this branch all succeed.