improve poll logging

[oliver-sanders]

• Improve the readability of poll messages in the job activity log.
• Add a new metric batch_sys_call_no_lines which provides the number of lines of stdout output received from the batch system call (e.g. qstat for pbs) for help with debugging polling issues.

[hjoliver]

Can you explain why the "number of lines" of batch sys stdout is helpful here?

[oliver-sanders]

We've had an interesting episode here with running tasks being marked as failed. We can't quite fathom how this has happened, our best bet is that qstat is returning a null stdout or something to that effect. As we are using negative rather than positive checking (i.e. check something isn't there rather than check that it is) in this event all tasks which are presumed to be running would be marked as failed. Later on when the still happily running job completes and cylc message sends back the "I'm done" message the task gets bumped from failed to succeeded.

We were hoping that if this were to happen again knowing whether qstat returned any output might help to determine where this bug lies.