pbs: handle poll if PBS client cannot connect

[matthewrmshin]

If PBS qstat cannot connect to its server, assume that jobs managed by it are still OK.

[hjoliver]

Looks fine. Can this be tested - e.g. by temporarily aliasing the batch sys poll command to a custom or non-existent command?

[matthewrmshin]

Yes, thinking about how to write an effective test.

[matthewrmshin]

Test added.

[oliver-sanders]

How stable is this message across different pbs versions?

[matthewrmshin]

We only have our own to test on our site, so no idea. Our admin team is confident that it is a good message to use though. @hjoliver may have a better insight with his connection with the people who develop PBS.

[hjoliver]

I'll ping my PBS connections...

[hjoliver]

(however, we can always adapt to other PBS versions in future releases if need be)

[hjoliver]

(... pinged)

[hjoliver]

From a contact at Altair:

" is this response standard across PBS versions?"
That is generally correct. I've experienced this across a few versions. Also re-tested on Azure, and message is:

------------
Connection refused
qstat: cannot connect to server xxxxxx (errno=111)
------------

Also correct to assume, if you get the message above, it is no reflection on jobs itself. Simply means the PBS daemons can't be contacted. As usual, PBS jobs which are already running, will keep running. Not so for array sub-jobs.

But wait.... for PBS v18.x and beyond.

• Even subjobs of array jobs, when they are running, will keep running if PBS daemons crashes or whatever.
• Qstat will have JSON format output as an option, and customizable delimiters
• Job's stdout/stderr can be retrieved while job is still running.

.... just in case you want to plan for Cylc 8, 9, or beyond.

[matthewrmshin]

(The PBS v18.x Qstat new features are pretty exciting, but I don't think they would affect the purpose of this PR.)