Use cylc subcommand for remote functionality

[matthewrmshin]

E.g.:

• On fetching a remote log by cylc cat-log cat-log: remote cylc sub-command #2503
• On tailing a remote log by the tailer of cylc gui. cat-log: remote cylc sub-command #2503
• On job host initialisation. Init remote via multiprocessing pool #2468
• On job jost clean-up (remove suite contact file). Init remote via multiprocessing pool #2468
• SSH+HTTP(S) client-server communication client commands in tasks: detect ssh comms #2214 (comment) Improve comm #2582

See also #2292, metomi/rose#2070 and metomi/rose#2075.

[hjoliver]

Just to note, the primary reason for this is: so we don't have to invoke a bash shell via ssh to ensure correct interpretation of complex commands regardless of the user's default shell.

[hjoliver]

(and: security people evidently don't like invoking login shells over ssh)

[hjoliver]

I'll be looking at the first two tick-boxes next week. The third is handled by current PR #2468

[hjoliver]

(after this we'll need to update the remote command transcript in the CUG)

[hjoliver]

(assigned @matthewrmshin for the job init part).

[hjoliver]

The cylc cat-log command is something of a mess, having gradually accrued different modes and target files. At this point it can print, list, cat, tail, test, or view-in-editor any suite or job log on the suite or job hosts. To single out the remote bits of this and put them behind a remote cylc command requires some refactoring, but some initial prototyping suggests it'll work.

Plus: the ability to use batch-system-specific cat and tail commands that target the batch job ID.

Plus: the need to ensure that remote tail processes die when the ssh connection is killed - this will actually be easier with a Python program at the remote end - we won't need the complicated shell expression to determine PPID in remote tail command template.

New design: a new function takes a plain log file path and, depending on args, it can print, list, cat, or tail it to stdout, or capture the file content and return it (the later for the edit mode). This function is only called on the host that the target file resides on, and will be called either locally on the suite host, or remotely on the job host via an ssh-invoked command, after all the suite-host processing that the determines exactly what the log file path is.

[hjoliver]

[Updated previous comment with additional complexities - the two "Plus:..." sentences.]