Replace Pyro with HTTP(S) communication layer

[benfitzpatrick]

This closes #1872.

I thought it was time to put it up for discussion.

Pyro has been replaced with cherrypy (server) and built-in Python library urllib2 (optionally upgrading to requests if possible). cherrypy is bundled with cylc as Pyro was, and there are no new dependencies apart from optional dependence on OpenSSL (for HTTPS rather than HTTP) and the requests library for better certificate verification (N.B. We can do better certificate verification than is done in this PR with pure Python 2.7).

Communication is via HTTP(S) secured with HTTP Digest Auth and the usual passphrase system, using 'cylc' as the username for full access and 'anon' for the public access. I have separated out the authentication so that theoretically someone can add another auth system if they want to do a fair bit of work!

All test-battery tests pass (although some seem flakier than usual). I think flakiness is due to some shutdown/ports file timing issue and will investigate. (EDIT: Seems much better now)

Some compromises and other information:

• SSL certificates and private keys are generated on registration, although they could (?) be generated on run and then assigned to the correct host. Otherwise they have to be wildcard certificates. We could also support some given sub-domains.
• SSL certificate remote and alternative suite/host fetching is a bit ugly and could be further consolidated with passphrase logic.
• SSL certificates are self-signed as we need one per suite. We can think about improving this for later browser interfaces (e.g. having a master certificate that has a child certificate for each suite and then importing that into the browser)
• suite host self-identification only works at present if the suite is registered on the same host that it is run on (we can force this to be true if we think it's necessary, or force generate-certificate-on-run)
• daemons and clients fall back to HTTP if they have to
• Digest Auth uses MD5 as standard so we should add support for a choice of auth (maybe even basic auth, which is 'OK' for HTTPS) I've added support for SHA1 if we need to for FIPS support.
• SSL is better supported at higher versions of Python than we use here - it is more secure at >2.7.9.
• There are going to be some debug statements that I need to remove

[matthewrmshin]

Initial comments:

• Well done!
• I wonder if it will make review easier you can do some rebase and squash. In particular, if you can move the changeset to bundle cherrypy (and remove pyro) to either end.
• It appears that you have evolved an older version of cylc.registration. I have since taken out some old logic.
• Some network service modules are basically an import statement. I wonder if they can be grouped together or not?
• Tests definitely a bit flaky. Test battery without global configuration passes after re-running only failed tasks on the 3rd attempt.

On your bullet points 1 and 4: generate-on-run is probably desirable for us, but may not be popular in some circumstances at other sites.

[trwhitcomb]

Well done indeed - this is an excellent change. I do have one question about module redesign - why the decision to split the clients and servers into separate modules instead of having a single module containing both?

[matthewrmshin]

@trwhitcomb Splitting the server and the client allows an individual client to only import modules that are relevant to the client, and not all the stuffs required to drive the server.

[benfitzpatrick]

Just to add to what Matt has said - importing lots of modules can take a surprisingly long amount of time - although we don't (much) care about the elapsed time of cylc trigger, we do care about cylc task message.

[benfitzpatrick]

Tests much more stable now. This was a port-grabbing problem. cherrypy did not like it when started with a particular (non-occupied) port which was then stolen just before it could start. Specifically, it would have died with os._exit (same as sys.exit) and taken the whole daemon down.

[benfitzpatrick]

I wonder if it will make review easier you can do some rebase and squash. In particular, if you can move the changeset to bundle cherrypy (and remove pyro) to either end.

Done

[matthewrmshin]

Looking better. I have tests/authentication/04-* failing with ChannelFailures, which I suppose is the port grabbing issue? Otherwise, tests certainly more stable than before.

[benfitzpatrick]

7cbe2ea and b87c240 add SHA1 support for HTTP Digest Auth to cherrypy/cylc and avoids using MD5 anywhere - so it should be more FIPS-compliant. The requests and Python's in-built urllib2 library both allow use of SHA1 (and autodetect its use), so only the server side needed altering. I decided not to go for basic auth.

[hjoliver]

@cylc/core - can this be merged as soon as I've reviewed it, as far as MO is concerned? Or are there admin changes (e.g. to network port range) that suggest we should bump this to cylc-7 (but still very soon, I hope!). If so, I'd like to obsolete cylc-5 syntax too - we can discuss offline if not so straightforward at your end.

[benfitzpatrick]

@hjoliver and @matthewrmshin, please take an initial look - probably best to ignore the import cherrypy/Pyro deletion commits.

[matthewrmshin]

It is worth checking if cherrypy does any excessive DNS lookups (like Pyro used to do) by using strace -e getsockname on something like cylc gscan.

[hjoliver]

@benfitzpatrick - sorry for the long delay on reviewing this. All seems to work well on my metomi VM - very nice - but on my Centos box at work, I get "ERROR Not authorized: ... access type 'private'" for any attempt to connect.

[hjoliver]

Scratch that, it's working now (on Centos). I suspect network issues earlier (I noticed many tests fail on the VM at home too, immediately after turning the laptop on, until I get internet connectivity...)

[hjoliver]

The --comms-timeout client option doesn't work. Start a suite with --no-detach then suspend it with Ctrl-Z, then run cylc scan --comms-timeout=2 SUITE or cylc dump --comms-timeout=2 SUITE etc. - hangs indefinitely.

[benfitzpatrick]

No conflicts (right now)

[benfitzpatrick]

Now unconflicted...

[benfitzpatrick]

Rebased and pushed again.

[benfitzpatrick]

https://www.timeanddate.com/countdown/generic?iso=20160923T17&p0=1358&msg=%231923+no+longer+Ben%27s+problem&font=sanserif

[benfitzpatrick]

@matthewrmshin, @hjoliver, please review and merge

[arjclark]

https://www.timeanddate.com/countdown/generic?iso=20160923T17&p0=1358&msg=%231923+no+longer+Ben%27s+problem&font=sanserif

I'll remember this when you're looking for suite support in your new role...

[matthewrmshin]

This is now good. Test battery passing robustly in my environment.

[hjoliver]

Can't see any problems after another high-speed blast through the code, and tests passing. I think we can merge this in and let it mature on master. 🎉

Excellent work @benfitzpatrick, as ever ... and sayonara 😢 I'll buy you a 🍺 in Lisbon.

That countdown timer is evil, by the way.

[hjoliver]

Approved!