pserve reloader revamp

[mmerickel]

This PR replaces the --reload support in pserve with the new hupper package [1].

Features

• The monitor is now a separate process that will not crash and starts up before any of your code.
• The monitor will not restart the process after a crash until a file is saved.
• The monitor works on windows (currently only on py35 but I'm working on it... the current pserve has lots of issues on windows, so it's still an improvement).
• You can now trigger a reload manually from a pyramid view or any other code via hupper.get_reloader().trigger_reload(). Kind of neat.
• You can trigger a reload by issuing a SIGHUP to the monitor process.
• If the watchdog package [2] is installed then monitoring will be done using inotify instead of cpu-intensive polling.
• Fixes pserve --reload in 1.6 is less reliable than in 1.5 #2223, fix pserve --reload to avoid crashing the monitor #2250 and pserve --reload should use inotify or equivalent when available #2081.

Incompatibilities

• Drops hidden APIs like pyramid.scripts.pserve.watch_file, and pyramid.scripts.pserve.add_file_callback. support extending pserve to monitor arbitrary files via a custom ini section #2732 will be a path to a fully supported version of these features. hupper also has its own public API to add files to monitor via hupper.get_reloader().watch_files(...).

I want to avoid merging this until windows support is improved in hupper but we're close enough to get some review.

[1] http://docs.pylonsproject.org/projects/hupper/en/latest/
[2] https://pypi.org/project/watchdog/

[mmerickel]

This is ready for review.

[stevepiercy]

Is there a need to update the Pyramid docs?

If so, is there a separate issue open?

[mmerickel]

There's no change to the docs unless we want to add some mentions to the pserve docs like... To make this work better you should install watchdog into your development environment. See http://docs.pylonsproject.org/projects/hupper/en/latest/#watchdog-support for more info on what I mean.

[stevepiercy]

I made a new issue #2806 for making mention.

[mmerickel]

New commits should handle #2806.

[digitalresistor]

Will review this in a little bit.

[mmerickel]

I just rebased this branch on top of the latest master, should be mergeable again.

[digitalresistor]

Would it make sense to rebase and squash the first three commits down to "use hupper"?

[mmerickel]

Sure if you want me to. In general Pyramid hasn't been very picky about commit history.

[digitalresistor]

I tend to not want to have WIP commits in the final history... but if it doesn't matter, I don't care. I've been using this branch + hupper + watchdog for the last 3 hours for a project I am working on. Haven't seen it crash yet!!!

Also, it seems to be much faster at reloading than the old version. With watchdog it is fast enough that doing a pip install -U webob==1.7.0rc1 will cause it to restart three times as pip installs the newly updated files ;-)

[digitalresistor]

Other than possibly squashing the WIP I am a huge and excited +2 on this :D.

[mmerickel]

Is it restarting more than once per second?

The behaviors are a little bit more complex than the old reloader because the file monitor is always running so some race conditions can definitely exist that I've tried to account for.

[mmerickel]

Okay I made the history pretty.

[digitalresistor]

It might be more than once per second (I have watchdog installed) but it's just really fast... I haven't seen it do something weird or get wedged.