Add autoreload option (#1983)

examples/getting_started/Introduction.ipynb

@@ -205,7 +205,7 @@
     "\n",
     "You can use this compositional approach to combine different components such as widgets, plots, text, and other elements needed for an app or dashboard in arbitrary ways. The ``interact`` example builds on a reactive programming model, where an input to the function changes and Panel reactively updates the output of the function. ``interact`` is a convenient way to create widgets from the arguments to your function automatically, but Panel also provides a more explicit reactive API letting you specifically define connections between widgets and function arguments, and then lets you compose the resulting dashboard manually from scratch.\n",
     "\n",
-    "In the example below we explicitly declare each of the components of an app: widgets, a function to return the plot, column and row containers, and the completed `occupancy` Panel app. Widget objects have multiple \"parameters\" (current value, allowed ranges, and so on), and here we will use Panel's ``depends`` decorator to declare that function's input values should come from the widgets' ``value`` parameters. Now when the function and the widgets are displayed, Panel will automatically update the displayed output whenever any of the inputs change:"
+    "In the example below we explicitly declare each of the components of an app: widgets, a function to return the plot, column and row containers, and the completed `occupancy` Panel app. Widget objects have multiple \"parameters\" (current value, allowed ranges, and so on), and here we will use Panel's ``bind`` function to declare that function's input values should come from the widgets' ``value`` parameters. Now when the function and the widgets are displayed, Panel will automatically update the displayed output whenever any of the inputs change:"
    ]
   },
   {
@@ -220,9 +220,7 @@
     "                                 options=list(data.columns))\n",
     "window  = pnw.IntSlider(name='window', value=10, start=1, end=60)\n",
     "\n",
-    "@pn.depends(variable, window)\n",
-    "def reactive_outliers(variable, window):\n",
-    "    return find_outliers(variable, window, 10)\n",
+    "reactive_outliers = pn.bind(find_outliers, variable, window, 10)\n",
     "\n",
     "widgets   = pn.Column(\"<br>\\n# Room occupancy\", variable, window)\n",
     "occupancy = pn.Row(reactive_outliers, widgets)\n",
@@ -263,6 +261,13 @@
     "occupancy.servable();"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "During development, particularly when working with a raw script using `panel serve --show --autoreload` can be very useful as the application will automatically update whenever the script or notebook or any of its imports change."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

examples/user_guide/Deploy_and_Export.ipynb

@@ -20,7 +20,7 @@
     "\n",
     "## Configuring output\n",
     "\n",
-    "As you may have noticed, almost all the Panel documentation is written using notebooks. Panel objects display themselves automatically in a notebook and take advantage of Jupyter Comms to support communication between the rendered app and the Jupyter kernel that backs it on the Python end. To display a Panel object in the notebook is as simple as putting it on the end of a cell. Note, however, that the ``panel.extension`` first has to be loaded to initialize the required JavaScript in the notebook context.  Also, if you are working in JupyterLab, the pyviz labextension has to be installed with:\n",
+    "As you may have noticed, almost all the Panel documentation is written using notebooks. Panel objects display themselves automatically in a notebook and take advantage of Jupyter Comms to support communication between the rendered app and the Jupyter kernel that backs it on the Python end. To display a Panel object in the notebook is as simple as putting it on the end of a cell. Note, however, that the ``panel.extension`` first has to be loaded to initialize the required JavaScript in the notebook context. In recent versions of JupyterLab this works out of the box but for older versions (`<3.0`) the PyViz labextension has to be installed with:\n",
     "\n",
     "    jupyter labextension install @pyviz/jupyterlab_pyviz\n",
     "\n",
@@ -292,6 +292,8 @@
     "\n",
     "    panel serve apps/*.py\n",
     "    \n",
+    "For development it can be particularly helpful to use the ``--autoreload`` option to `panel serve` as that will automatically reload the page whenever the application code or any of its imports change.\n",
+    "    \n",
     "The ``panel serve`` command has the following options:\n",
     "\n",
     "    positional arguments:\n",
@@ -354,6 +356,8 @@
     "      --num-procs N         Number of worker processes for an app. Using 0 will\n",
     "                            autodetect number of cores (defaults to 1)\n",
     "      --warm                Whether to execute scripts on startup to warm up the server.\n",
+    "      --autoreload\n",
+    "                            Whether to automatically reload user sessions when the application or any of its imports change.\n",
     "      --static-dirs KEY=VALUE [KEY=VALUE ...]        \n",
     "                            Static directories to serve specified as key=value\n",
     "                            pairs mapping from URL route to static file directory.\n",

panel/command/serve.py

@@ -16,6 +16,7 @@
 from ..auth import OAuthProvider
 from ..config import config
 from ..io.rest import REST_PROVIDERS
+from ..io.reload import record_modules, watch
 from ..io.server import INDEX_HTML, get_static_routes
 from ..io.state import state
 from ..util import edit_readonly
@@ -116,6 +117,10 @@ class Serve(_BkServe):
         ('--warm', dict(
             action  = 'store_true',
             help    = "Whether to execute scripts on startup to warm up the server."
+        )),
+        ('--autoreload', dict(
+            action  = 'store_true',
+            help    = "Whether to autoreload source when script changes."
         ))
     )
 
@@ -148,19 +153,26 @@ def customize_kwargs(self, args, server_kwargs):
         with edit_readonly(state):
             state.base_url = urljoin('/', prefix)
 
-        if args.warm:
-            argvs = {f: args.args for f in files}
-            applications = build_single_handler_applications(files, argvs)
-            for app in applications.values():
-                app.create_document()
-
         # Handle tranquilized functions in the supplied functions
         if args.rest_provider in REST_PROVIDERS:
             pattern = REST_PROVIDERS[args.rest_provider](files, args.rest_endpoint)
             patterns.extend(pattern)
         elif args.rest_provider is not None:
             raise ValueError("rest-provider %r not recognized." % args.rest_provider)
 
+        config.autoreload = args.autoreload
+
+        if config.autoreload:
+            for f in files:
+                watch(f)
+
+        if args.warm or args.autoreload:
+            argvs = {f: args.args for f in files}
+            applications = build_single_handler_applications(files, argvs)
+            with record_modules():
+                for app in applications.values():
+                    app.create_document()
+
         config.session_history = args.session_history
         if args.rest_session_info:
             pattern = REST_PROVIDERS['param'](files, 'rest')

panel/config.py

@@ -80,6 +80,9 @@ class _config(_base_config):
         Whether to set custom Signature which allows tab-completion
         in some IDEs and environments.""")
 
+    autoreload = param.Boolean(default=False, doc="""
+        Whether to autoreload server when script changes.""")
+
     safe_embed = param.Boolean(default=False, doc="""
         Ensure all bokeh property changes trigger events which are
         embedded. Useful when only partial updates are made in an

panel/io/reload.py

@@ -0,0 +1,150 @@
+import fnmatch
+import os
+import sys
+import types
+
+from contextlib import contextmanager
+from functools import partial
+
+from .callbacks import PeriodicCallback
+from .state import state
+
+_watched_files = set()
+_modules = set()
+_callbacks = {}
+
+# List of paths to ignore
+DEFAULT_FOLDER_BLACKLIST = [
+    "**/.*",
+    "**/anaconda",
+    "**/anaconda2",
+    "**/anaconda3",
+    "**/dist-packages",
+    "**/miniconda",
+    "**/miniconda2",
+    "**/miniconda3",
+    "**/node_modules",
+    "**/pyenv",
+    "**/site-packages",
+    "**/venv",
+    "**/virtualenv",
+]
+
+
+def in_blacklist(filepath):
+    return any(
+        file_is_in_folder_glob(filepath, blacklisted_folder)
+        for blacklisted_folder in DEFAULT_FOLDER_BLACKLIST
+    )
+
+def file_is_in_folder_glob(filepath, folderpath_glob):
+    """
+    Test whether a file is in some folder with globbing support.
+
+    Parameters
+    ----------
+    filepath : str
+        A file path.
+    folderpath_glob: str
+        A path to a folder that may include globbing.
+    """
+    # Make the glob always end with "/*" so we match files inside subfolders of
+    # folderpath_glob.
+    if not folderpath_glob.endswith("*"):
+        if folderpath_glob.endswith("/"):
+            folderpath_glob += "*"
+        else:
+            folderpath_glob += "/*"
+
+    file_dir = os.path.dirname(filepath) + "/"
+    return fnmatch.fnmatch(file_dir, folderpath_glob)
+
+def autoreload_watcher():
+    """
+    Installs a periodic callback which checks for changes in watched
+    files and sys.modules.
+    """
+    cb = partial(_reload_on_update, {})
+    _callbacks[state.curdoc] = pcb = PeriodicCallback(callback=cb)
+    pcb.start()
+
+def watch(filename):
+    """
+    Add a file to the watch list.
+
+    All imported modules are watched by default.
+    """
+    _watched_files.add(filename)
+
+@contextmanager
+def record_modules():
+    """
+    Records modules which are currently imported.
+    """
+    modules = set(sys.modules)
+    yield
+    if _modules:
+        return
+    for module_name in set(sys.modules).difference(modules):
+        if module_name.startswith('bokeh_app'):
+            continue
+        module = sys.modules[module_name]
+        try:
+            spec = getattr(module, "__spec__", None)
+            if spec is None:
+                filepath = getattr(module, "__file__", None)
+                if filepath is None: # no user
+                    continue
+            else:
+                filepath = spec.origin
+
+            filepath = os.path.abspath(filepath)
+
+            if filepath is None or in_blacklist(filepath):
+                continue
+
+            if not os.path.isfile(filepath): # e.g. built-in
+                continue
+            _modules.add(module_name)
+        except Exception:
+            continue
+
+def _reload(module=None):
+    if module is not None:
+        for module in _modules:
+            del sys.modules[module]
+    if state.curdoc in _callbacks:
+        _callbacks[state.curdoc].stop()
+        del _callbacks[state.curdoc]
+    if state.location:
+        state.location.reload = True
+
+def _check_file(modify_times, path, module=None):
+    try:
+        modified = os.stat(path).st_mtime
+    except Exception:
+        return
+    if path not in modify_times:
+        modify_times[path] = modified
+        return
+    if modify_times[path] != modified:
+        _reload(module)
+        modify_times[path] = modified
+
+def _reload_on_update(modify_times):
+    for module_name in _modules:
+        # Some modules play games with sys.modules (e.g. email/__init__.py
+        # in the standard library), and occasionally this can cause strange
+        # failures in getattr.  Just ignore anything that's not an ordinary
+        # module.
+        module = sys.modules[module_name]
+        if not isinstance(module, types.ModuleType):
+            continue
+        path = getattr(module, "__file__", None)
+        if not path:
+            continue
+        if path.endswith(".pyc") or path.endswith(".pyo"):
+            path = path[:-1]
+        _check_file(modify_times, path, module_name)
+    for path in _watched_files:
+        _check_file(modify_times, path)

panel/io/server.py

@@ -2,11 +2,13 @@
 Utilities for creating bokeh Server instances.
 """
 import datetime as dt
+import html
 import inspect
 import os
 import pathlib
 import signal
 import sys
+import traceback
 import threading
 import uuid
 
@@ -21,6 +23,7 @@
 
 # Bokeh imports
 from bokeh.application import Application as BkApplication
+from bokeh.application.handlers.code import CodeHandler
 from bokeh.application.handlers.function import FunctionHandler
 from bokeh.command.util import build_single_handler_application
 from bokeh.document.events import ModelChangedEvent
@@ -39,6 +42,7 @@
 from tornado.wsgi import WSGIContainer
 
 # Internal imports
+from .reload import autoreload_watcher
 from .resources import BASE_TEMPLATE, Bundle, Resources
 from .state import state
 
@@ -174,6 +178,71 @@ async def get(self, *args, **kwargs):
 
 per_app_patterns[0] = (r'/?', DocHandler)
 
+def modify_document(self, doc):
+    from bokeh.io.doc import set_curdoc as bk_set_curdoc
+    from ..config import config
+
+    if config.autoreload:
+        path = self._runner.path
+        argv = self._runner._argv
+        handler = type(self)(filename=path, argv=argv)
+        self._runner = handler._runner
+
+    module = self._runner.new_module()
+
+    # If no module was returned it means the code runner has some permanent
+    # unfixable problem, e.g. the configured source code has a syntax error
+    if module is None:
+        return
+
+    # One reason modules are stored is to prevent the module
+    # from being gc'd before the document is. A symptom of a
+    # gc'd module is that its globals become None. Additionally
+    # stored modules are used to provide correct paths to
+    # custom models resolver.
+    sys.modules[module.__name__] = module
+    doc._modules.append(module)
+
+    old_doc = curdoc()
+    bk_set_curdoc(doc)
+    old_io = self._monkeypatch_io()
+
+    if config.autoreload:
+        set_curdoc(doc)
+        state.onload(autoreload_watcher)
+
+    try:
+        def post_check():
+            newdoc = curdoc()
+            # script is supposed to edit the doc not replace it
+            if newdoc is not doc:
+                raise RuntimeError("%s at '%s' replaced the output document" % (self._origin, self._runner.path))
+
+        def handle_exception(handler, e):
+            from bokeh.application.handlers.handler import handle_exception
+            from ..pane import HTML
+
+            # Clean up
+            del sys.modules[module.__name__]
+            doc._modules.remove(module)
+            bokeh.application.handlers.code_runner.handle_exception = handle_exception
+            tb = html.escape(traceback.format_exc())
+
+            # Serve error
+            HTML(
+                f'<b>{type(e).__name__}</b>: {e}</br><pre style="overflow-y: scroll">{tb}</pre>',
+                css_classes=['alert', 'alert-danger'], sizing_mode='stretch_width'
+            ).servable()
+
+        if config.autoreload:
+            bokeh.application.handlers.code_runner.handle_exception = handle_exception
+        self._runner.run(module, post_check)
+    finally:
+        self._unmonkeypatch_io(old_io)
+        bk_set_curdoc(old_doc)
+
+CodeHandler.modify_document = modify_document
+
 #---------------------------------------------------------------------
 # Public API
 #---------------------------------------------------------------------