New KernelLauncher API for kernel discovery system

.travis.yml

@@ -1,17 +1,21 @@
 language: python
 python:
     - "nightly"
-    - '3.6-dev'
+    - 3.6
     - 3.5
     - 3.4
-    - 3.3
     - 2.7
 sudo: false
 install:
   - pip install --upgrade setuptools pip
   - pip install --upgrade --pre -e .[test] pytest-cov pytest-warnings codecov
 script:
-  - py.test --cov jupyter_client jupyter_client
+  - |
+    if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      py.test --cov jupyter_client jupyter_client --ignore jupyter_client/tests/test_async_manager.py
+    else
+      py.test --cov jupyter_client jupyter_client
+    fi
 after_success:
   - codecov
 matrix:

docs/index.rst

@@ -23,6 +23,7 @@ with Jupyter kernels.
 
    kernels
    wrapperkernels
+   kernel_providers
 
 .. toctree::
    :maxdepth: 2

docs/kernel_providers.rst

@@ -0,0 +1,156 @@
+================
+Kernel providers
+================
+
+.. note::
+   This is a new interface under development, and may still change.
+   Not all Jupyter applications use this yet.
+   See :ref:`kernelspecs` for the established way of discovering kernel types.
+
+Creating a kernel provider
+==========================
+
+By writing a kernel provider, you can extend how Jupyter applications discover
+and start kernels. For example, you could find kernels in an environment system
+like conda, or kernels on remote systems which you can access.
+
+To write a kernel provider, subclass
+:class:`jupyter_client.discovery.KernelProviderBase`, giving your provider an ID
+and overriding two methods.
+
+.. class:: MyKernelProvider
+
+   .. attribute:: id
+
+      A short string identifying this provider. Cannot contain forward slash
+      (``/``).
+
+   .. method:: find_kernels()
+
+      Get the available kernel types this provider knows about.
+      Return an iterable of 2-tuples: (name, attributes).
+      *name* is a short string identifying the kernel type.
+      *attributes* is a dictionary with information to allow selecting a kernel.
+
+   .. method:: make_manager(name)
+
+      Prepare and return a :class:`~jupyter_client.KernelManager` instance
+      ready to start a new kernel instance of the type identified by *name*.
+      The input will be one of the names given by :meth:`find_kernels`.
+
+For example, imagine we want to tell Jupyter about kernels for a new language
+called *oblong*::
+
+    # oblong_provider.py
+    from jupyter_client.discovery import KernelProviderBase
+    from jupyter_client import KernelManager
+    from shutil import which
+
+    class OblongKernelProvider(KernelProviderBase):
+        id = 'oblong'
+
+        def find_kernels(self):
+            if not which('oblong-kernel'):
+                return  # Check it's available
+
+            # Two variants - for a real kernel, these could be something like
+            # different conda environments.
+            yield 'standard', {
+                'display_name': 'Oblong (standard)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+            yield 'rounded', {
+                'display_name': 'Oblong (rounded)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+
+        def make_manager(self, name):
+            if name == 'standard':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '0'})
+            elif name == 'rounded':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '1'})
+            else:
+                raise ValueError("Unknown kernel %s" % name)
+
+You would then register this with an *entry point*. In your ``setup.py``, put
+something like this::
+
+    setup(...
+        entry_points = {
+        'jupyter_client.kernel_providers' : [
+            # The name before the '=' should match the id attribute
+            'oblong = oblong_provider:OblongKernelProvider',
+        ]
+    })
+
+Finding kernel types
+====================
+
+To find and start kernels in client code, use
+:class:`jupyter_client.discovery.KernelFinder`. This uses multiple kernel
+providers to find available kernels. Like a kernel provider, it has methods
+``find_kernels`` and ``make_manager``. The kernel names it works
+with have the provider ID as a prefix, e.g. ``oblong/rounded`` (from the example
+above).
+
+::
+
+    from jupyter_client.discovery import KernelFinder
+    kf = KernelFinder.from_entrypoints()
+
+    ## Find available kernel types
+    for name, attributes in kf.find_kernels():
+        print(name, ':', attributes['display_name'])
+    # oblong/standard : Oblong (standard)
+    # oblong/rounded : Oblong(rounded)
+    # ...
+
+    ## Start a kernel by name
+    manager = kf.make_manager('oblong/standard')
+    manager.start_kernel()
+
+.. module:: jupyter_client.discovery
+
+.. autoclass:: KernelFinder
+
+   .. automethod:: from_entrypoints
+
+   .. automethod:: find_kernels
+
+   .. automethod:: make_manager
+
+Kernel providers included in ``jupyter_client``
+===============================================
+
+``jupyter_client`` includes two kernel providers:
+
+.. autoclass:: KernelSpecProvider
+
+   .. seealso:: :ref:`kernelspecs`
+
+.. autoclass:: IPykernelProvider
+
+Glossary
+========
+
+Kernel instance
+  A running kernel, a process which can accept ZMQ connections from frontends.
+  Its state includes a namespace and an execution counter.
+
+Kernel type
+  The software to run a kernel instance, along with the context in which a
+  kernel starts. One kernel type allows starting multiple, initially similar
+  kernel instances. For instance, one kernel type may be associated with one
+  conda environment containing ``ipykernel``. The same kernel software in
+  another environment would be a different kernel type. Another software package
+  for a kernel, such as ``IRkernel``, would also be a different kernel type.
+
+Kernel provider
+  A Python class to discover kernel types and allow a client to start instances
+  of those kernel types. For instance, one kernel provider might find conda
+  environments containing ``ipykernel`` and allow starting kernel instances in
+  these environments.

docs/kernels.rst

@@ -6,7 +6,7 @@ Making kernels for Jupyter
 
 A 'kernel' is a program that runs and introspects the user's code. IPython
 includes a kernel for Python code, and people have written kernels for
-`several other languages <https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages>`_.
+`several other languages <https://github.com/jupyter/jupyter/wiki/Jupyter-kernels>`_.
 
 When Jupyter starts a kernel, it passes it a connection file. This specifies
 how to set up communications with the frontend.
@@ -132,6 +132,13 @@ JSON serialised dictionary containing the following keys and values:
   is found, a kernel with a matching `language` will be used.
   This allows a notebook written on any Python or Julia kernel to be properly associated
   with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's.
+- **interrupt_mode** (optional): May be either ``signal`` or ``message`` and
+  specifies how a client is supposed to interrupt cell execution on this kernel,
+  either by sending an interrupt ``signal`` via the operating system's
+  signalling facilities (e.g. `SIGINT` on POSIX systems), or by sending an
+  ``interrupt_request`` message on the control channel (see
+  :ref:`msging_interrupt`). If this is not specified
+  the client will default to ``signal`` mode.
 - **env** (optional): A dictionary of environment variables to set for the kernel.
   These will be added to the current environment variables before the kernel is
   started.

docs/messaging.rst

@@ -21,7 +21,7 @@ Versioning
 
 The Jupyter message specification is versioned independently of the packages
 that use it.
-The current version of the specification is 5.2.
+The current version of the specification is 5.3.
 
 .. note::
    *New in* and *Changed in* messages in this document refer to versions of the
@@ -959,6 +959,27 @@ Message type: ``shutdown_reply``::
    socket, they simply send a forceful process termination signal, since a dead
    process is unlikely to respond in any useful way to messages.
 
+.. _msging_interrupt:
+
+Kernel interrupt
+----------------
+
+In case a kernel can not catch operating system interrupt signals (e.g. the used
+runtime handles signals and does not allow a user program to define a callback),
+a kernel can choose to be notified using a message instead. For this to work,
+the kernels kernelspec must set `interrupt_mode` to ``message``. An interruption
+will then result in the following message on the `control` channel:
+
+Message type: ``interrupt_request``::
+
+    content = {}
+
+Message type: ``interrupt_reply``::
+
+    content = {}
+
+.. versionadded:: 5.3
+
 
 Messages on the IOPub (PUB/SUB) channel
 =======================================

jupyter_client/_version.py

@@ -1,5 +1,5 @@
 version_info = (5, 1, 0)
 __version__ = '.'.join(map(str, version_info))
 
-protocol_version_info = (5, 2)
+protocol_version_info = (5, 3)
 protocol_version = "%i.%i" % protocol_version_info