Command line execution of a jupyter notebook fails in default Anaconda 4.1.

[ijstokes]

From @rtrhd on July 2, 2016 15:30

Command line execution of a jupyter notebook with jupyter nbconvert --to notebook --execute fails after upgrading to Anaconda 4.1.

On saving a jupyter ipython notebook the kernel name is changed from the conventional python2 or python3 to the conda specific form Python [Root] but this is not recognised by nbconvert.

This form is also not recognised when the notebook is opened in the browser in an Anaconda 4.0 installation causing problems sharing notebooks.

The behaviour was observed on OSX/3.5 and Linux/2.7.

Explicit removal of the package nb_conda_kernels resolves the problem, but this package is included by default in Anaconda 4.1.

jupyter nbconvert --to notebook --execute test_python_root.ipynb
[NbConvertApp] Converting notebook test_python_root.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: Python [Root]
Traceback (most recent call last):
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/kernelspec.py", line 173, in get_kernel_spec
    resource_dir = d[kernel_name.lower()]
KeyError: 'python [root]'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ron/anaconda3/bin/jupyter-nbconvert", line 6, in <module>
    main()
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 596, in launch_instance
    app.start()
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/nbconvertapp.py", line 293, in start
    self.convert_notebooks()
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/nbconvertapp.py", line 457, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/nbconvertapp.py", line 428, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/nbconvertapp.py", line 357, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/exporters/exporter.py", line 165, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/exporters/exporter.py", line 183, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/exporters/notebook.py", line 26, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/exporters/exporter.py", line 130, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/exporters/exporter.py", line 302, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb,resources)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/nbconvert/preprocessors/execute.py", line 141, in preprocess
    cwd=path)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/manager.py", line 429, in start_new_kernel
    km.start_kernel(**kwargs)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/manager.py", line 230, in start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/manager.py", line 170, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/manager.py", line 82, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/Users/ron/anaconda3/lib/python3.5/site-packages/jupyter_client/kernelspec.py", line 175, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named Python [Root]

Copied from original issue: ContinuumIO/anaconda-issues#877

[ijstokes]

From @bollwyvl on July 5, 2016 15:54

In the near term, the kernel_name (in the current environment) can be specified at the CLI:

jupyter nbconvert Notebook.ipynb --to=html --ExecutePreprocessor.kernel_name=python

Ideally, we'd be able to overload the kernel manager in nbconvert as we have done in the notebook. However, it looks like there is no way to override the kernel discovery of the --execute flag with configuration.

In the case of root, we should likely not append the name... or better still, figure out a less-invasive way to do it.

[ijstokes]

Ack, this is pretty bad. We need to figure out a much better solution to this problem. nbconvert is a thing lots of people do. I'm going to move this to nb_conda_kernel repo where the action will take place (and where the problem originates).

[dhimmel]

I use nb_conda_kernels and I'd like to execute notebooks in place using nbconvert, since I have pipelines of notebooks and would like to execute these programatically. Unfortunately the --ExecutePreprocessor.kernel_name=python workaround is not working for me.

Here are the steps to reproduce. Define the following environment.yml:

name: conda-test
dependencies:
- conda-forge::nb_conda_kernels=2.1.0
- ipykernel=4.5.2
- ipywidgets=6.0.0
- nbconvert=5.1.1
- notebook=5.0.0

Install and activate the conda-test environment with:

conda env create --file environment.yml
source activate conda-test

Create a notebook with a single cell as follows, and use the kernel chooser menu to choose Python [conda env:conda-test]

import sys
sys.base_prefix

Running from an interactive Jupyter browser, the cell outputs:

'/home/dhimmel/anaconda3/envs/conda-test'

That is the correct version of python. With the environment activated, run:

jupyter nbconvert --inplace --execute test.ipynb

Producing the following output (which is this issue):

jupyter_client.kernelspec.NoSuchKernel: No such kernel named conda-env-conda-test-py

However, if I try the workaround and run:

jupyter nbconvert --inplace --execute --ExecutePreprocessor.kernel_name=python test.ipynb

The cell outputs:

'/home/dhimmel/anaconda3/envs/integrate'

Ahh! This is another one of my environments, not the environment which is activated. I don't believe this used to be an issue... IIRC, when a conda environment was activated, nbconvert would use that kernel. Perhaps this issue was comorbid with jupyter/jupyter#245.

[damianavila]

Mmm... I tried you command and I am getting the correct prefix, the conda-test one.
Are you sure you are not in the integrate environment when you run the workaround?

[dhimmel]

Are you sure you are not in the integrate environment when you run the workaround?

Yes certain. However, I was using the integrate environment around the time of jupyter/jupyter#245. Therefore, I suspect there is some configuration that differs between us and is causing the problem.

@damianavila do you know which configuration files determine kernels for nbviewer? If you are okay with uploading your configuration files, I can compare them to mine. One issue is that I've struggled to learn which config files do what wrt to kernels.

[damianavila]

@damianavila do you know which configuration files determine kernels for nbviewer?

nbviewer does not run any kernel, AFAIK... and the kernel name gets saved in the notebook document itself...

If you are okay with uploading your configuration files, I can compare them to mine. One issue is that I've struggled to learn which config files do what wrt to kernels.

Which configuration files do you want me to post? There are several ones... for kernels installed manually by yourself, you should specifically look is some directories to check the kernelspec, for instance:

damian-S400CA :: ~ :: 
$ ls -la /home/damian/miniconda3/envs/conda-test/share/jupyter/kernels/
total 12
drwxrwxr-x 3 damian damian 4096 may  2 11:34 .
drwxrwxr-x 4 damian damian 4096 may  2 11:34 ..
drwxrwxr-x 2 damian damian 4096 may  2 11:34 python3
damian-S400CA :: ~ :: 
$ cat /home/damian/miniconda3/envs/conda-test/share/jupyter/kernels/python3/
kernel.json     logo-32x32.png  logo-64x64.png  
damian-S400CA :: ~ :: 
$ cat /home/damian/miniconda3/envs/conda-test/share/jupyter/kernels/python3/kernel.json 
{
 "argv": [
  "/media/data/miniconda3/envs/conda-test/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

[dhimmel]

@damianavila the conda-test kernel looks the same between us:

cat ~/anaconda3/envs/conda-test/share/jupyter/kernels/python3/kernel.json

{
 "argv": [
  "/home/dhimmel/anaconda3/envs/conda-test/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

Installed kenels

Here's the Jupyter kernel documentation, which was helpful for the following sections. I ran jupyter kernelspec list:

Available kernels:
  cognoma-machine-learning    /home/dhimmel/.local/share/jupyter/kernels/cognoma-machine-learning
  python3                     /home/dhimmel/.local/share/jupyter/kernels/python3

I must have installed the cognoma-machine-learning environment before I used nb_conda_kernels. therefore, I removed that environment with jupyter kernelspec remove cognoma-machine-learning. Then, through jupyter --paths, I found the directory and located the offending kernel specification ~/.local/share/jupyter/kernels/python3/kernel.json, which contained:

{
 "language": "python",
 "argv": [
  "/home/dhimmel/anaconda3/envs/integrate/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3"
}

I edited kernel.json so the first element in argv was /home/dhimmel/anaconda3/bin/python (my conda root environment's python). @damianavila, which python do you have here?

Now when I rerun nbconvert with --ExecutePreprocessor.kernel_name=python, I get '/home/dhimmel/anaconda3'. And when I run it without --ExecutePreprocessor.kernel_name with the activated environment, I still get a No such kernel named conda-env-conda-test-py error.

Mmm... I tried you command and I am getting the correct prefix, the conda-test one.

So the question is why is @damianavila getting the correct prefix while I am not. Perhaps I am using newer versions of Jupyter / conda / anaconda (since I don't think this bug used to exist)?

@ijstokes any advice? My goal is to use nbconvert to execute a notebook that uses a specified conda environment with nb_conda_kernels.

[damianavila]

Suggestion: remove any kernelspec from /.local/share/jupyter. That is probably clobbering the kernel in you environment...

If I run jupyter kernelspec list I see:

$ jupyter kernelspec list
Available kernels:
  python3    /media/data/miniconda3/envs/conda-test/share/jupyter/kernels/python3

Which is the python executable in you conda-test environment.

[dhimmel]

That is probably clobbering the kernel in you environment

@damianavila yes! I ran jupyter kernelspec remove python3. Now the --ExecutePreprocessor.kernel_name=python workaround works. The notebook cell outputs '/home/dhimmel/anaconda3/envs/conda-test'.

Without the conda-test environment activated, running jupyter kernelspec list now returns:

Available kernels:
  python3    /home/dhimmel/anaconda3/lib/python3.6/site-packages/ipykernel/resources

With conda-test activated, jupyter kernelspec list now returns:

Available kernels:
  python3    /home/dhimmel/anaconda3/envs/conda-test/share/jupyter/kernels/python3

[damianavila]

@dhimmel nice to see it is working for you 😉

[mcg1969]

I'm closing this as a duplicate of #77. I suppose I could close that one instead, I offered some comments in that issue that are at least relevant. I invite comments on the wisdom of my idea.