Environment inconsistent between IPython and Jupyter notebook launched in environment #2953
Comments
|
Let me detail the issue more carefully. Thank you @brendancol for making the initial note of it. From a shell: 504-Dask % activate py2-env
discarding /Users/dmertz/anaconda/bin from PATH
prepending /Users/dmertz/anaconda/envs/py2-env/bin to PATH
(py2-env)505-Dask % ipython
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jun 29 2016, 11:09:23)
Type "copyright", "credits" or "license" for more information.
IPython 4.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from dask.distributed import Executor, progress
In [2]:
Do you really want to exit ([y]/n)? y
(py2-env)506-Dask % jupyter notebook
[I 22:02:28.843 NotebookApp] Serving notebooks from local directory: /Users/dmertz/Projects/training-products/Dask
[I 22:02:28.844 NotebookApp] 0 active kernels
[I 22:02:28.844 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/
[I 22:02:28.844 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).Within the launched notebook environment I see this:
For completeness, here are the kernel messages: [W 22:03:14.947 NotebookApp] 404 GET /nbextensions/IPython-notebook-extensions-3.x/usability/runtools/main.js?v=20160630220228 (::1) 12.78ms referer=http://localhost:8888/notebooks/07_Distributed_Custom.ipynb
[I 22:03:15.481 NotebookApp] Kernel started: 16078bbd-6c08-4cdd-9999-80bbbe652114
/Users/dmertz/anaconda/envs/py27/lib/python2.7/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
"You should import from ipykernel or jupyter_client instead.", ShimWarning)
[I 22:03:48.729 NotebookApp] Kernel restarted: 16078bbd-6c08-4cdd-9999-80bbbe652114
/Users/dmertz/anaconda/envs/py27/lib/python2.7/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
"You should import from ipykernel or jupyter_client instead.", ShimWarning)
[I 22:05:15.465 NotebookApp] Saving file at /07_Distributed_Custom.ipynb
^C[I 22:08:55.988 NotebookApp] interrupted
Serving notebooks from local directory: /Users/dmertz/Projects/training-products/Dask
1 active kernels
The Jupyter Notebook is running at: http://localhost:8888/
Shutdown this notebook server (y/[n])? y
[C 22:08:57.306 NotebookApp] Shutdown confirmed
[I 22:08:57.307 NotebookApp] Shutting down kernels
[I 22:08:57.726 NotebookApp] Kernel shutdown: 16078bbd-6c08-4cdd-9999-80bbbe652114My environment looks like this: (py2-env)507-Dask % conda env export
name: py2-env
dependencies:
- appnope=0.1.0=py27_0
- backports=1.0=py27_0
- backports.shutil_get_terminal_size=1.0.0=py27_0
- backports_abc=0.4=py27_0
- bokeh=0.12.0=py27_0
- boto3=1.3.1=py27_0
- botocore=1.4.28=py27_0
- click=6.6=py27_0
- cloudpickle=0.2.1=py27_2
- configparser=3.5.0b2=py27_1
- dask=0.10.0=py27_1
- decorator=4.0.10=py27_0
- distributed=1.11.0=py27_0
- docutils=0.12=py27_2
- entrypoints=0.2.2=py27_0
- freetype=2.5.5=1
- functools32=3.2.3.2=py27_0
- futures=3.0.5=py27_0
- get_terminal_size=1.0.0=py27_0
- ipykernel=4.3.1=py27_0
- ipython=4.2.0=py27_1
- ipython_genutils=0.1.0=py27_0
- ipywidgets=4.1.1=py27_0
- jinja2=2.8=py27_1
- jmespath=0.9.0=py27_0
- jsonschema=2.5.1=py27_0
- jupyter=1.0.0=py27_3
- jupyter_client=4.3.0=py27_0
- jupyter_console=4.1.1=py27_0
- jupyter_core=4.1.0=py27_0
- libpng=1.6.22=0
- locket=0.2.0=py27_1
- markupsafe=0.23=py27_2
- mistune=0.7.2=py27_1
- mkl=11.3.3=0
- msgpack-python=0.4.6=py27_0
- nbconvert=4.2.0=py27_0
- nbformat=4.0.1=py27_0
- notebook=4.2.1=py27_0
- numpy=1.11.0=py27_1
- openssl=1.0.2h=1
- pandas=0.18.1=np111py27_0
- partd=0.3.4=py27_0
- path.py=8.2.1=py27_0
- pathlib2=2.1.0=py27_0
- pexpect=4.0.1=py27_0
- pickleshare=0.7.2=py27_0
- pip=8.1.2=py27_0
- psutil=4.3.0=py27_0
- ptyprocess=0.5.1=py27_0
- pygments=2.1.3=py27_0
- pyqt=4.11.4=py27_3
- python=2.7.12=0
- python-dateutil=2.5.3=py27_0
- python.app=1.2=py27_4
- pytz=2016.4=py27_0
- pyyaml=3.11=py27_4
- pyzmq=15.2.0=py27_1
- qt=4.8.7=3
- qtconsole=4.2.1=py27_0
- readline=6.2=2
- requests=2.10.0=py27_0
- s3fs=0.0.6=py27_0
- setuptools=23.0.0=py27_0
- simplegeneric=0.8.1=py27_1
- singledispatch=3.4.0.3=py27_0
- sip=4.16.9=py27_0
- six=1.10.0=py27_0
- sqlite=3.13.0=0
- ssl_match_hostname=3.5.0.1=py27_0
- tblib=1.3.0=py27_0
- terminado=0.6=py27_0
- tk=8.5.18=0
- toolz=0.8.0=py27_0
- tornado=4.3=py27_1
- traitlets=4.2.1=py27_0
- wheel=0.29.0=py27_0
- yaml=0.1.6=0
- zlib=1.2.8=3
- pip:
- backports-abc==0.4
- backports.shutil-get-terminal-size==1.0.0
- backports.ssl-match-hostname==3.5.0.1
- ipython-genutils==0.1.0
- jupyter-client==4.3.0
- jupyter-console==4.1.1
- jupyter-core==4.1.0 |
|
@quasiben notes this:
However:
|
|
@DavidMertz within |
|
|
Doesn't Jupyter have some configuration file somewhere that tells it where to launch a kernel? Perhaps that has been configured somehow to go to |
|
I think this is related to these: #2548, jupyter/notebook#1508 Jupyter relies on files outside of the conda environment system for configuration. If that is what is causing the mismatched kernel here, there is nothing conda itself can do about it. It suggests instead that somehow we need to patch Jupyter itself to play nicely with virtual environments (not just conda's). |
|
In particular: what does your directory |
|
Nothing indicating suspicious paths: (py2-env)519-Dask % tree ~/.jupyter/
/Users/dmertz/.jupyter/
├── migrated
└── nbconfig
├── edit.json
└── notebook.json
1 directory, 3 files
(py2-env)520-Dask % find ~/.jupyter -type f -print0 | xargs -0 -I '{}' sh -c "echo; echo {}; cat {}"
/Users/dmertz/.jupyter/migrated
2016-01-09T18:52:56.588665
/Users/dmertz/.jupyter/nbconfig/edit.json
{
"Editor": {
"codemirror_options": {
"vimMode": false,
"keyMap": "default"
}
}
}
/Users/dmertz/.jupyter/nbconfig/notebook.json
{
"calico-spell-check": true,
"load_extensions": {
"calico-spell-check": true,
"IPython-notebook-extensions-3.x/usability/runtools/main": true,
"calico-cell-tools": true,
"calico-document-tools": true
},
"calico-cell-tools": true,
"calico-document-tools": true |
|
Also trying to rule out one more thing: |
|
Can you post the full output of
|
|
The entire output from my run of In particular, the "Native kernel not available" message you show does not occur for me. |
|
@bollwyvl, would you by any chance have an idea what's going on? I must be frank I'm still not convinced this is a conda issue. But what can we do to determine that with some certainty? |
|
@DavidMertz (i think will fill in the details) but we some latent kernel files lying around in |
|
I think the right place to put any such "help" is in one of the Jupyter-related conda packages. I'm thinking of perhaps a post-link script that searches for these "external" influences on Jupyter and warns the user about them. |
|
Skipping some intermediate dead ends we tried, here's the actual issue. I might be in a conda environment like E.g. within the notebook: In [1] import sys; sys.path[:3]
Out [1]
['',
'/Users/dmertz/anaconda/envs/py27/lib/python27.zip',
'/Users/dmertz/anaconda/envs/py27/lib/python2.7']It's finding this information from here: % cat /usr/local/share/jupyter/kernels/python2/kernel.json
{
"display_name": "Python 2",
"language": "python",
"argv": [
"/Users/dmertz/anaconda/envs/py27/bin/python",
"-m",
"IPython.kernel",
"-f",
"{connection_file}"
]
}
% ls -o /usr/local/share/jupyter/kernels/python2/kernel.json
-rw-r--r-- 1 dmertz 182 Aug 19 2015 /usr/local/share/jupyter/kernels/python2/kernel.jsonIf I move |
|
Thanks for the update, David. I find it interesting that |
|
A wrinkle here is that I want to be able to launch other kernels from within a Notebook (and my students want that capability). So when I open (usually) a Python 3 notebook in an environment with Py3.5, it's nice to have the menu to launch other notebooks that use R, Julia, Python2, or other kernels, as I can do currently. But at the very least, the notebook I open from a command shell should preferentially use the current environment, even if I have the option of opening other kernels within the Jupyter navigation page. This sort of makes me think that I think this behavior makes sense because there is no canonical "right" Python2 kernel to use for the "New/Notebook/Python 2" menu... and the one I launched Jupyter from seems like a better answer than "the one that got configured somehow a year ago." |
|
Yeah, I just can't see this; and while I'm not the decisionmaker here, I would have to voice my opposition to it to those who are. It requires giving conda a special understanding of a specific project, counter to its long-term goal of being platform/language agnostic. It would require The flaw here is quite clearly Jupyter's. It needs to learn to play nicer with virtual environments. (And not just conda virtual environments.) At the very least, it needs to intelligently merge the local configuration information in On the other hand, I could see a way to resolve most of my objections and still provide the specific This still gives conda (or more accurately, conda packages) the ability to modify files outside of its installed environment, something I am definitely not wild about. But we already provide that with post-link scripts. |
|
@kalefranz now that the cause of this issue is clear, what do you think? |
|
I think I'm driving and shouldn't be reading this right now 😬😱 Will catch up soon. |
|
This issue might have been fixed for us. Flying now and typing on tablet, so can't verify. But I installed the latest Anaconda 4.1.0, with a newer Jupyter in the bundle. I noticed in the New/Notebook menu it now shows all my environments, instead of simply (some unspecified) Python 2. This hints that they have learned a better strategy for identifying the best Python path to use. I'll check more carefully tomorrow or tonight. |
Just to give you more context, this is something we (CIO) do... not Jupyter itselft (even when I am Jupyter dev, I originally wrote this on CIO time)... and it is called |
|
Cool feature @damianavila! Unfortunately, I'm not sure what makes it appear. I thought it should be an update to some Jupyter package, but that doesn't seem to be the case exactly. Or maybe only the py35_3 packages but not the py27_* versions. Under root, I get the cool new New/Notebook behavior that shows all the environments. However, under py2-env I only get the old kernels listed, and also get the wrong kernel chosen for opening an existing notebook, per I hadn't yet tried to upgrade Jupyter in % diff py2-env-jupyter-versions root-env-jupyter-versions
1c1
< jupyter 1.0.0 py27_3 defaults
---
> jupyter 1.0.0 py35_3 defaults
5,7c5,7
< jupyter_client 4.3.0 py27_0 defaults
< jupyter_console 4.1.1 py27_0 defaults
< jupyter_core 4.1.0 py27_0 defaults
---
> jupyter_client 4.3.0 py35_0 defaults
> jupyter_console 4.1.1 py35_0 defaults
> jupyter_core 4.1.0 py35_0 defaultsThe identical versions of everything obviously Jupyter related is installed both places, except that one is the py35 packages, the other the py27 ones. Maybe the extension has a non-obvious name, and it's possible to install or update that to get the right behavior? |
|
Oh... Doh! @damianavila already mentioned it was called (py2-env)538-tmp % conda list | grep nb_conda_kernels
nb_conda_kernels 1.0.3 py27_0 defaultsThe bad news is that after doing this install, I neither get the selection of all the environment kernels, nor open existing notebooks using the current environment. So while a cool feature, Damien's extension does not cure this particular problem (nor do ANYTHING under the old environment!?) |
This is probably because the nb_conda_kernels from repo.cio do not have the post-script step to enable the extension... and this is one asymmetry we need to cover @bollwyvl... see the Notebook flow for more details, I have written some of my thoughts there (I was not following the latest developments on this front because I was assigned to other tasks). Essentially, you need to enable the extension on that specific environments if you want to see the extension working from a notebook server living on that environment... something like this should help, in the py2-env run:
where PREFIX is the prefix of your env... This step is done in post-script in our original recipes and by a metapackage in the latest Anaconda Distribution... so you are getting the worst experience installing it because you are installing from repo.cio and this is why I say there is an asymmetry we need to fix somehow... I can help you a little more during the week. |
|
Excellent progress... albeit not perfect. If I do this: % python -m nb_conda_kernels.install --enable --prefix /Users/dmertz/anaconda/envs/py2-envThen I get the extension enabled. While an extra step, this at least seems like something we can document (and that I can teach and know). Launching An arguable flaw is that when a notebook is actually opened (either via navigating within the notebook file manager or by launching one from the command line directly), I get the below dialog. I can select the environment I used to open Jupyter from here, but I feel like it ought to actually default to that instead.
|
That's because you have that notebook saved with the kernel name pointing to the previous python2 name... you should not see that with new notebook nor after saving the old ones with the new kernel you select... probably we should try to deal with this in a better way and let the people use one of the python envs available.... see the discussion here for more thoughts: anaconda/nb_conda_kernels#11 (comment) |
|
I think this has some relationship to this community-created issue around non-portable Kernel names that encode the environment into the kernel meta-data entry: ContinuumIO/anaconda-issues#877 But for @mcg1969 's sake, I would definitely say that this is not a Conda issue, but rather a combination of Jupyter configuration complexity (e.g. jupyter/notebook#1508) and how we implement the extension. I'm going to move it to the |
|
This issue was moved to anaconda/nb_conda_kernels#33 |
|
Hi there, thank you for your contribution to Conda! This issue has been automatically locked since it has not had recent activity after it was closed. Please open a new issue if needed. |
See https://github.com/ContinuumIO/anaconda-cluster/issues/1022 for the details (I'll copy them over below too). We initially thought the error was elsewhere than where it actually applies.
The text was updated successfully, but these errors were encountered: