pip install --editable and pip install clash for namespace packages

[mitsuhiko]

A namespace package installed editable and another one installed regularly don't work well together.

[harobed]

Same issue here :(

[harobed]

I've posted a code to test this bug : http://public.hg.stephane-klein.info/hgwebdir.cgi/test_pip_namespace_bug_3/

[carljm]

The root of the problem here is that setuptools includes two methods for making namespace packages work: the __init__.py method (which is documented, and usable by humans), and the ...-nspkg.pth file method, which is only used with --single-version-externally-managed (which pip uses). The two methods are not compatible with each other.

After some discussion with mitsuhiko and others on IRC, it seems the best (partial) solution for pip is to have "pip install -e" add a modified version of the standard setuptools ...nspkg.pth file for each develop-installed package. (The necessary modifications are to use the develop egg-link path instead of sitedir, and to skip the check for init.py entirely, since a develop-installed namespace package will have an init.py). This means pip will at least be compatible with itself (pip-installed and pip-editable-installed namespace packages will work together). Pip-installed and setuptools/distribute-installed namespace packages will not be compatible.

[embray]

I recently got bitten by this too. Another problem caused by it is that since an __init__.py isn't installed for the namespace package (assuming you've only installed one package in that namespace, and it was installed --single-version-externally managed) it breaks nose's module search. Maybe that's nose's fault, but I still think it's reasonable to expect if a directory doesn't contain an __init__.py that it's not a Python package.

I think pip should somehow just kill the nspkg.pth thing altogether and install the __init__.py. As long as the package itself is designed correctly (i.e. the __init__.py is empty except for extend_path/declare_namespace) it shouldn't be a problem. --single-version-externally-managed was designed with system packagers in mind, but they're not going to be using pip in the first place. So I wonder if there's some way to disable this behavior altogether without being too intrusive.

[tseaver]

+1 for switching away from '--single-version-externally-managed': that option is not intended for pip's use cases at all. If pip can't do at least as good a job installing stuff as easy_install, what is its purpose?

[carljm]

Switching away from --single-version-externally-managed entirely is not going to happen; flat installs are a feature. Switching away from it in the case of namespace packages only is a possibility I'd consider, to avoid these issues. I don't like it, but there doesn't seem to be any great option, given the inherent incompatibility between the two types of namespace package support built in to setuptools/distribute.

Saying that --single-version-externally-managed is "not intended for pip's use cases" is somewhere between wrong and a red herring. The flag is intended to allow flat single-version installations, managed by some tool other than easy_install. That's precisely what pip uses it for; the fact that the setuptools author originally (and incorrectly) thought that only system packagers would be interested in such a feature is irrelevant.

The breakage here is an inherent bug in the technique setuptools uses for namespace packages with that flag, and the bug is just as present when the flag is used by its originally intended audience, system packagers (I first saw the bug in interaction between a "setup.py develop" installed package and a system-package-installed namespace package).

[mkhattab]

I'm experiencing this bug as well. I guess this is one of those unresolvable bugs that's here to stay. Oh well.

[carljm]

I would accept a patch to avoid --single-version-externally-managed when a namespace package is involved, which I think would solve this issue. I don't currently have any plans to write that patch.

[k4ml]

Why not an option that we can pass to pip install not to use --single-version-externally-managed ? I test by commenting out that in ./pip/req.py:568 and all packages now installed into the egg dir, just like easy_install does. Sometime I want to do this if I used both pip and easy_install so that my site-packages does not mix with some packages install flat and some not.

[carljm]

I don't see a problem with a --egg option to opt out of --single-version-externally-managed.

[k4ml]

k4ml@93cd6b1

Can someone comment if I'm moving into the right direction ? This is my first time hacking on pip code so this is based on my few minutes glimpse into part of it, just to get pip install --egg somepackages install everything into single egg dir.

[carljm]

Can someone comment if I'm moving into the right direction ?

Looks good to me, just needs a test. Tests are mostly high-level
integration tests using ScriptTest, should be easy to write one based on
existing examples. In this case you'll want to just test that installing
a sample package results in a .egg file, not a flat installation, in
site-packages. You should install a package from the local filesystem,
not the network: tests/packages/FSPkg will probably work fine. Adding
the test to tests/test_basic.py is ok.

Thanks!

[k4ml]

Pull requests - #541

I'm also thinking of making it possible to set this flag in pip.conf. What you think about that ?

[carljm]

I'm also thinking of making it possible to set this flag in pip.conf. What you think about that ?

Let's discuss on the pull request.

[carljm]

Merged pull request #541, which provides one workaround. Thanks @k4ml!

[qwcode]

Notice: the --egg option, which was added to fix this issue may potentially be removed, with no alternative workaround offered. See discussion in #1749

[Ivoz]

Here is a script to actually reproduce this issue

https://gist.github.com/Ivoz/d9bff05069e0ec53e6ea

[underrun]

i'm not a huge fan of the --egg solution, but there needs to be a less cumbersome workaround for this.

without a workaround, while developing, I can't use --editable once and then just edit and test. every time I save my code I've got to run pip install -I --no-deps . which gets really old and is fragile (read: sometimes I forget).

part of the problem with the way this entire issue manifests is that it is silent. you just can't import a module even though it is installed and pip tells you it is there.

it would be a positive step just to make pip complain if trying to install editable packages as part of a namespace where a non editable package in that namespace is already installed. or alternately to complain if trying to install a non editable package that is part of a namespace where an editable package in that namespace is already installed.

another option, if we actually want to make namespace packages and editable installs always work, could be that if a namespace package is installed as editable, pip could reorganize all the other packages in that namespace as editable along side the desired package.

[hongqn]

I submitted 2 proposals to fix this issue in setuptools https://bitbucket.org/pypa/setuptools/issue/250/develop-and-install-single-version. FYI.

[achimnol]

Just yet another bump here!

[amcgregor]

I'll just mention that every single marrow package utilizes namespaces (some packages populating up to four or five) and that this particular issue has been a bit of a bane of my existence when it comes to helping new users get set up with development tools. I've got just shy of 60 packages using this. The entry_points note is also interesting, as virtually every package contributing to a namespace also contributes entry_points.

The pip approach only works, in my experience, if every package is installed, editable, from disk, that cooperates on the namespace. If one package is installed non-editable via pip, the whole ball of yarn begins to unravel with some very, very obtuse symptoms for new users. (Such as intermittently importable modules; a general sanity check of importing the parent namespace and verifying namespace.__path___ has rapidly identified the culprit borked package in testing.) Mixing properly pip installed packages and pure setup.py develop'd source ones (avoiding pip) appears, however, to work.

We also seem to be able to get into strange situations where a single namespace-contributing package can be installed three different ways, sometimes all simultaneously. (Repeated calls to pip uninstall, each finding more files, is as hilarious to see as it is unfortunate.) Dependency installation when using pip install -e appears to be inconsistent, as well. In a majority of cases we end up with properly unpacked namespaces (installed), pth-link files (installed editable), and in one case we somehow managed to get a zipped .egg installed, despite zip_safe being False on that dependent package and no .egg distribution being provided on Pypi.

Frustration with namespace issues reaches a peak after the fourth time a virtual environment is nuked to start over from scratch. ;)

[k4ml]

Not solution but I just put some notes. When mixing between 'editable' packages, namespaced packages and normal packages, I have more luck with buildout + mr.developer.

[ilendir]

While the fixup hack @lelit mentioned seems to work for some packages, it also breaks some packages/builds for us.

After some playing around with pip install and editable mode, I came up with the same idea as @carljm (adding a -nspkg.pth file for every editable package), but nobody seems to have implemented that (or was this the now deprecated --egg option?).

Is this still an issue on newer versions of python with PEP420 implemented? (read: would it help to switch to a newer version of python?)

[lelit]

Using PEP420 style effectively helped me several times now wrt the editable mode.

Unfortunately it comes with its own glitches: for example, setuptools' find_packages does not support it, so my newer packages contains the following hack:

-    packages=find_packages('src'),
+    packages=['toplevel.child.' + package
+              for package in find_packages('src/toplevel/child/')],

[jaraco]

nobody seems to have implemented adding a -nspkg.pth for every editable package.

Check out Setuptools 31 which adds support for this feature, and also co-exists with PEP-420 packages on Python 3.5+.

[theacodes]

For anyone curious, here's an exhaustive list how namespace packages work across pip install, pip install -e, python setup.py install, and python setup.py develop:

https://github.com/jonparrott/namespace-pkg-tests/blob/master/table.md

tl;dr: You can use pip install -e and python setup.py develop as long as every other package in your namespace has been installed using pip.

[dstufft]

I'm going to close this issue. It appears that setuptools 31 has fixed this for our reproduction scripts, and beyond that there isn't anything pip can do here.

[piotr-dobrogost]

@jonparrott
What versions of pip and setuptools were used to obtain results at https://github.com/jonparrott/namespace-pkg-tests/blob/master/table.md ?

[amcgregor]

@dstufft While I appreciate that it's supposedly fixed, I'd like to see @jonparrott 's compatibility table re-run as proof. I hope for the best, plan for the worst on tickets that are nearly 7 years old and have such widely scoped failure scenarios. Confidence is not high given this problem's extensive history, and re-running the nox suite myself, the failures would indicate a lack of actual resolution.

[dstufft]

In @jonparrott's examples, the failing cases could be distilled down to "using python setup.py install directly (excluding the PEP 420 failures on Python 2, which are expected). This is failing even in cases where pip is not involved at all (such as python setup.py install + python setup.py develop).

This ticket was for combinations of pip install . and pip install -e or python setup.py develop.

The graph already posted by @jonparrott bears this out that this ticket is resolved and any remaining issues here is an issue with python setup.py install and not with pip.

[theacodes]

I just re-ran my tests and everything is the same as I initially ran it. I agree with @dstufft's assessment - pip has done everything it can do to resolve this.

@piotr-dobrogost the test uses the latest versions of both. As of this writing, pip 9.0.1 and setuptools 34.3.2.