Editable install does not work with package_dir

[lelit]

I'd like to reduce the depth of my packages working tree, that usually look like

.
├── setup.py
└── src
    └── toplevel
        ├── __init__.py
        └── middle
            ├── __init__.py
            └── leaf
                ├── __init__.py
                ├── ...

where toplevel and often toplevel.middle are namespaces, and given that I mostly target Python 3 these days taking advantage of its implicit namespaces those directories do not even contain an __init__.py.

What I want is a shallow source tree, like

.
├── setup.py
└── src
    ├── __init__.py
    ├── ...

Distutls package_dir allows me to get there immediately, with a setup.py that looks like

from setuptools import setup

setup(name='toplevel.middle.leaf',
      packages=['toplevel.middle.leaf'],
      package_dir={'toplevel.middle.leaf': 'src'},
      namespace_packages=['toplevel', 'toplevel.middle'],
      )

Unfortunately this does not work with /editable/ installation of the package, as you can see by checking out this simple repo and running its runme.sh script: it creates two venvs and installs the silly package in both, one as a plain install and one in develop mode.

It that emits:

$ ./runme.sh 
Unpacking /tmp/bugex
  Running setup.py (path:/tmp/pip-mxukz95n-build/setup.py) egg_info for package from file:///tmp/bugex

Requirement already satisfied (use --upgrade to upgrade): setuptools in ./plain/lib/python3.4/site-packages (from bugex.foo==1.0)
Installing collected packages: bugex.foo
  Running setup.py install for bugex.foo

    Installing /tmp/bugex/plain/lib/python3.4/site-packages/bugex.foo-1.0-nspkg.pth
Successfully installed bugex.foo
Cleaning up...
That is me: bugex.foo
Obtaining file:///tmp/bugex
  Running setup.py (path:/tmp/bugex/setup.py) egg_info for package from file:///tmp/bugex

Requirement already satisfied (use --upgrade to upgrade): setuptools in ./edit/lib/python3.4/site-packages (from bugex.foo==1.0)
Installing collected packages: bugex.foo
  Running setup.py develop for bugex.foo

    Creating /tmp/bugex/edit/lib/python3.4/site-packages/bugex.foo.egg-link (link to .)
    Adding bugex.foo 1.0 to easy-install.pth file

    Installed /tmp/bugex
Successfully installed bugex.foo
Cleaning up...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'bugex'

[xavfernandez]

Might be related to #3

[lelit]

Yes, it may be. The use case is different, though.

[scopatz]

Hello all, I believe that we are seeing this issue with xonsh (xonsh/xonsh#1126). Xonsh relies on implicit namespace packages. This prevents us from recommending that users install with pip install ., and instead must instruct them to use ./setup.py install. It would be great if there was a plan for this to be addressed. Thanks in advance!

[pradyunsg]

Hey! It seems like it's been a while since this issue was active.

Could someone try reproducing this on the current master? If it does still happen, please provide exact steps for how to reproduce. =)

Thanks and sorry for the wait.

[pietrodn]

I saw it happening when i ran python setup.py develop on pymavlink.

[pradyunsg]

@pietrodn Can you check if this is an issue when using pip to install the same package as editable?

[pietrodn]

Sorry but I did not have time to do a complete test until now.
I tried to install the package first with pip, then with setuptools, and I got the same results:

pip

$ mkdir test && cd test
$ pipenv --three
$ pipenv shell  # enter virtualenv
$ git clone https://github.com/ArduPilot/pymavlink.git
$ cd pymavlink
$ git clone https://github.com/mavlink/mavlink.git mavlink  # needed dependency
$ pip install -e .
$ python3 -c "import pymavlink"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pymavlink'

setuptools

$ mkdir test && cd test
$ pipenv --three
$ pipenv shell  # enter virtualenv
$ git clone https://github.com/ArduPilot/pymavlink.git
$ cd pymavlink
$ git clone https://github.com/mavlink/mavlink.git mavlink  # needed dependency
$ python setup.py develop
$ python3 -c "import pymavlink"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pymavlink'

[Sup3rGeo]

I believe the same problem is happening when trying to install allure-python-commons using pip editable. It also uses package_dir.

Not from master though, using pip 9.0.3

[gzagatti]

I am also having the same problem.

I cannot install my library in development mode using the package_dir. My project structure could be reduced to something like this:

project/
├── python
│   └── __init__.py
└── setup.py

Inside my setup.py, I have something like this:

setup(
   package_dir = {'project': 'python'},
   packages = ['project']
)

When I do pip install -e . or python setup.py develop, python will create the project.egg-info/ and the project.egg-link in the site-packages folder. Inside project.egg-info/SOURCES.txt, we find:

...
./python/__init__.py
...

On the other hand when we do pip install . or python setup.py install, python will create the build folder which will contain the directories structured in the way we specify with package_dir, this will be copied to the site-packages and everything will run perfectly. The SOURCE.txt in EGG-INFO will however remain the same as above, which is counterintuitive to me. The repo in site-packages will look like this:

project/
├── project
│   └── __init__.py
└── setup.py

A workaround I found for development purposes is to then create a symlink in the development repository emulating the desired structured declared using package.dir.

[pradyunsg]

@gzagatti What version of pip?

Could you please open a new issue?

[Sup3rGeo]

@pradyunsg why open a new issue if the problem remains the same?

Also, do you need any extra information to reproduce it?

If you want to check this yourself, you can clone https://github.com/allure-framework/allure-python, pip install editable any of the projects and try importing a module.

Basically it would nice to know the status about this one. Thanks!

[RonnyPfannschmidt]

i believe this one is a setuptools issue, as editable installs dont go trough pip and wheels but trough setuptools alone

[jaraco]

The issue is reported in pypa/setuptools#230. It's a long-standing issue awaiting a fix.

[pradyunsg]

Thanks for looking into this @jaraco. I'll close this issue so that the interested people have one place to subscribe to.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.