fix installation using --target #3450

esc · 2016-02-03T10:28:59Z

We encountered a problem on CentOS 7 machines when using the --target option
to install packages that are equipped with c-extensions.

Observe:

me@box ~ $ uname -a
Linux box.d.rz.is 3.10.0-327.4.4.el7.x86_64 #1 SMP Tue Jan 5 16:07:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

me@box ~ $ python --version
Python 2.7.5

me@box ~ $ virtualenv --version
1.10.1

me@box ~ $ virtualenv venv
New python executable in venv/bin/python
Installing Setuptools.......................................................................................................................................................done.
Installing Pip............................................................................................................................................................................................................................................................................done.

me@box ~ $ source venv/bin/activate

(venv)me@box ~ $ mkdir target

(venv)me@box ~ $ ls
target  venv

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev -U pip
Collecting pip
  Downloading https://my_proxy:5000/root/pypi/+f/205/6f553d5b593d3/pip-8.0.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 14.3MB/s
Installing collected packages: pip
  Found existing installation: pip 1.4.1
    Uninstalling pip-1.4.1:
      Successfully uninstalled pip-1.4.1
Successfully installed pip-8.0.2

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target pyyaml
Collecting pyyaml
  Downloading https://my_proxy:5000/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB)
    100% |████████████████████████████████| 372kB 18.6MB/s
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed pyyaml
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpz2LoDs/lib/python'

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target simplejson
Collecting simplejson
  Downloading https://my_proxy:5000/root/pypi/+f/b84/41f1053edd9dc/simplejson-3.8.1.tar.gz (76kB)
    100% |████████████████████████████████| 77kB 19.0MB/s
Installing collected packages: simplejson
  Running setup.py install for simplejson ... done
Successfully installed simplejson
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpEFNOZq/lib/python'

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target markupsafe
Collecting markupsafe
  Downloading https://my_proxy:5000/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz
Installing collected packages: markupsafe
  Running setup.py install for markupsafe ... done
Successfully installed markupsafe
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmp4UuvDJ/lib/python'

(venv)me@box ~ $ ls /tmp/tmp4UuvDJ/
lib64

As you can see using --target first installs into a temporary directory and
then installes into the desired target. Well, at least it tries to: for any
packages that have c-extensions (in this case pyyaml, simplejson and
markupsafe) it incorrectly looks for a lib directory whereas it should be
looking for a lib64 directory. This patch fixes that for us by looking in a
fallback location. We can confirm that this fixes the issue for us.

Please advise.

We encountered a problem on CentOS 7 machines when using the `--target` option to install packages that are equipped with c-extensions. Observe: ``` me@box ~ $ uname -a Linux box.d.rz.is 3.10.0-327.4.4.el7.x86_64 #1 SMP Tue Jan 5 16:07:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux me@box ~ $ python --version Python 2.7.5 me@box ~ $ virtualenv --version 1.10.1 me@box ~ $ virtualenv venv New python executable in venv/bin/python Installing Setuptools.......................................................................................................................................................done. Installing Pip............................................................................................................................................................................................................................................................................done. me@box ~ $ source venv/bin/activate (venv)me@box ~ $ mkdir target (venv)me@box ~ $ ls target venv (venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev -U pip Collecting pip Downloading https://my_proxy:5000/root/pypi/+f/205/6f553d5b593d3/pip-8.0.2-py2.py3-none-any.whl (1.2MB) 100% |████████████████████████████████| 1.2MB 14.3MB/s Installing collected packages: pip Found existing installation: pip 1.4.1 Uninstalling pip-1.4.1: Successfully uninstalled pip-1.4.1 Successfully installed pip-8.0.2 (venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target pyyaml Collecting pyyaml Downloading https://my_proxy:5000/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB) 100% |████████████████████████████████| 372kB 18.6MB/s Installing collected packages: pyyaml Running setup.py install for pyyaml ... done Successfully installed pyyaml Exception: Traceback (most recent call last): File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main status = self.run(options, args) File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run for item in os.listdir(lib_dir): OSError: [Errno 2] No such file or directory: '/tmp/tmpz2LoDs/lib/python' (venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target simplejson Collecting simplejson Downloading https://my_proxy:5000/root/pypi/+f/b84/41f1053edd9dc/simplejson-3.8.1.tar.gz (76kB) 100% |████████████████████████████████| 77kB 19.0MB/s Installing collected packages: simplejson Running setup.py install for simplejson ... done Successfully installed simplejson Exception: Traceback (most recent call last): File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main status = self.run(options, args) File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run for item in os.listdir(lib_dir): OSError: [Errno 2] No such file or directory: '/tmp/tmpEFNOZq/lib/python' (venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target markupsafe Collecting markupsafe Downloading https://my_proxy:5000/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz Installing collected packages: markupsafe Running setup.py install for markupsafe ... done Successfully installed markupsafe Exception: Traceback (most recent call last): File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main status = self.run(options, args) File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run for item in os.listdir(lib_dir): OSError: [Errno 2] No such file or directory: '/tmp/tmp4UuvDJ/lib/python' (venv)me@box ~ $ ls /tmp/tmp4UuvDJ/ lib64 ``` As you can see using `--target` first installs into a temporary directory and then installes into the desired `target`. Well, at least it tries to: for any packages that have c-extensions (in this case pyyaml, simplejson and markupsafe) it incorrectly looks for a `lib` directory whereas it should be looking for a `lib64` directory. This patch fixes that for us by looking in a fallback location. We can confirm that this fixes the issue for us. Please advise.

esc · 2016-02-03T10:29:39Z

cc @snordhausen

esc · 2016-02-03T10:31:30Z

I can spice this up with tests, cleaner code and better error messages if you would principally accept the patch.

esc · 2016-02-03T11:22:44Z

currently failing because pypi is down

esc · 2016-02-09T18:30:56Z

This is green, would appreciate some feedback from the developers, our project is currently blocking on this.

rbtcollins · 2016-02-09T22:13:07Z

I think this will break if things are installed into both purelib and platlib. We shouldn't presume its either-or.

esc · 2016-02-11T10:21:59Z

@rbtcollins that is correct, but this patch would already fix if it were either or, which fixes issues we are having today. If any package installs into purelib and platlib it would be currently broken and would continue to be broken after this PR is merged.

Do you know of any package that installs into purelib and platlib?

esc · 2016-02-11T10:34:35Z

I rewrote the code to copy both purelub and platlib, though I have no idea if it will work or not, an example package would come in handy.

rbtcollins · 2016-02-11T19:02:32Z

Thanks. So - we're about to deprecate the --target option unless we can put together a sensible use case for it - this bug report reminded us how broken it is. Could you perhaps explain the use case you have that leads you to use it?

esc · 2016-02-11T19:20:31Z

Yes, this is quite important when it comes to making AWS Lambda-Functions in Python. In fact -t is explicitly mentioned in the official AWS howto here:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Not having this would cause some serious breakage in our builds and deployments since we rely heavily on it via: https://github.com/ImmobilienScout24/pybuilder_aws_plugin

Granted, the way AWS expects us to package so-called lambda-zips isn't exactly standard, but we would request to keep --target until a suitable alternative method of packaging lambda-zips has been devised.

esc · 2016-02-11T19:26:37Z

I'll fix the PR tomorrow, currently not on the correct device.

rbtcollins · 2016-02-11T19:37:23Z

What about the virtualenv style documented lower on that same page?

esc · 2016-02-11T19:41:52Z

Hmm, that get's a bit messy, since we already have a virtualenv for development and running the build tool, I suppose one could make it work. The nice thing about --target is that we can select the dependencies from the build tool and simply install them into a directory. Also, a virtualenv always includes pip, I suppose.

wm75 · 2016-02-16T15:57:59Z

Please do not deprecate the --target option! Our use case is a command line upgrade command for a package (MiModD) we are maintaining, which uses pip install --target under the hood to grab the correct distribution from pypi and install it into a temporary directory. We then modify the "installed" files before copying them over to the actual location of the currently installed version.
While we could certainly find ways around it, it would mean quite a lot of extra work and would mean that the current version of MiModD would not work when used with later pip versions.

dstufft · 2016-05-18T11:23:44Z

Accidentally closed this, reopening. Sorry!

esc · 2016-05-18T11:25:22Z

You can close this in favour of #3684

BrownTruck · 2016-05-19T11:50:40Z

Hello!

As part of an effort to ease the contribution process and adopt a more standard workflow pip has switched to doing development on the master branch. However, this Pull Request was made against the develop branch so it will need to be resubmitted against master.

If you do nothing, this Pull Request will be automatically migrated by @BrownTruck for you.

If you would like to retain control over this pull request then you should resubmit it against the master branch, closing and referencing the original Pull Request.

If you choose to migrate this Pull Request yourself, here is an example message that you can copy and paste:

We encountered a problem on CentOS 7 machines when using the --target option
to install packages that are equipped with c-extensions.

Observe:

me@box ~ $ uname -a
Linux box.d.rz.is 3.10.0-327.4.4.el7.x86_64 #1 SMP Tue Jan 5 16:07:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

me@box ~ $ python --version
Python 2.7.5

me@box ~ $ virtualenv --version
1.10.1

me@box ~ $ virtualenv venv
New python executable in venv/bin/python
Installing Setuptools.......................................................................................................................................................done.
Installing Pip............................................................................................................................................................................................................................................................................done.

me@box ~ $ source venv/bin/activate

(venv)me@box ~ $ mkdir target

(venv)me@box ~ $ ls
target  venv

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev -U pip
Collecting pip
  Downloading https://my_proxy:5000/root/pypi/+f/205/6f553d5b593d3/pip-8.0.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 14.3MB/s
Installing collected packages: pip
  Found existing installation: pip 1.4.1
    Uninstalling pip-1.4.1:
      Successfully uninstalled pip-1.4.1
Successfully installed pip-8.0.2

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target pyyaml
Collecting pyyaml
  Downloading https://my_proxy:5000/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip (371kB)
    100% |████████████████████████████████| 372kB 18.6MB/s
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed pyyaml
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpz2LoDs/lib/python'

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target simplejson
Collecting simplejson
  Downloading https://my_proxy:5000/root/pypi/+f/b84/41f1053edd9dc/simplejson-3.8.1.tar.gz (76kB)
    100% |████████████████████████████████| 77kB 19.0MB/s
Installing collected packages: simplejson
  Running setup.py install for simplejson ... done
Successfully installed simplejson
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmpEFNOZq/lib/python'

(venv)me@box ~ $ pip install --index-url https://my_proxy:5000/dev/dev --target target markupsafe
Collecting markupsafe
  Downloading https://my_proxy:5000/root/pypi/+f/f5a/b3deee4c37cd6/MarkupSafe-0.23.tar.gz
Installing collected packages: markupsafe
  Running setup.py install for markupsafe ... done
Successfully installed markupsafe
Exception:
Traceback (most recent call last):
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/data/home/me/venv/lib/python2.7/site-packages/pip/commands/install.py", line 357, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmp4UuvDJ/lib/python'

(venv)me@box ~ $ ls /tmp/tmp4UuvDJ/
lib64


As you can see using --target first installs into a temporary directory and
then  installes into the desired target. Well, at least it tries to: for any
packages that have c-extensions (in this case pyyaml, simplejson and
markupsafe) it incorrectly looks for a lib directory whereas it should be
looking for a lib64 directory. This patch fixes that for us by looking in a
fallback location. We can confirm that this fixes the issue for us.

Please advise.

---

*This was migrated from pypa/pip#3450 to reparent it to the ``master`` branch. Please see original pull request for any previous discussion.*

If this pull request is no longer needed, please feel free to close it.

BrownTruck · 2016-05-26T10:52:25Z

This Pull Request has been automatically migrated to #3730 to reparent it to the master branch. Please see the new pull request for any new discussion.

-- fix using pip with --target option (pypa/pip#3056 and pypa/pip#3450)

snordhausen referenced this pull request in Scout24/pybuilder_aws_plugin Feb 8, 2016

Raise exception when 'pip install...' fails

9f457af

xavfernandez added this to the 8.1 milestone Feb 9, 2016

install both from purelib and platlib if they exist

00e286f

esc added 2 commits February 11, 2016 11:39

oups....

34a2d85

fix

35b7319

esc added 2 commits February 12, 2016 09:36

more fix

7f5fd78

moar fixes

154bacb

xavfernandez mentioned this pull request Feb 16, 2016

pip >= 7.1.1 fails to install packages containing extensions when --target option is in use #3365

Closed

dstufft removed this from the 8.1 milestone Mar 4, 2016

This was referenced Apr 22, 2016

Installing "--target" fails #3572

Closed

pip install --target fails #3056

Closed

xavfernandez added the C: target pip install's --target option's behaviour handling label Apr 24, 2016

xavfernandez mentioned this pull request May 13, 2016

Fixed #3682 : pip install --target ignores platlib directories #3684

Closed

dstufft closed this May 18, 2016

dstufft reopened this May 18, 2016

BrownTruck added the asked to reparent label May 19, 2016

BrownTruck mentioned this pull request May 26, 2016

fix installation using --target #3730

Closed

BrownTruck added migrated to master and removed asked to reparent labels May 26, 2016

BrownTruck closed this May 26, 2016

janhybs added a commit to flow123d/flow123d that referenced this pull request Jul 20, 2016

Fix pip on centos, add more debug info

f77fa52

-- fix using pip with --target option (pypa/pip#3056 and pypa/pip#3450)

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 3, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix installation using --target #3450

fix installation using --target #3450

esc commented Feb 3, 2016 •

edited by BrownTruck

Loading

esc commented Feb 3, 2016

esc commented Feb 3, 2016

esc commented Feb 3, 2016

esc commented Feb 9, 2016

rbtcollins commented Feb 9, 2016

esc commented Feb 11, 2016

esc commented Feb 11, 2016

rbtcollins commented Feb 11, 2016

esc commented Feb 11, 2016

esc commented Feb 11, 2016

rbtcollins commented Feb 11, 2016 via email

esc commented Feb 11, 2016

wm75 commented Feb 16, 2016

dstufft commented May 18, 2016

esc commented May 18, 2016

BrownTruck commented May 19, 2016

BrownTruck commented May 26, 2016

fix installation using --target #3450

fix installation using --target #3450

Conversation

esc commented Feb 3, 2016 • edited by BrownTruck Loading

esc commented Feb 3, 2016

esc commented Feb 3, 2016

esc commented Feb 3, 2016

esc commented Feb 9, 2016

rbtcollins commented Feb 9, 2016

esc commented Feb 11, 2016

esc commented Feb 11, 2016

rbtcollins commented Feb 11, 2016

esc commented Feb 11, 2016

esc commented Feb 11, 2016

rbtcollins commented Feb 11, 2016 via email

esc commented Feb 11, 2016

wm75 commented Feb 16, 2016

dstufft commented May 18, 2016

esc commented May 18, 2016

BrownTruck commented May 19, 2016

BrownTruck commented May 26, 2016

esc commented Feb 3, 2016 •

edited by BrownTruck

Loading