Forced use of --allow-external and --allow-unverified very inconvenient and not very intuitive

[bladeoflight16]

Recently, I tried to install a module and got the following error message:

(virtualenv) C:\[snip]\trunk>pip install pypyodbc
Downloading/unpacking pypyodbc
  Could not find any downloads that satisfy the requirement pypyodbc
  Some externally hosted files were ignored (use --allow-external pypyodbc to allow).
Cleaning up...
No distributions at all found for pypyodbc
Storing debug log for failure in C:\Users\[snip]\pip\pip.log

I immediately tried to following command and was confused by the output:

(virtualenv) C:\[snip]\trunk>pip install --allow-external pypyodbc
You must give at least one requirement to install (see "pip help install")

A moment of Googling and reading some documention helped clear up the issue and I settled on this command rather than duplicate the package name:

(virtualenv) C:\[snip]\trunk>pip install --allow-all-external pypyodbc
Downloading/unpacking pypyodbc
  Could not find any downloads that satisfy the requirement pypyodbc
  Some insecure and unverifiable files were ignored (use --allow-unverified pypyodbc to allow).
Cleaning up...
No distributions at all found for pypyodbc
Storing debug log for failure in C:\Users\[snip]\pip\pip.log

At this point, I was rather annoyed, and I tried to use an "all" variant of the --allow-unverified argument, which of course didn't exist:

(virtualenv) C:\[snip]\trunk>pip install --allow-all-external --allow-all-unverified pypyodbc

Usage:
  pip install [options] <requirement specifier> ...
  pip install [options] -r <requirements file> ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --allow-all-unverified

I finally got it to work with --allow-unverified pypyodbc, but I feel this was a very poor user experience. I checked, and using --allow-unverified by itself doesn't work:

(virtualenv) C:\[snip]\trunk>pip install --allow-unverified pypyodbc pypyodbc
Downloading/unpacking pypyodbc
  Could not find any downloads that satisfy the requirement pypyodbc
  Some externally hosted files were ignored (use --allow-external pypyodbc to allow).

Having to type out both arguments seems unnecessarily bulky and tedious to me. I understand the value in having the ability to be more specific, but I suspect that in 99% of the use cases where this is an issue, users will want to bypass both. As such, there should be a combined argument that tells pip "I don't care about the security right now. Just install this." I haven't come up with a good name for such an argument, but there should probably be individual and all variants of this argument. If users using such an argument with poor judgement is considered to be an issue, I would point out the "We're all adults here" mentality set forth by the Python community.

Furthermore, I think the error message could be improved. It was not immediately obvious to me that the error message was suggesting I repeat the package name with the argument. It may be worthwhile to suggest an entire command line so that this is clear, or perhaps suggesting the all variant in addition to the package variant will make more clear that the package name is part of the argument.

[argriffing]

I went through this too. More command line examples in the docs would have helped.

[dstufft]

I think it probably makes sense that if you specify --allow-unverified FOO that it implies --allow-external FOO as well.

As far as an --allow-all-unverified that was asked for in #1268 which captures the reasoning behind there not being an all variant of that option and my opinion still is aligned with my posts in that issue.

[flyte]

I was completely stumped by this as well.

When trying to pip install uwsgi==1.2.3:

  Could not find a version that satisfies the requirement uwsgi==1.2.3 (from versions: 1.4.10, 1.4.9, 1.9.1, 1.9.10, 1.9.11, 1.9.12, 1.9.13, 1.9.14, 1.9.15, 1.9.16, 1.9.17.1, 1.9.17, 1.9.18.1, 1.9.18.2, 1.9.18, 1.9.19, 1.9.2, 1.9.20, 1.9.21.1, 1.9.21, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9, 2.0)
  Some externally hosted files were ignored (use --allow-external to allow).

and after I apparently got the allow-external part right

  Could not find a version that satisfies the requirement uwsgi==1.2.3 (from versions: 1.4.10, 1.4.9, 1.9.1, 1.9.10, 1.9.11, 1.9.12, 1.9.13, 1.9.14, 1.9.15, 1.9.16, 1.9.17.1, 1.9.17, 1.9.18.1, 1.9.18.2, 1.9.18, 1.9.19, 1.9.2, 1.9.20, 1.9.21.1, 1.9.21, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9, 2.0)
  Some insecure and unverifiable files were ignored (use --allow-unverified uwsgi to allow).

So I try all of the following:

pip --allow-external install uwsgi==1.2.3
pip install --allow-external uwsgi==1.2.3
pip install --allow-external uwsgi uwsgi==1.2.3
pip install --allow-external uwsgi==1.2.3 uwsgi==1.2.3
pip install uwsgi==1.2.3 --allow-external
pip install --allow-all-external uwsgi==1.2.3
pip install --allow-unverified --allow-external uwsgi==1.2.3
pip install --allow-external uwsgi==1.2.3 --allow-unverified uwsgi==1.2.3 uwsgi==1.2.3

and many more, until I finally figure out the way that works:

pip install --allow-external uwsgi --allow-unverified uwsgi uwsgi==1.2.3

[dstufft]

Yea sorry for the bad message- Part of that is because in some cases we can't tell if there's an unverified file available until after we've allowed external. But in 1.5.1 we'll be switching the message to give a full command, and we'll be switching it so --allow-unverified FOO implies --allow-external FOO.

[flyte]

Thanks 👍

[ZeerDonker]

The error message that pip gives should also give information about the

--allow-all-external

option in case a requirement file is being used.

[bladeoflight16]

I think some examples of how to incorporate these flags into a requirements.txt file are also in order. I'm having quite a bit of trouble figuring it out, despite the comments in that other thread about it. (The trouble may be compounded by the fact that I have more than one requirement in the file.) In fact, it may be worth having a separate documentation page addressing these flags and potential ways of working with them (the reasoning behind them, how to use them at the command line, how to use them in a requirements file, potential risks, and possibly suggestion of downgrading if old behavior is needed).

[qwcode]

I updated the 1.5 docs 4047b4b
If that's enough, let me know.

[dstufft]

Fixed in #1457

[lsemel]

Is there an example of how to update an existing requirements.txt file? I have a long requirements filed pinned to specific versions of a large list of python packages, which now fails to install. What specifically do I need to add to get the versions I need to install installed? Looking at the docs file, it looks like I now have to go through each and every line of this file and find external packages for each package that fails, which is extremely time consuming. How do we get the exact old behavior?

[bladeoflight16]

The only way to actually get the old behavior is to use an old version of pip. As a result of your comment, I have opened #1518. I believe this addresses the difficulty you're experiencing. Feel free to comment about concerns I haven't addressed there.

[lsemel]

Thanks for opening #1518, @bladeoflight16. I've also noticed that older versions of packages are failing under pip 1.5. See http://stackoverflow.com/questions/21386880/cant-install-old-versions-with-pip-anymore for an example of where the current version of the package works and an older one that previously worked now fails. Does this mean that whenever a package version goes out of date, the requirements.txt has to be changed to allow external and unverified? That seems like a maintenance nightmare for any project with more than a handful of packages installed.

[dstufft]

The requirement for --allow-external and/or --allow-unverified has nothing to do with the age or version of the package. It is entirely concerned with where it is hosted in relation to the index server.

[bladeoflight16]

@lsemel To elaborate on dstufft's response, what you could be describing is that some old versions of a package are hosted externally and insecurely, whereas newer versions are hosted non-externally and securely. This would represent a decision by the maintainers of the project (likely in response to this change in pip): they decided to host newer versions in a location that will not trigger this warning, but they opted not to move older versions to the same location. In this scenario, unless the maintainers decide to move all versions of the package to a new location, old versions will continue to require these arguments, whereas versions from now on (not just the newest one) will not.

[exhuma]

I am running into this annoyance as well. We have an internal private Python repo where we push all our libraries and apps. Some applications have more than one requirement, so specifying --allow-external <something> is extremely annoying. I see the value of the security and I am reluctant to put the {}-all variants in there.

I would much rather white-list our entire private repo than do it manually for each package. Also, from our point of view, pypi is "external" and our own repo is "internal", yet it sees it as external... maybe --allow-non-pypi?.,.. but in the end that's only semantics.

[dstufft]

What are you using for your internal repo? The external check I'd supposed to only activate if the repo has opted in to it (as PyPI does) to make it work for internal repos automatically. Perhaps there is a bug turning it on anyways.

On Feb 6, 2014, at 3:40 AM, exhuma notifications@github.com wrote:

I am running into this annoyance as well. We have an internal private Python repo where we push all our libraries and apps. Some applications have more than one requirement, so specifying --allow-external is extremely annoying. I see the value of the security and I am reluctant to put the {}-all variants in there.

I would much rather white-list our entire private repo than do it manually for each package. Also, from our point of view, pypi is "external" and our own repo is "internal", yet it sees it as external... maybe --allow-non-pypi?.,.. but in the end that's only semantics.

—
Reply to this email directly or view it on GitHub.

[exhuma]

Thanks for the quick reply :)

It's just a simple apache web-listing (a remote folder exposed via httpd and +Indexes).

Following this issue, I also just read your blog post discussing requirements.txt vs setup.py. It was quite interesting! I managed to install my application using a requirements file. But it still is annoying to specify the "allow" flags for each dependency from the internal repo.

[dstufft]

Yea something is wrong it sounds like. You're pointing to it with --find-links? Can you show me the html pip is accessing? You can email it to donald@stufft.io if you don't want to publish it. Also can you tell me the exact command you're typing.

[exhuma]

I was actually using dependency_links inside setup.py. So the "old way" of doing things :)

The actual command to install the package was ./env/bin/pip install -e .

Given you post about the requirements file, I am considering re-thinking our deployment process. I understand the idea behind splitting abstract and concrete dependencies, but I dislike it so I avoided it until now.

The actual HTML output looks something like this (cut out some parts for brevity):

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /pyrepo</title>
 </head>
 <body>
<h1>Index of /pyrepo</h1>
<table><tr><th><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr><tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[DIR]"></td><td><a href="/">Parent Directory</a></td><td>&nbsp;</td><td align="right">  - </td><td>&nbsp;</td></tr>
[...]
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="auto_icms-1.7.tar.gz">auto_icms-1.7.tar.gz</a></td><td align="right">21-Jan-2014 14:28  </td><td align="right">2.4M</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.0.1.tar.gz">bitstream-1.0.1.tar.gz</a></td><td align="right">05-Nov-2012 12:06  </td><td align="right">3.3K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.0dev4.tar.gz">bitstream-1.0dev4.tar.gz</a></td><td align="right">25-Sep-2012 16:44  </td><td align="right">3.3K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.1.0.tar.gz">bitstream-1.1.0.tar.gz</a></td><td align="right">23-Nov-2012 14:24  </td><td align="right">3.4K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.2.0.tar.gz">bitstream-1.2.0.tar.gz</a></td><td align="right">26-Nov-2012 17:04  </td><td align="right">3.4K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.3.0.tar.gz">bitstream-1.3.0.tar.gz</a></td><td align="right">10-Jan-2013 10:01  </td><td align="right">3.5K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.4.0.tar.gz">bitstream-1.4.0.tar.gz</a></td><td align="right">21-Jan-2013 16:51  </td><td align="right">4.0K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.4.1.tar.gz">bitstream-1.4.1.tar.gz</a></td><td align="right">22-Mar-2013 15:29  </td><td align="right">3.9K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-1.5.0.tar.gz">bitstream-1.5.0.tar.gz</a></td><td align="right">17-Apr-2013 16:06  </td><td align="right">5.4K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="bitstream-2.0.0.tar.gz">bitstream-2.0.0.tar.gz</a></td><td align="right">17-May-2013 14:07  </td><td align="right">5.5K</td><td>&nbsp;</td></tr>
[...]
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="ipbase-1.2.1.tar.gz">ipbase-1.2.1.tar.gz</a></td><td align="right">27-Nov-2013 17:10  </td><td align="right"> 40K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="ipbase-1.2.2.tar.gz">ipbase-1.2.2.tar.gz</a></td><td align="right">27-Dec-2013 16:39  </td><td align="right"> 40K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="ipbase-1.3.0.tar.gz">ipbase-1.3.0.tar.gz</a></td><td align="right">17-Jan-2014 11:53  </td><td align="right"> 46K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="ipbase-1.3.1.tar.gz">ipbase-1.3.1.tar.gz</a></td><td align="right">04-Feb-2014 14:42  </td><td align="right"> 43K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="ipbase-1.4.0.tar.gz">ipbase-1.4.0.tar.gz</a></td><td align="right">05-Feb-2014 16:40  </td><td align="right"> 52K</td><td>&nbsp;</td></tr>
[...]
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="toolbox-2.0.tar.gz">toolbox-2.0.tar.gz</a></td><td align="right">24-Oct-2013 16:43  </td><td align="right">145K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="toolbox-2.1.tar.gz">toolbox-2.1.tar.gz</a></td><td align="right">15-Nov-2013 11:44  </td><td align="right">158K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="toolbox-3.0.tar.gz">toolbox-3.0.tar.gz</a></td><td align="right">21-Nov-2013 11:35  </td><td align="right">158K</td><td>&nbsp;</td></tr>
<tr><th colspan="5"><hr></th></tr>
</table>
</body></html>

[Ivoz]

If you have a simple pypi-ish internal mirror, you can use pip install --index-url=http://my.lan/pyrepo/ <package> or pip install --no-index --find-links=http://my.lan/pyrepo/ <package>

Then no need for dependency links.

[exhuma]

@Ivoz Agreed. This is also as far as I can remember the process used in our deployment scripts. I only happened to fall into this security issue, as I ran a manual installation on another host for testing.

Our development process (as currently documented) still uses a setup.py with dependency_links simply because it's easier (as you only have to maintain one list of dependencies) and it works for us.

Once pip 1.4 hits our machines, we'll have to update our process, and I just want to get my facts straight before this happens.

It's a lazyness thing... typing ./env/bin/pip install -e . is way easier that ./env/bin/pip install -e --find-links=http://my.lan/pyrepo/ . :) Maybe I'll just opt in for something like fab develop which does the right thing behind the scenes.

[dstufft]

I think there's a problem with dependency links and external detection. I'm traveling at the moment but I plan on taking a look at this this weekend and seeing what I can figure out. External url stuff should only take effect when you're installing from a repo that opts into it. So unless you've added the metatag, which it doesn't look like you have, this is a legit bug.

I don't have the greatest editing environment right now, can you split this out into a new bug and CC me in it?

[exhuma]

@dstufft done. I hope the message is not overly vague... It's also not something of high importance to me. It will only impact our development environment when pip 1.5 hits our dev boxes which is not yet the case. Also, deployment scripts are fine too. So there's no immediate hurry.

[Ivoz]

@exhuma note, you can also specify in a ~/.pypirc file an alternate index to use, or alternatively in an environment variable (e.g PIP_FIND_LINKS) so on a command line or in many scripts you do not need to type these over and over.

see http://www.pip-installer.org/en/latest/configuration.html#configuration

You can also specify an alternative index in a requirements.txt.

[rbpasker]

TL;DR: if you get errors for, say, 'egenix-mx-base', change your requirements.txt as follows:

old:

    egenix-mx-base==3.2.6

new:

    --allow-unverified egenix-mx-base
    egenix-mx-base==3.2.6

note that version id is not specified in the --allow-unverified line

[lsemel]

Is there any way to avoid repeating the package name for each and every
line in the requirements.txt that fails?

---

Lee Semel
leesemel.com - linkedin http://www.linkedin.com/in/leesemel -
twitterhttp://twitter.com/#%21/semel

On Wed, Mar 5, 2014 at 4:03 PM, bob pasker notifications@github.com wrote:

TL;DR: if you get errors for, say, 'egenix-mx-base', change your
requirements.txt as follows:

old:

egenix-mx-base==3.2.6

new:

--allow-unverified egenix-mx-base
egenix-mx-base==3.2.6

note that version id is not specified in the --allow-unverified line

Reply to this email directly or view it on GitHubhttps://github.com//issues/1423#issuecomment-36793071
.

[mjallday]

can someone please update this with how to solve this issue when specifying dependencies in setup.py?

moving to requirements.txt does not seem like a good solution since I don't then understand how to specify dependencies for anything we have that depends on the package that moves to requirements.txt.

googling this issue continues to bring up this github thread which only discusses the ramifications, not the solution (for me anyway).

[powellc]

@mjallday I second your concern. I'm still not sure how to get an unverified package in a requirements flie. I've added --allow-all-external to my requirements install command (pip install --allow-all-external -r requirements.txt) but it still hangs on unverified packages.

I'm non-plussed about this change to pip and while I appreciate the necessity of protecting us from malicious packages, there has to be a way to implement this that is not nearly so awkward.

[powellc]

I would also add that in the interface, using "==" for package versions and "=" for the packages that are allowed unverified is confusing and has led to more than a few bugs on my end.

All in all I would love for this whole external and unverified packages scheme to be completely rethought. If I can find the time I'll try to submit a pull request.

[erikjms]

A voice from a casual python user (I never inhale):

And so probably I have not googled sufficiently to understand all of what is going on with pip 1.5.6, but I am not a sysadmin (excepting this one single sys here), and I don't install lots of different environments. Most often I install packages on an as-needed basis so that I can run various free/open-source sound and graphics applications that happen to be written in python. My needs are convenience and clarity so that I can go quickly from installing an application and/or its dependencies to using it. Virtually every python package I have needed ever has been hosted on pypi. This has not changed.

This is my experience in a nutshell:

Since updating pip, what used to look like this
$ pip install wxpython

now must look like this to work
$ pip install --allow-external wxpython --allow-unverified wxpython wxpython==2.9.1.1

edit: omygod. that did not actually work. I really need to replace that dark red in my terminal profile with something a little brighter. I will figure this out, but do you see what I am getting at?

When this did not work:
$ pip install --allow-external wxpython --allow-unverified wxpython

I found myself reading this thread because I was completely exasperated. The pip documentation online does not hint at the necessity of naming a single package three times for relatively simple installs like this one. Last week trying to use pip I think I gave up and decided to go do something else--I left no notes on how to deal with the sinking feeling that I recognized immediately on reading "Use --allow-external". Tonight though I had the energy to google.

Obviously I have copied and pasted the working install command in my sysadmin cheat notepad for future reference, but, I don't know. Maybe my opinion carries no weight since I am but a dilettante when it comes to programming, but I can't be the only non-coder coder out here scratching my head.

[Ivoz]

@erikjms I think the pip maintainers can agree although there was a clear aim in introducing these flags, the User Experience of the implementation wasn't thought through enough at the time and ended up being unnecessarily confusing. @dstufft @qwcode @pfmoore we should definitely discuss before 1.6 to make sure this has been improved.

[bladeoflight16]

@Ivoz Thank you so much. I think this is all we were really asking for.

[ogrotten]

So what's the currently recommended command to install an external pkg? $ pip install --allow-all-external --allow-unverified mysql-connector-python didn't work, nor does any of it's simpler derivations.

edit: nevermind. have to have the name twice at the end.

$ pip install --allow-all-external --allow-unverified mysql-connector-python mysql-connector-python

To be perfectly honest, and not to hash over all the same arguments above, I wouldn't have a problem with this syntax if it were sensibly documented.

[xavfernandez]

allow-unverified option needs a package
You must do:

$ pip install --allow-all-external --allow-unverified mysql-connector-python mysql-connector-python

[macropin]

I'm not sure why this was closed. This title of this ticket is still valid.

[powellc]

I think the instructions were supposedly clarified, but I would agree @macropin

[macropin]

@powellc I think the principle of least astonishment needs to be applied here...

A simple pip install --unsafe <pkg_name> should suffice.

[isbm]

Holy cow, I am still struggling install that damn apt that pip continues skipping. Maybe I shall buy a tambourine instead... Hello to those, who designed this! 😠

[bladeoflight16]

@isbm Pretty sure that has nothing to do with pip. This StackOverflow question comes up fairly early in the Google results. If you have more difficulties, this really isn't the appropriate place to address them. Try one of the Stack Exchange sites.

@macropin It was closed because a change was made so that --allow-unverified now implies --allow-external. That said, I do find your suggestion more intuitive, but it's not as bad as it used to be.

[ut-adamc]

@macropin: I agree.

[zx1986]

pip install --allow-all-external --allow-unverified elementtree elementtree
works for me too ....

[anirudhp2014]

--allow-external never worker for me .

[xavfernandez]

They'll both be deprecated in a few weeks, as per http://legacy.python.org/dev/peps/pep-0470/ .