When single package specified several times with different extras, only first set is considered

[MarSoft]

• Pip version: 9.0.1
• Python version: 2.7.13
• Operating system: Arch Linux

Description:

When having several entries for one package with different extras requested (in particular when first entry uses git URL and second just cites name), extras for second entry are not recognized.
Example requirements.txt:

some-package[extra1,extra2]
some-package[extra3]

In this example, some-package will be installed, but only dependencies for extra1 and extra2 will be installed, but no dependencies for extra3.
Notice there are no version constraints.

Here is a bit of what I could find from pip internals:
When pip install command encounters a package reference, it adds that reference with all parameters to RequirementSet using RequirementSet.add_requirement method passing it a single install_req object.
Then, add_requirement (around line 260) checks if there is already required package with the same name. And if it is (which is the case for second line in my example), it just returns existing_req, ignoring new value with additional extras (around line 301).

I have noticed that there actually is a code to update extras. But this code is only used if not install_req.constraint and existing_req.constraint.
I am not sure if I understand it correctly, but looks like constraint flag is only set for packages coming from constraints file. So in my case (when both package references come from requirements file) this clause is just not executed.

What I've run:

Here is the simplest way I found to reproduce the problem. This particular command looks meaningless (as one could just write requests[security,socks]), but it does make sense if e.g. requirements are separated in different files, and some of them are taken from URLs, etc.

$ pip install 'requests[security]' 'requests[socks]'
Collecting requests[security]
  Using cached requests-2.18.3-py2.py3-none-any.whl
Requirement already satisfied: idna<2.6,>=2.5 in ./tmpenv/lib/python2.7/site-packages (from requests[security])
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./tmpenv/lib/python2.7/site-packages (from requests[security])
Collecting chardet<3.1.0,>=3.0.2 (from requests[security])
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests[security])
  Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting cryptography>=1.3.4; extra == "security" (from requests[security])
  Using cached cryptography-2.0.3-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyOpenSSL>=0.14; extra == "security" (from requests[security])
  Using cached pyOpenSSL-17.2.0-py2.py3-none-any.whl
Collecting cffi>=1.7 (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached cffi-1.10.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting enum34 (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached asn1crypto-0.22.0-py2.py3-none-any.whl
Collecting ipaddress (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached ipaddress-1.0.18-py2-none-any.whl
Requirement already satisfied: six>=1.4.1 in ./tmpenv/lib/python2.7/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Collecting pycparser (from cffi>=1.7->cryptography>=1.3.4; extra == "security"->requests[security])
Installing collected packages: chardet, certifi, pycparser, cffi, enum34, asn1crypto, ipaddress, cryptography, pyOpenSSL, requests
Successfully installed asn1crypto-0.22.0 certifi-2017.7.27.1 cffi-1.10.0 chardet-3.0.4 cryptography-2.0.3 enum34-1.1.6 ipaddress-1.0.18 pyOpenSSL-17.2.0 pycparser-2.18 requests-2.18.3

Notice that PySocks was is not installed!

[pradyunsg]

Thanks for filing this issue!

This issue stems from the fact that pip currently does not have a dependency resolver, which is being tracked in #988.

I'm going to close this issue as a duplicate of #988. :)

[pradyunsg]

I just realised -- this behaviour change is being tracked in #4391. This issue is duplicate of that (not #988).