setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management #308

pdecat · 2018-06-05T22:43:22Z

When installing pylxd with pip, pip invokes its setup.py with the egg_info command to retrieve egg metadata about pylxd (cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#build-system-interface).

This has the side effect of installing pylxd dependencies which currently breaks because of an incompatibility between current requests 2.18.4 and urllib3 1.23.

Example with current tarball from https://pypi.org/project/pylxd/#files:

# wget -q https://files.pythonhosted.org/packages/a0/5e/af099af60d089b28df6b550d34c1e807ce4e3906d257744f55573c4d3cbb/pylxd-2.2.6.tar.gz                            
# tar xf pylxd-2.2.6.tar.gz
# cd pylxd-2.2.6/
# virtualenv test
New python executable in /home/pdecat/workspaces/tmp/pylxd-2.2.6/test/bin/python
Installing setuptools, pip, wheel...done.
# ./test/bin/pip freeze
# ./test/bin/python setup.py egg_info
Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests_unixsocket-0.1.5-py2.7.egg
Searching for requests!=2.8.0,>=2.5.2
Reading https://pypi.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
Best match: requests 2.18.4
Processing requests-2.18.4-py2.py3-none-any.whl
Installing requests-2.18.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg
Searching for pbr>=1.8
Reading https://pypi.org/simple/pbr/
Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
Best match: pbr 4.0.4
Processing pbr-4.0.4-py2.py3-none-any.whl
Installing pbr-4.0.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/pbr-4.0.4-py2.7.egg
Searching for urllib3>=1.8
Reading https://pypi.org/simple/urllib3/
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
Best match: urllib3 1.23
Processing urllib3-1.23-py2.py3-none-any.whl
Installing urllib3-1.23-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    pbr=True)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

As mentioned in several places, runtime dependencies should not be put in setup_requires as it breaks pip's dependency management:

Warning Finally, beware of the setup_requires keyword arg in setup.py. The (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip's protections. If you need to use such a package, see Controlling setup_requires.

cf. https://pip.readthedocs.io/en/stable/user_guide/#installation-bundles

Controlling setup_requires

Setuptools offers the setup_requires setup() keyword for specifying dependencies that need to be present in order for the setup.py script to run. Internally, Setuptools uses easy_install to fulfill these dependencies.

pip has no way to control how these dependencies are located. None of the Package Index Options have an effect.

cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#controlling-setup-requires

psf/requests#4669
psf/requests#4671

The text was updated successfully, but these errors were encountered:

pdecat · 2018-06-06T06:11:58Z

To provide a bit more context, this currently prevents installing pylxd even when all its dependencies are pinned in a requirements.txt file:

# docker run -ti python:2.7 bash
root@df4a2f10d39a:/# cat > requirements-lxd.txt <<EOF
> asn1crypto==0.24.0        # via cryptography
> certifi==2018.4.16        # via requests
> cffi==1.11.5              # via cryptography
> chardet==3.0.4            # via requests
> cryptography==2.2.2       # via pylxd, pyopenssl
> enum34==1.1.6             # via cryptography
> idna==2.6                 # via cryptography, idna, requests
> ipaddress==1.0.22         # via cryptography
> pbr==4.0.4                # via pylxd
> pycparser==2.18           # via cffi
> pylxd==2.2.6
> pyopenssl==18.0.0         # via pylxd
> python-dateutil==2.7.3    # via pylxd
> requests-toolbelt==0.8.0  # via pylxd
> requests-unixsocket==0.1.5  # via pylxd
> requests==2.18.4          # via pylxd, requests-toolbelt, requests-unixsocket
> six==1.11.0               # via cryptography, pylxd, pyopenssl, python-dateutil
> urllib3==1.22             # via requests, requests-unixsocket, urllib3
> ws4py==0.5.1              # via pylxd
> EOF

Note the urllib3==1.22.

root@df4a2f10d39a:/# virtualenv test
New python executable in /test/bin/python
Installing setuptools, pip, wheel...done.

root@df4a2f10d39a:/# ./test/bin/pip install --progress-bar=off -r requirements-lxd.txt                                                                                                                         
Collecting asn1crypto==0.24.0 (from -r requirements-lxd.txt (line 1))                                                                                                                                          
  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)                                        
                                                                                                                                                                                                               
Collecting certifi==2018.4.16 (from -r requirements-lxd.txt (line 2))                                                                                                                                          
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)                                        
                                                                                                                                                                                                               
Collecting cffi==1.11.5 (from -r requirements-lxd.txt (line 3))                                                                                                                                                
  Downloading https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)                                 
                                                                                                                                                                                                               
Collecting chardet==3.0.4 (from -r requirements-lxd.txt (line 4))                                                                                                                                              
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)                                            
                                                                                                                                                                                                               
Collecting cryptography==2.2.2 (from -r requirements-lxd.txt (line 5))                                                                                                                                         
  Downloading https://files.pythonhosted.org/packages/dd/c2/3a5bfefb25690725824ade71e6b65449f0a9f4b29702cce10560f786ebf6/cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)                          
                                                                                                                                                                                                               
Collecting enum34==1.1.6 (from -r requirements-lxd.txt (line 6))                                                                                                                                               
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl                                                         
Collecting idna==2.6 (from -r requirements-lxd.txt (line 7))                                                                                                                                                   
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)                                                  
                                                                                                                                                                                                               
Collecting ipaddress==1.0.22 (from -r requirements-lxd.txt (line 8))                                                                                                                                           
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl                                                 
Collecting pbr==4.0.4 (from -r requirements-lxd.txt (line 9))                                                                                                                                                  
  Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl (98kB)                                                 

Collecting pycparser==2.18 (from -r requirements-lxd.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)

Collecting pylxd==2.2.6 (from -r requirements-lxd.txt (line 11))
  Downloading https://files.pythonhosted.org/packages/a0/5e/af099af60d089b28df6b550d34c1e807ce4e3906d257744f55573c4d3cbb/pylxd-2.2.6.tar.gz (71kB)

    Complete output from command python setup.py egg_info:

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/requests_unixsocket-0.1.5-py2.7.egg
    Searching for requests!=2.8.0,>=2.5.2
    Reading https://pypi.org/simple/requests/
    Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
    Best match: requests 2.18.4
    Processing requests-2.18.4-py2.py3-none-any.whl
    Installing requests-2.18.4-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs
    writing requirements to /tmp/pip-install-Knq2xX/pylxd/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/requests-2.18.4-py2.7.egg
    Searching for pbr>=1.8
    Reading https://pypi.org/simple/pbr/
    Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
    Best match: pbr 4.0.4
    Processing pbr-4.0.4-py2.py3-none-any.whl
    Installing pbr-4.0.4-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/pbr-4.0.4-py2.7.egg
    Searching for urllib3>=1.8
    Reading https://pypi.org/simple/urllib3/
    Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
    Best match: urllib3 1.23
    Processing urllib3-1.23-py2.py3-none-any.whl
    Installing urllib3-1.23-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs
    writing requirements to /tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-Knq2xX/pylxd/setup.py", line 34, in <module>
        pbr=True)
      File "/test/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/test/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/test/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/test/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Knq2xX/pylxd/

Signed-off-by: Patrick Decat <pdecat@gmail.com>

ajkavanagh · 2018-06-06T12:17:35Z

Thanks very much for working through this one.

pdecat · 2018-06-06T14:10:36Z

You're welcome :)
Any idea if/when a 2.2.7 release can be expected to resolve this issue?

pdecat · 2018-06-06T14:33:39Z

An alternative to an expedited 2.2.7 release may be to build and publish eggs for 2.2.6 including urlllib3 1.22:

# virtualenv .venv27
# ./.venv27/bin/pip install urllib3==1.22
# ./.venv27/bin/python2 setup.py bdist_egg

# python3 -m venv .venv
# ./.venv/bin/pip install urllib==1.22
# ./.venv/bin/python3 setup.py bdist_egg

~~Note: I'm talking eggs because pylxd's current setup.py does not support wheels.~~

pdecat · 2018-06-06T14:48:53Z

Hmm, supporting wheels is just a matter of installing the wheel package in the venv:

# python3 -m venv .venv
# ./.venv/bin/pip install urllib==1.22
# ./.venv/bin/pip install wheel
# ./.venv/bin/python3 setup.py bdist_wheel

ajkavanagh · 2018-06-07T18:02:56Z

@pdecat I'll push out 2.2.7 tomorrow - I had to do some testing against various older distros and all the lxds!

pdecat · 2018-06-07T18:06:12Z

Great, thanks!

…anonical#309) Signed-off-by: Patrick Decat <pdecat@gmail.com>

Signed-off-by: Patrick Decat <pdecat@gmail.com>

ajkavanagh · 2018-06-08T14:47:03Z

@pdecat Just to let you know that 2.2.7 has been released with the relevant patches in it. Thanks again.

pdecat · 2018-06-09T12:43:31Z

Thanks! Issue resolved for us.

…liy (cf. canonical/pylxd#308)

…anonical#309) Signed-off-by: Patrick Decat <pdecat@gmail.com>

…anonical#309) (canonical#310) Signed-off-by: Patrick Decat <pdecat@gmail.com>

Install packages from PIP_PACKAGES before wheel build to be able to limit dependencies specified in "setup_requires" of packages listed in upper-constraints.txt, that pip has no way to control. This is intended to work around canonical/pylxd#308 or similar issues. Change-Id: I36bd4347727b4c7f4beea2cdba6674445fbb4ca4

Patch Set 1: See also canonical/pylxd#308 Patch-set: 1

pdecat changed the title ~~setup.py should not put dependencies in setup_requires as it breaks pip dependency management~~ setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management Jun 5, 2018

pdecat mentioned this issue Jun 6, 2018

Avoid generating metadata in pip download --no-deps ... pypa/pip#1884

Open

pdecat added a commit to pdecat/pylxd that referenced this issue Jun 6, 2018

Put runtime dependencies in install_requires (fixes canonical#308)

9d9d911

pdecat mentioned this issue Jun 6, 2018

Put runtime dependencies in install_requires (fixes #308) #309

Merged

pdecat added a commit to pdecat/pylxd that referenced this issue Jun 6, 2018

Put runtime dependencies in install_requires (fixes canonical#308)

0b2e6b3

Signed-off-by: Patrick Decat <pdecat@gmail.com>

pdecat added a commit to pdecat/pylxd that referenced this issue Jun 6, 2018

Put runtime dependencies in install_requires (fixes canonical#308)

b7d4ad3

Signed-off-by: Patrick Decat <pdecat@gmail.com>

ajkavanagh added the bug label Jun 6, 2018

ajkavanagh added this to the 2.2.7 milestone Jun 6, 2018

ajkavanagh closed this as completed in #309 Jun 6, 2018

ajkavanagh pushed a commit that referenced this issue Jun 6, 2018

Put runtime dependencies in install_requires (fixes #308) (#309)

fd4d999

Signed-off-by: Patrick Decat <pdecat@gmail.com>

pdecat added a commit to pdecat/pylxd that referenced this issue Jun 8, 2018

Keep pbr dependency in setup_requires (fixes canonical#308, follows c…

2f8a3be

…anonical#309) Signed-off-by: Patrick Decat <pdecat@gmail.com>

pdecat mentioned this issue Jun 8, 2018

Keep pbr dependency in setup_requires (fixes #308, follows #309) #310

Merged

ajkavanagh pushed a commit that referenced this issue Jun 8, 2018

Keep pbr dependency in setup_requires (fixes #308, follows #309) (#310)

4b8ab18

Signed-off-by: Patrick Decat <pdecat@gmail.com>

Shr3ps pushed a commit to claranet/cloud-deploy that referenced this issue Jul 10, 2018

GHOST-620: upgrade to pylxd 2.2.7 to resolve urllib3 1.23 incompatibi…

c1b933a

…liy (cf. canonical/pylxd#308)

lasizoillo pushed a commit to lasizoillo/pylxd that referenced this issue Jul 13, 2018

Put runtime dependencies in install_requires (fixes canonical#308) (c…

0fbf460

…anonical#309) Signed-off-by: Patrick Decat <pdecat@gmail.com>

lasizoillo pushed a commit to lasizoillo/pylxd that referenced this issue Jul 13, 2018

Keep pbr dependency in setup_requires (fixes canonical#308, follows c…

fffeae6

…anonical#309) (canonical#310) Signed-off-by: Patrick Decat <pdecat@gmail.com>

kissiel mentioned this issue Dec 13, 2023

Pin urllib to <2.0 for Metabox (BugFix) canonical/checkbox#892

Merged

tanaypf9 pushed a commit to tanaypf9/pf9-requirements that referenced this issue May 20, 2024

Update patch set 1

a6ec37f

Patch Set 1: See also canonical/pylxd#308 Patch-set: 1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management #308

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management #308

pdecat commented Jun 5, 2018 •

edited

Loading

Controlling `setup_requires`

pdecat commented Jun 6, 2018

ajkavanagh commented Jun 6, 2018

pdecat commented Jun 6, 2018

pdecat commented Jun 6, 2018 •

edited

Loading

pdecat commented Jun 6, 2018

ajkavanagh commented Jun 7, 2018

pdecat commented Jun 7, 2018

ajkavanagh commented Jun 8, 2018

pdecat commented Jun 9, 2018

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management #308

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management #308

Comments

pdecat commented Jun 5, 2018 • edited Loading

Controlling setup_requires

pdecat commented Jun 6, 2018

ajkavanagh commented Jun 6, 2018

pdecat commented Jun 6, 2018

pdecat commented Jun 6, 2018 • edited Loading

pdecat commented Jun 6, 2018

ajkavanagh commented Jun 7, 2018

pdecat commented Jun 7, 2018

ajkavanagh commented Jun 8, 2018

pdecat commented Jun 9, 2018

pdecat commented Jun 5, 2018 •

edited

Loading

Controlling `setup_requires`

pdecat commented Jun 6, 2018 •

edited

Loading