Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools 50 breaks pip installation on Debian/Ubuntu #17

Closed
JelleZijlstra opened this issue Aug 30, 2020 · 16 comments
Closed

setuptools 50 breaks pip installation on Debian/Ubuntu #17

JelleZijlstra opened this issue Aug 30, 2020 · 16 comments
Labels
duplicate This issue or pull request already exists

Comments

@JelleZijlstra
Copy link

On newly created Ubuntu 16.04 AWS instances, we started seeing that python3.6 -m pip no longer works after get-pip.py (version 19.3.1). get-pip puts pip in /usr/lib/python3.6/site-packages/pip-20.2.2.dist-info, but sys.path for our 3.6 executable is ['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages'].

On a newly created box I was able to confirm that using SETUPTOOLS_USE_DISTUTILS=stdlib fixes the problem:

    1  python3.6 /get_pip.py 
    2  python3.6 -m pip  # works
    3  python3.6 -m pip install -U setuptools  # updates from 41.0.1 to 50.0.0
    4  python3.6 -m pip  # still works
    5  python3.6 /get_pip.py 
    6  python3.6 -m pip  # can't find pip any more
    7  SETUPTOOLS_USE_DISTUTILS=stdlib python3.6 /get_pip.py 
    8  python3.6 -m pip  # works again

Sorry this isn't a completely self-contained repro case for the bug; hopefully this is enough information to figure out what's going on.

@ianw
Copy link

ianw commented Aug 31, 2020

What you're seeing is pypa/setuptools#2232 which has been re-introduced, mentioned pypa/setuptools@04e3df2

In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages.
It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked.

Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules.

Another consequence is the binary location, which moved from /usr/local/bin/ to /usr/bin. That's kind of a moot point though, because your binaries won't be able to find their libraries anyway.

@vmarkovtsev
Copy link

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

rswarbrick referenced this issue in rswarbrick/ibex Aug 31, 2020
There's a long and painful discussion on the setuptools issues
list (I count 9 issues raised the day after the release). This is all
to do with how Debian/Ubuntu installs stuff with pip.

There's an "official" workaround that involves setting things in your
environment, but that looks a bit fiddly with the Azure pipelines
stuff and this has been such a disaster that I'm pretty certain the
setuptools maintainers will release something more sensible soon.

See e.g.
https://github.com/pypa/setuptools/issues/2350#issuecomment-683512526
for a careful description of what's going on.
rswarbrick referenced this issue in rswarbrick/ibex Aug 31, 2020
There's a long and painful discussion on the setuptools issues
list (I count 9 issues raised the day after the release). This is all
to do with how Debian/Ubuntu installs stuff with pip.

There's an "official" workaround that involves setting things in your
environment, but that looks a bit fiddly with the Azure pipelines
stuff and this has been such a disaster that I'm pretty certain the
setuptools maintainers will release something more sensible soon.

See e.g.
https://github.com/pypa/setuptools/issues/2350#issuecomment-683512526
for a careful description of what's going on.
rswarbrick referenced this issue in lowRISC/ibex Aug 31, 2020
There's a long and painful discussion on the setuptools issues
list (I count 9 issues raised the day after the release). This is all
to do with how Debian/Ubuntu installs stuff with pip.

There's an "official" workaround that involves setting things in your
environment, but that looks a bit fiddly with the Azure pipelines
stuff and this has been such a disaster that I'm pretty certain the
setuptools maintainers will release something more sensible soon.

See e.g.
https://github.com/pypa/setuptools/issues/2350#issuecomment-683512526
for a careful description of what's going on.
rswarbrick referenced this issue in rswarbrick/opentitan Aug 31, 2020
There's a long and painful discussion on the setuptools issues
list (I count 9 issues raised the day after the release). This is all
to do with how Debian/Ubuntu installs stuff with pip.

There's an "official" workaround that involves setting things in your
environment, but that looks a bit fiddly with the Azure pipelines
stuff and this has been such a disaster that I'm pretty certain the
setuptools maintainers will release something more sensible soon.

See e.g.
https://github.com/pypa/setuptools/issues/2350#issuecomment-683512526
for a careful description of what's going on.

Signed-off-by: Rupert Swarbrick <rswarbrick@lowrisc.org>
rswarbrick referenced this issue in lowRISC/opentitan Aug 31, 2020
There's a long and painful discussion on the setuptools issues
list (I count 9 issues raised the day after the release). This is all
to do with how Debian/Ubuntu installs stuff with pip.

There's an "official" workaround that involves setting things in your
environment, but that looks a bit fiddly with the Azure pipelines
stuff and this has been such a disaster that I'm pretty certain the
setuptools maintainers will release something more sensible soon.

See e.g.
https://github.com/pypa/setuptools/issues/2350#issuecomment-683512526
for a careful description of what's going on.

Signed-off-by: Rupert Swarbrick <rswarbrick@lowrisc.org>
@JelleZijlstra JelleZijlstra changed the title setuptools 50 breaks pip installation setuptools 50 breaks pip installation on Debian/Ubuntu Aug 31, 2020
@ianw
Copy link

ianw commented Aug 31, 2020

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

I think it's all related; if you hit the inbuilt distutils you're not getting the overrides that Debian has set. So pip install -e is the most straight forward way to do this.

It is confusing however; if you just do pip install . then you'll build and install a wheel, which won't exhibit this problem. But if you try pip install --no-binary :all: . in your source directory it will be using distutils and and install it to site-packages, not dist-packages. You could also run setup.py directly which will exhibit the problem too.

vladdyk referenced this issue in sleepio/docker-serverless Sep 1, 2020
…ching (#16)

The issue has been affecting many people today

https://github.com/pypa/setuptools/issues/2350

This seems to be the fix to unblock us from deploying without having to modify a bunch of source files across repos. To override this value, a project can set this env var in its own buildspec.yml. 

Tested by deploying from local, also tested with ops-service-cluster in QA:

https://r0p115rf9k.execute-api.us-east-1.amazonaws.com/newqa/deploy/build?arn=arn:aws:codebuild:us-east-1:200595175417:build/ops-service-cluster:641d3471-92d1-4000-a41b-8503ca5d4844
@pradyunsg
Copy link
Member

Quoting the changelog entry, for the relevant change:

pypa/setuptools#2232: Once again, Setuptools overrides the stdlib distutils on import. For environments or invocations where this behavior is undesirable, users are provided with a temporary escape hatch. If the environment variable SETUPTOOLS_USE_DISTUTILS is set to stdlib, Setuptools will fall back to the legacy behavior. Use of this escape hatch is discouraged, but it is provided to ease the transition while proper fixes for edge cases can be addressed.

Please see pypa/setuptools#2232 for the relevant discussion on this.

@jaraco jaraco pinned this issue Sep 2, 2020
@encukou
Copy link
Contributor

encukou commented Sep 2, 2020

Note that this affects other distros than just Ubuntu/Debian. It seems that distutils is the only place where you can define install schemes (and even if that's not the case, people assume that). So, it's quite widely patched.
See also: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059

@davidnewman02
Copy link

As a minimal reproduction based on the comment here: pypa/setuptools#2356 (comment)

docker run -it ubuntu:16.04

apt update; apt install -y python3 python3-venv
python3 -m venv venv
source venv/bin/activate
pip install setuptools==50.0.0
python -c "import distutils"

@rubyFeedback
Copy link

This is a common issue in debian-based distributions; debian also re-packages ruby, causing newcomers to complain on IRC about default assumptions (made via the path) not working, thus outsourcing their problems to e. g. the IRC crowd who then have to try to fix it (and typically recommend using package version managers to just avoid the debian-hassle, rather than e. g. apt-get something). While I think debian's way to handle this is really really bad, I also think that ultimately tools should be flexible enough to support this weird splitting-of-packages that debian uses.

@jaraco
Copy link
Member

jaraco commented Sep 2, 2020

I recognize that this issue is serious and needs addressing. Here are the possible options in my order of descending preference:

  • Owners of the debian distutils patch identify the behaviors needed to support the needs of Debian (alongside all other platforms) and contribute those to pypa/distutils.
  • Apply Debian-specific behaviors to pypa/distutils (previously-rejected approach).
  • Disable the overriding of distutils on Debian until Debian maintainers can devise a better approach.
  • Back out the overriding of distutils for all Setuptools users until Debian can devise a better approach.

Are there other options? Are there Debian maintainers here who can help advise/discuss?

@hugovk
Copy link
Contributor

hugovk commented Sep 2, 2020

Looks like @doko42 is a maintainer for distutils (and setuptools) on Debian: can you help with the above?

@encukou
Copy link
Contributor

encukou commented Sep 2, 2020

I am not a Debian maintainer, but looking at their patch, their needs are the same as in Fedora (and PyPy, too): that is, defining and selecting install schemes.
On Discourse, there is an idea of moving this data/logic to sysconfig. Would this work as a better approach (for option 4)?

Unfortunately, on today's systems that customize distutils in this way, the stdlib's distutils is the only source of this information. I don't see a way to change this until Python 3.10.

@pradyunsg
Copy link
Member

I believe @kitterma is also involved in Debian's Python. :)

@canassa
Copy link

canassa commented Sep 2, 2020

FYI, this doesn't only affect Debian. The AWS lambdas running Python were also broken.

https://stackoverflow.com/questions/63688774/some-aws-lambda-functions-stopped-working-with-no-module-named-setuptools-dist

@unchris
Copy link

unchris commented Sep 2, 2020

* Back out the overriding of distutils for all Setuptools users until Debian can devise a better approach.

If this means "undo the changes in 50.0 and release 51.0 which behaves like 49.latest worked" then I think that might be the preference of the vast majority of the community that uses setuptools.

In this thread you can see just a snapshot of the work being generated right now, just in the open-source community on GitHub, and it's jarring. People are pinning setuptools to versions before 50, despite the fact it's not recommended to pin setuptools in requirements.txt-style installs, and PyPA's own documentation recommends the recipe python -m pip install --upgrade pip setuptools wheel when installing packages via pip

Add this to all of the community who have adopted Docker containers and there will be weeks of people discovering the various open issues on setuptools right now and hoping to find a fix but ultimately not being super knowledgeable about distutils or setuptools or how they interact with distro X because, honestly, nobody outside of PyPA and a small circle in the Python community Venn diagram have any idea what all this is about.

I would add an argument about how now would be the perfect time for CVEs in pre-setuptools 50.0 to drop but I'm only security conscious enough to know it would be bad so I'll leave the speculation for someone more intelligent than I am there.

It may be your least preferred option but from the point of view of myself, and I reckon much of the Python-using community, it's the only option you should be considering given the widespread effects of this issue.

/2c, and I meant every word from a place of good faith. "don't break userland" should apply to more things than the Linux kernel

@jaraco
Copy link
Member

jaraco commented Sep 2, 2020

I'm convinced the behavior needs to be suppressed for now. I was hoping we could work through the impacts and push through this time (based on the fallout from the Setuptools 48 release being much less severe).

@pradyunsg
Copy link
Member

Setuptools 50.1.0 changes contains pypa/setuptools#2376, which changes the default to be more conservative.

nickgeorge referenced this issue in google/fhir Sep 8, 2020
This is to avoid the issues introduced by setuptools==50.0.0 on Ubuntu/Debian platforms (see https://github.com/pypa/setuptools/issues/2350).

PiperOrigin-RevId: 329511139
@jaraco jaraco transferred this issue from pypa/setuptools Sep 13, 2020
@jaraco jaraco added the duplicate This issue or pull request already exists label Sep 13, 2020
@jaraco
Copy link
Member

jaraco commented Sep 13, 2020

Closing as a duplicate of #2.

@jaraco jaraco closed this as completed Sep 13, 2020
manthey added a commit to girder/girder that referenced this issue Feb 1, 2023
The issue appears when doing 'pip install -e <package>', the package
does not show up when doing 'pip freeze'.  This is caused by Ubuntu
installing a patched version of distutils (see pypa/distutils/issues/17
and pypa/setuptools/issues/3301; see also pypa/setuptools/issues/3625
for when this will stop working).

This adds a workaround.  Better would be to create a virtualenv and
install into that and be insulated from odd OS upstream choices.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests