setuptools 50 breaks pip installation on Debian/Ubuntu

[JelleZijlstra]

On newly created Ubuntu 16.04 AWS instances, we started seeing that python3.6 -m pip no longer works after get-pip.py (version 19.3.1). get-pip puts pip in /usr/lib/python3.6/site-packages/pip-20.2.2.dist-info, but sys.path for our 3.6 executable is ['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages'].

On a newly created box I was able to confirm that using SETUPTOOLS_USE_DISTUTILS=stdlib fixes the problem:

    1  python3.6 /get_pip.py 
    2  python3.6 -m pip  # works
    3  python3.6 -m pip install -U setuptools  # updates from 41.0.1 to 50.0.0
    4  python3.6 -m pip  # still works
    5  python3.6 /get_pip.py 
    6  python3.6 -m pip  # can't find pip any more
    7  SETUPTOOLS_USE_DISTUTILS=stdlib python3.6 /get_pip.py 
    8  python3.6 -m pip  # works again

Sorry this isn't a completely self-contained repro case for the bug; hopefully this is enough information to figure out what's going on.

[ianw]

What you're seeing is pypa/setuptools#2232 which has been re-introduced, mentioned pypa/setuptools@04e3df2

In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages.
It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked.

Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules.

Another consequence is the binary location, which moved from /usr/local/bin/ to /usr/bin. That's kind of a moot point though, because your binaries won't be able to find their libraries anyway.

[vmarkovtsev]

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

[ianw]

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

I think it's all related; if you hit the inbuilt distutils you're not getting the overrides that Debian has set. So pip install -e is the most straight forward way to do this.

It is confusing however; if you just do pip install . then you'll build and install a wheel, which won't exhibit this problem. But if you try pip install --no-binary :all: . in your source directory it will be using distutils and and install it to site-packages, not dist-packages. You could also run setup.py directly which will exhibit the problem too.

[pradyunsg]

Quoting the changelog entry, for the relevant change:

pypa/setuptools#2232: Once again, Setuptools overrides the stdlib distutils on import. For environments or invocations where this behavior is undesirable, users are provided with a temporary escape hatch. If the environment variable SETUPTOOLS_USE_DISTUTILS is set to stdlib, Setuptools will fall back to the legacy behavior. Use of this escape hatch is discouraged, but it is provided to ease the transition while proper fixes for edge cases can be addressed.

Please see pypa/setuptools#2232 for the relevant discussion on this.

[encukou]

Note that this affects other distros than just Ubuntu/Debian. It seems that distutils is the only place where you can define install schemes (and even if that's not the case, people assume that). So, it's quite widely patched.
See also: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059

[davidnewman02]

As a minimal reproduction based on the comment here: pypa/setuptools#2356 (comment)

docker run -it ubuntu:16.04

apt update; apt install -y python3 python3-venv
python3 -m venv venv
source venv/bin/activate
pip install setuptools==50.0.0
python -c "import distutils"

[rubyFeedback]

This is a common issue in debian-based distributions; debian also re-packages ruby, causing newcomers to complain on IRC about default assumptions (made via the path) not working, thus outsourcing their problems to e. g. the IRC crowd who then have to try to fix it (and typically recommend using package version managers to just avoid the debian-hassle, rather than e. g. apt-get something). While I think debian's way to handle this is really really bad, I also think that ultimately tools should be flexible enough to support this weird splitting-of-packages that debian uses.

[jaraco]

I recognize that this issue is serious and needs addressing. Here are the possible options in my order of descending preference:

• Owners of the debian distutils patch identify the behaviors needed to support the needs of Debian (alongside all other platforms) and contribute those to pypa/distutils.
• Apply Debian-specific behaviors to pypa/distutils (previously-rejected approach).
• Disable the overriding of distutils on Debian until Debian maintainers can devise a better approach.
• Back out the overriding of distutils for all Setuptools users until Debian can devise a better approach.

Are there other options? Are there Debian maintainers here who can help advise/discuss?

[hugovk]

Looks like @doko42 is a maintainer for distutils (and setuptools) on Debian: can you help with the above?

[encukou]

I am not a Debian maintainer, but looking at their patch, their needs are the same as in Fedora (and PyPy, too): that is, defining and selecting install schemes.
On Discourse, there is an idea of moving this data/logic to sysconfig. Would this work as a better approach (for option 4)?

Unfortunately, on today's systems that customize distutils in this way, the stdlib's distutils is the only source of this information. I don't see a way to change this until Python 3.10.

[pradyunsg]

I believe @kitterma is also involved in Debian's Python. :)

[canassa]

FYI, this doesn't only affect Debian. The AWS lambdas running Python were also broken.

https://stackoverflow.com/questions/63688774/some-aws-lambda-functions-stopped-working-with-no-module-named-setuptools-dist

[unchris]

* Back out the overriding of distutils for all Setuptools users until Debian can devise a better approach.

If this means "undo the changes in 50.0 and release 51.0 which behaves like 49.latest worked" then I think that might be the preference of the vast majority of the community that uses setuptools.

In this thread you can see just a snapshot of the work being generated right now, just in the open-source community on GitHub, and it's jarring. People are pinning setuptools to versions before 50, despite the fact it's not recommended to pin setuptools in requirements.txt-style installs, and PyPA's own documentation recommends the recipe python -m pip install --upgrade pip setuptools wheel when installing packages via pip

Add this to all of the community who have adopted Docker containers and there will be weeks of people discovering the various open issues on setuptools right now and hoping to find a fix but ultimately not being super knowledgeable about distutils or setuptools or how they interact with distro X because, honestly, nobody outside of PyPA and a small circle in the Python community Venn diagram have any idea what all this is about.

I would add an argument about how now would be the perfect time for CVEs in pre-setuptools 50.0 to drop but I'm only security conscious enough to know it would be bad so I'll leave the speculation for someone more intelligent than I am there.

It may be your least preferred option but from the point of view of myself, and I reckon much of the Python-using community, it's the only option you should be considering given the widespread effects of this issue.

/2c, and I meant every word from a place of good faith. "don't break userland" should apply to more things than the Linux kernel

[jaraco]

I'm convinced the behavior needs to be suppressed for now. I was hoping we could work through the impacts and push through this time (based on the fallout from the Setuptools 48 release being much less severe).

[pradyunsg]

Setuptools 50.1.0 changes contains pypa/setuptools#2376, which changes the default to be more conservative.

[jaraco]

Closing as a duplicate of #2.