gh-95299: Stop installing setuptools as a part of ensurepip and venv #101039
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pradyunsg
added
stdlib
|
Hmm... @vstinner @encukou any opinions on what I should do with That test has an implicit assumption that Notably, some leading questions that affect what direction I could take for updating that test:
|
pradyunsg
force-pushed
the
remove-setuptools-ensurepip
branch
from
17a121e to
32b3ccc
Compare
|
Since it's cheap to revert commits/remove them from a PR, and nice to have a code change to discuss details over, I've gone ahead and pushed a change that assumes that the answer to the questions I raised above is yes, yes and yes. :) Mentioning a GitHub UI thing that I find very useful (and non-initutive): you can click "force-pushed" on the event above to see what changed in the force pushes between the hashes that GitHub lists in the from ... to ... at the end of the event. |
pradyunsg
force-pushed
the
remove-setuptools-ensurepip
branch
from
32b3ccc to
6939d48
Compare
hugovk
reviewed
Jan 14, 2023
vsajip
approved these changes
Jan 15, 2023
IMO, that would be best. Relatedly, I've been thinking how to handle Anyway, if that sounds like more than you signed up for, go ahead with Setuptools. |
I spent some time today looking into this after my work day, and I've not figured out the details in a portable manner. I'd prefer if this could be tackled in a follow up instead. :) |
pradyunsg
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @pradyunsg for commit 7457fd3 🤖 If you want to schedule another build, you need to add the |
bedevere-bot
removed
the
🔨 test-with-buildbots
pfmoore
approved these changes
Jan 31, 2023
pfmoore
reviewed
Jan 31, 2023
|
I added a review comment that should fix the buildbot errors. |
merwok
reviewed
Jan 31, 2023
merwok
reviewed
Jan 31, 2023
|
Should this also go in What's New, under Removed? |
The inclusion of setuptools was always explicitly documented as an internal implementation detail, and as such, I don't think we have any obligation to report its removal. It could be mentioned in the same way that we might document the removal of an internal API - I did a quick check and the 3.11 What's New included "Removed the undocumented private |
|
Please add it as a convenience to users, even if there's technically no obligation. It's been very easy to end up depending on setuptools accidentally. IMO, for removals/breaks, it's best to write What's New entries for the target audience of a poor soul who isn't too well-versed in Python but inherited some complex system with a “temporary” hack. Point them to a solution, don't apologize, but don't berate them either. (cc @python/proofreaders -- hope the above is good general advice) |
bedevere-bot
removed
the
🔨 test-with-buildbots
|
All the buildbot failures seem like false negatives, with test_asyncio, test_eintr, test_toolsaand test_typing failing for unrelated reasons. |
|
Given that this has multiple reviews, that the CI and buildbots are happy (all the non-flaky ones), that all outstanding comments are addressed and the multiple approvals at various points, I'm going to go ahead and merge this. If there's more needed here, we can handle that in a follow up! |
|
Thanks all! Holler on the issue, if there's any concerns! :) |
carljm
added a commit
to carljm/cpython
that referenced
this pull request
Apr 20, 2023
* main: (24 commits) pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561) pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545) pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634) pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633) pythongh-102856: Initial implementation of PEP 701 (python#102855) pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589) pythongh-83004: Harden msvcrt further (python#103420) pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626) pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314) pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586) pythongh-103583: Always pass multibyte codec structs as const (python#103588) pythongh-103617: Fix compiler warning in _iomodule.c (python#103618) pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600) pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576) pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613) [Doc] Fix a typo in optparse.rst (python#103504) pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531) pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039) pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569) pythongh-67230: update whatsnew note for csv changes (python#103598) ...
webknjaz
added a commit
to webknjaz/cpython
that referenced
this pull request
Sep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [2]: python#95299
webknjaz
added a commit
to webknjaz/cpython
that referenced
this pull request
Sep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [3]: python#95299
webknjaz
added a commit
to webknjaz/cpython
that referenced
this pull request
Sep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [3]: python#95299
webknjaz
added a commit
to webknjaz/cpython
that referenced
this pull request
Sep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [3]: python#95299
webknjaz
added a commit
to webknjaz/cpython
that referenced
this pull request
Jan 25, 2024
This is a refactoring change that aims to simplify ``ensurepip``. Before it, this module had legacy infrastructure that made an assumption that ``ensurepip`` would be provisioning more then just a single wheel. That assumption is no longer true since [[1]][[2]][[3]]. In this change, the improvement is done around removing unnecessary loops and supporting structures to change the assumptions to expect only the bundled or replacement ``pip`` wheel. [1]: python@ece20db [2]: python#101039 [3]: python#95299
edmorley
added a commit
to edmorley/get-pip
that referenced
this pull request
Jul 7, 2024
Currently `get-pip.py` installs not only pip, but also setuptools and wheel by default, unless the `--no-setuptools` / `--no-wheel` (or `PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used. This has historically been necessary, however, modern versions of pip will now fallback to `pyproject.toml` (PEP 517: [1]) based builds (which will default to a setuptools backend, and thus automatically install setuptools and wheel in the isolated build environment) if either setuptools is not installed (as of pip 22.1: [2]), or if wheel is not installed (as of pip 23.1: [3]). In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv` modules no longer install setuptools, and only install pip ([4]). As such, it is now time for `get-pip.py` to stop installing setuptools and wheel by default on Python 3.12+, in order to: - Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+. - Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip. Closes pypa#200. [1]: https://peps.python.org/pep-0517/ [2]: pypa/pip#10717 [3]: pypa/pip#11871 [4]: python/cpython#101039
edmorley
added a commit
to edmorley/get-pip
that referenced
this pull request
Jul 7, 2024
Currently `get-pip.py` installs not only pip, but also setuptools and wheel by default, unless the `--no-setuptools` / `--no-wheel` (or `PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used. This has historically been necessary, however, modern versions of pip will now fallback to `pyproject.toml` (PEP 517: [1]) based builds (which will default to a setuptools backend, and thus automatically install setuptools and wheel in the isolated build environment) if either setuptools is not installed (as of pip 22.1: [2]), or if wheel is not installed (as of pip 23.1: [3]). In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv` modules no longer install setuptools, and only install pip ([4]). As such, it is now time for `get-pip.py` to stop installing setuptools and wheel by default on Python 3.12+, in order to: - Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+. - Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip. Closes pypa#200. [1]: https://peps.python.org/pep-0517/ [2]: pypa/pip#10717 [3]: pypa/pip#11871 [4]: python/cpython#101039
edmorley
added a commit
to edmorley/get-pip
that referenced
this pull request
Jul 29, 2024
Currently `get-pip.py` installs not only pip, but also setuptools and wheel by default, unless the `--no-setuptools` / `--no-wheel` (or `PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used. This has historically been necessary, however, modern versions of pip will now fallback to `pyproject.toml` (PEP 517: [1]) based builds (which will default to a setuptools backend, and thus automatically install setuptools and wheel in the isolated build environment) if either setuptools is not installed (as of pip 22.1: [2]), or if wheel is not installed (as of pip 23.1: [3]). In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv` modules no longer install setuptools, and only install pip ([4]). As such, it is now time for `get-pip.py` to stop installing setuptools and wheel by default on Python 3.12+, in order to: - Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+. - Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip. Closes pypa#200. [1]: https://peps.python.org/pep-0517/ [2]: pypa/pip#10717 [3]: pypa/pip#11871 [4]: python/cpython#101039
edmorley
added a commit
to edmorley/get-pip
that referenced
this pull request
Jul 29, 2024
Currently `get-pip.py` installs not only pip, but also setuptools and wheel by default, unless the `--no-setuptools` / `--no-wheel` (or `PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used. This has historically been necessary, however, modern versions of pip will now fallback to `pyproject.toml` (PEP 517: [1]) based builds (which will default to a setuptools backend, and thus automatically install setuptools and wheel in the isolated build environment) if either setuptools is not installed (as of pip 22.1: [2]), or if wheel is not installed (as of pip 23.1: [3]). In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv` modules no longer install setuptools, and only install pip ([4]). As such, it is now time for `get-pip.py` to stop installing setuptools and wheel by default on Python 3.12+, in order to: - Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+. - Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip. Closes pypa#200. [1]: https://peps.python.org/pep-0517/ [2]: pypa/pip#10717 [3]: pypa/pip#11871 [4]: python/cpython#101039
edmorley
added a commit
to heroku/buildpacks-python
that referenced
this pull request
Aug 7, 2024
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels. However: - Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally. There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway. As such, it's no longer necessary for us to install setuptools and wheel globally. This matches the behaviour of the `venv` and `ensurepip` modules in Python 3.12+, where setuptools and wheel installation has also been removed. And it also matches the default behaviour of Poetry too, whose `install --sync` command removes any implicitly installed packages in the current environment (other than pip). See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252 GUS-W-16437776.
edmorley
added a commit
to heroku/buildpacks-python
that referenced
this pull request
Aug 7, 2024
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels. However: - Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally. There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway. As such, it's no longer necessary for us to install setuptools and wheel globally. This matches the behaviour of the `venv` and `ensurepip` modules in Python 3.12+, where setuptools and wheel installation has also been removed. And it also matches the default behaviour of Poetry too, whose `install --sync` command removes any implicitly installed packages in the current environment (other than pip). See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252 GUS-W-16437776.
edmorley
added a commit
to heroku/buildpacks-python
that referenced
this pull request
Aug 7, 2024
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels. However: - Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally. There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway. As such, it's no longer necessary for us to install setuptools and wheel globally. This matches the behaviour of the `venv` and `ensurepip` modules in Python 3.12+, where setuptools and wheel installation has also been removed. And it also matches the default behaviour of Poetry too, whose `install --sync` command removes any implicitly installed packages in the current environment (other than pip). See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252 GUS-W-16437776.
Tenzer
added a commit
to Tenzer/aws-lambda-python-runtime-interface-client
that referenced
this pull request
Oct 10, 2024
It is necessary for building the package when running the integration tests, but more recent Python versions no longer include it by default: python/cpython#101039.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the bundled setuptools wheel from ensurepip, and stops installing setuptools in environments created by venv.
I based this PR off of my understanding of venv and ensurepip, and using the following search to locate all the relevant bits of code + ensuring that tests are happy locally.
In this PR, I've intentionally not touched the example in https://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder. That example is largely outdated and is likely better served being updated in a dedicated PR for it.