bpo-41282: (PEP 632) Load install schemes from sysconfig #24549
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The test failure |
|
Someday I will get around to making a PR with the diff to PyPy's version of |
|
I'm ambivalent on the deviations this makes from the plan, and while I've not thoroughly reviewed the code, I couldn't see anything worth flagging during a cursory look through the patch. |
frenzymadness
force-pushed
the
install_schemas_upstream
branch
from
1d485c9
to
7876985
Compare
|
Rebased to master to restart CI. |
|
This PR is stale because it has been open for 30 days with no activity. |
|
This approach seems generally in the right direction. Presumably this change would also get ported to pypa/distutils and thus bundled with setuptools. If this provides the supported mechanism for distros to override prefixes and schemes, that's great. Does this change work with Python 3.6+ also? @FFY00 Is aiding in the effort, so I'd like his opinion too. |
|
This looks fine, I was planning doing something similar. I think with this merged and ported to pypa/distutils we are just missing a install scheme argument in the CLI and a mechanism for Python distributors to define their own site schemes (they would define a custom sysconfig scheme and that scheme would be used in the site module in addition to the default one). |
github-actions
bot
removed
the
stale
frenzymadness
force-pushed
the
install_schemas_upstream
branch
from
7876985
to
04346fd
Compare
Thanks for the review. It was a good idea to test this with older Pythons because it helped me to uncover a minor bug - the platlibdir variable is available only in Python >=3.9. Fixed. The latest version here works with Python 3.6+.
As far as I know from Fedora, the way we define our own schemas is by patching the distutils module. After this change, we are just going to patch sysconfig instead of distutils. But finding a better way to do the same in an easier way is definitely worth it. Given the positive reviews, @encukou, could you please merge this? Or is there anything more I can do? |
|
We need to get this in before beta 1. |
encukou
reviewed
Apr 13, 2021
Co-authored-by: Petr Viktorin <encukou@gmail.com>
encukou
added
the
🔨 test-with-buildbots
bedevere-bot
removed
the
🔨 test-with-buildbots
|
|
The buildbot test failed on timeout which I believe has nothing to do with this PR. |
kreathon
pushed a commit
to kreathon/cpython
that referenced
this pull request
May 2, 2021
) With this patch, `distutils.command.install.INSTALL_SCHEMES` are loaded from `sysconfig._INSTALL_SCHEMES`. The distutils module is deprecated and will be removed in 3.12 (PEP 632). This change makes the `sysconfig._INSTALL_SCHEMES` the single point of truth for install schemes while keeping `distutils.command.install.INSTALL_SCHEMES` exactly the same. If we, during the transition to the sysconfig, change something, this makes sure that it also propagates to distutils until the module gets removed. Moreover, as discussed [on Discourse], Linux distros need to patch distutils/sysconfig to make sure the packages will land in proper locations. This patch makes it easier because it leaves only one location where install schemes are defined which is much easier to patch/adjust. [on Discourse]: https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134 The implementation is slightly different than the plan but I think it's the easiest way how to do it and it also makes the downstream patch simple, flexible and easy to maintain. It's also necessary to implement this before setuptools starts bundling the distutils module so the default install schemes stay in the standard library. The removed code from sysconfig does not seem to have any negative effect because, honestly, it seems that nothing actually uses the install schemes from sysconfig at all. There were many big changes in these modules where they were trying to include packaging in stdlib and then reverted that. Also, the test of distutils install command does not count with the different locations which is good evidence that the reason to have this piece of code is no longer valid. https://bugs.python.org/issue41282
encukou
added a commit
to encukou/cpython
that referenced
this pull request
May 24, 2021
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
pythonGH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
ned-deily
added a commit
that referenced
this pull request
May 24, 2021
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
GH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
May 24, 2021
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
pythonGH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
(cherry picked from commit 563bd5a)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
miss-islington
added a commit
that referenced
this pull request
May 24, 2021
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
GH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
(cherry picked from commit 563bd5a)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this patch
distutils.command.install.INSTALL_SCHEMESare loaded fromsysconfig._INSTALL_SCHEMES.The
distutilsmodule is deprecated and will be removed in 3.12 (PEP 632). This change makes thesysconfig._INSTALL_SCHEMESthe single point o truth for install schemes while keepingdistutils.command.install.INSTALL_SCHEMESexactly the same. If we, during the transition to the sysconfig, change something, this makes sure that it propagates also to distutils until the module gets removed.Moreover, as discussed there, Linux distros need to patch distutils/sysconfig to make sure the packages will land in proper locations. This patch makes it easier because it leaves only one location where install schemes are defined which is much easier to patch/adjust. If accepted, a downstream patch in Fedora might look like this.
The implementation is slightly different than the plan but I think it's the easiest way how to do it and it also makes the downstream patch simple, flexible and easy to maintain.
It's also necessary to implement this before setuptools starts bundling the distutils module so the default install schemes stay in the standard library.
The removed code from
sysconfigdoes not seem to have any negative effect because, honestly, it seems that nothing actually uses the install schemes from sysconfig at all. There were many big changes in these modules where they were trying to include packaging in stdlib and then reverted that. Also, the test of distutils install command does not count with the different locations which is good evidence that the reason to have this piece of code is no longer valid.Cc: @hroncok @encukou @jaraco @pradyunsg
https://bugs.python.org/issue41282