Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare python3 spkg-configure.m4 for Python 3.12 (setuptools instead of distutils) #36983

Merged
merged 5 commits into from
Feb 13, 2024
Merged

Prepare python3 spkg-configure.m4 for Python 3.12 (setuptools instead of distutils) #36983

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a2fc656
build/pkgs/python3/spkg-configure.m4, m4/sage_check_python_for_venv.m…
mkoeppe Dec 15, 2023
2120c62
build/pkgs/python3/distros: Add setuptools packages
mkoeppe Dec 15, 2023
79c6a2d
Merge branch 'tox_voidlinux' into python3_configure_prepare_312
mkoeppe Jan 5, 2024
8fa7f12
Merge branch 'develop' into python3_configure_prepare_312
mkoeppe Jan 14, 2024
0e54d2c
Merge branch 'develop' into python3_configure_prepare_312
mkoeppe Feb 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion build/pkgs/_python3.12/distros/debian.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
python3.12
python3.12-dev
python3.12-distutils
python3.12-venv
9 changes: 4 additions & 5 deletions build/pkgs/python3/SPKG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,15 @@ By default, Sage will try to use system's ``python3`` to set up a virtual
environment, a.k.a. `venv <https://docs.python.org/3.10/library/venv.html>`_
rather than building a Python 3 installation from scratch.

Sage will accept versions 3.9.x to 3.10.x.
Sage will accept versions 3.9.x to 3.11.x.

You can also use ``--with-python=/path/to/python3_binary`` to tell Sage to use
``/path/to/python3_binary`` to set up the venv. Note that setting up the venv requires
a number of Python modules to be available within the Python in question. Currently,
as of Sage 9.7, these modules are as follows: ``sqlite3``, ``ctypes``, ``math``,
``hashlib``, ``socket``, ``zlib``, ``distutils.core``, ``ssl`` -
they will be checked for by the ``configure`` script.
as of Sage 10.3, these modules are as follows: ``sqlite3``, ``ctypes``, ``math``,
``hashlib``, ``socket``, ``zlib``, ``ssl``, ``ensurepip``.

Use the ``configure`` option ``--without-system-python3`` in case you want Python 3
Use the ``configure`` option ``--without-system-python3`` if you want Python 3
built from scratch.


Expand Down
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/alpine.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
python3-dev
py3-setuptools
2 changes: 1 addition & 1 deletion build/pkgs/python3/distros/debian.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
python3
libpython3-dev
python3-distutils
python3-setuptools
python3-venv
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/fedora.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
python3-devel
python-setuptools
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/freebsd.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
lang/python
devel/py-setuptools
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/homebrew.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
# This installs /usr/local/bin/python3 -> python3.9
python3
python-setuptools
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/macports.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
python310
py-setuptools
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/opensuse.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
python3${PYTHON_MINOR}-devel
python3${PYTHON_MINOR}-setuptools
1 change: 1 addition & 0 deletions build/pkgs/python3/distros/void.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
python3-devel
python3-setuptools
6 changes: 3 additions & 3 deletions build/pkgs/python3/spkg-configure.m4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ SAGE_SPKG_CONFIGURE([python3], [
dnl Check if we can do venv with a system python3
dnl instead of building our own copy.
dnl Trac #31160: We no longer check for readline here.
check_modules="sqlite3, ctypes, math, hashlib, socket, zlib, distutils.core, ssl, ensurepip"
AC_CACHE_CHECK([for python3 >= ]MIN_VERSION[, < ]LT_VERSION[ with modules $check_modules], [ac_cv_path_PYTHON3], [
check_modules="sqlite3, ctypes, math, hashlib, socket, zlib, ssl, ensurepip"
AC_CACHE_CHECK([for python3 >= ]MIN_VERSION[, < ]LT_VERSION[ with modules $check_modules and setuptools/distutils], [ac_cv_path_PYTHON3], [
AS_IF([test x"$ac_path_PYTHON3" != x], [dnl checking explicitly specified $with_python
AC_MSG_RESULT([])
AC_PATH_PROG([ac_path_PYTHON3], [$ac_path_PYTHON3])
Expand All @@ -40,7 +40,7 @@ SAGE_SPKG_CONFIGURE([python3], [
ac_path_PYTHON3_found=:
AC_MSG_RESULT([yes])
dnl introduction for AC_MSG_RESULT printed by AC_CACHE_CHECK
AC_MSG_CHECKING([for python3 >= ]MIN_VERSION[, < ]LT_VERSION[ with modules $check_modules])
AC_MSG_CHECKING([for python3 >= ]MIN_VERSION[, < ]LT_VERSION[ with modules $check_modules and setuptools/distutils])
])
AS_IF([test -z "$ac_cv_path_PYTHON3"], [
AC_MSG_ERROR([the python3 selected using --with-python=$with_python is not suitable])
Expand Down
28 changes: 21 additions & 7 deletions m4/sage_check_python_for_venv.m4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,23 @@ AC_DEFUN([SAGE_CHECK_PYTHON_FOR_VENV], [
AX_COMPARE_VERSION([$python3_version], [ge], MIN_VERSION, [
AX_COMPARE_VERSION([$python3_version], [lt], LT_VERSION, [
dnl Because the system python is not used directly but rather in a venv without site-packages,
dnl we test whether the module will be available in a venv.
dnl we should test whether the module will be available in a venv.
dnl Otherwise, some system site-package may be providing this module to the system python.
dnl However, on Python >= 3.12, we need setuptools to run our extension compilation tests
dnl because distutils has been removed from the standard library.
AX_COMPARE_VERSION([$python3_version], [ge], [3.12.0], [
conftest_venv_options="--system-site-packages"
distutils_core="setuptools"
Comment on lines +23 to +25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move to setuptools unconditionally, instead of only for python 3.12?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, that would require people to have an additional system package installed that isn't going to be used other than for this configure check.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just setuptools. Besides, what makes it ok to require setuptools for python 3.12 but not for python 3.11?

In fact, it seems you don't really need to require the presence of neither distutils nor setuptools at all. If distutils or setuptools is available from system, it can be used, otherwise setuptools will be installed in the venv and everything will be fine.

The only reason one checks for a few modules here is because they are built-in python modules that can't be installed after the fact on top of a system python. But setuptools can (and will) be installed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what makes it ok to require setuptools for python 3.12

You know the answer. It's OK because without it we cannot test what we want to test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If distutils or setuptools is available from system, it can be used, otherwise setuptools will be installed in the venv and everything will be fine.

Not really. We stopped using distutils from the system a long time ago (by setting SETUPTOOLS_USE_DISTUTILS=local), and we do not use setuptools from the system (except in the experimental system-site mode, which I'll not comment on).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand. All of what I wrote is about SageMath

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"We had quite a bit of trouble with this on macOS (both plain Xcode and homebrew)."

I was asking for an example of this, i.e. broken systems that cause trouble with sagemath.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

macOS Xcode + homebrew was pretty problematic in #31227.

And the link to python/cpython#90539 (comment) that I sent earlier has the pointers to Sage tickets where Python was unusable on Homebrew, Cygwin, and pyenv.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also openSUSE was affected by the same (#33153).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks.

distutils_extension="setuptools.extension"
], [
conftest_venv_options=
distutils_core="distutils.core"
distutils_extension="distutils.extension"
])
all_required_modules="]REQUIRED_MODULES[, $distutils_core, $distutils_extension"
dnl m4_define([conftest_venv], [config-venv]) .... for debugging only
rm -rf conftest_venv
AS_IF(["]PYTHON_EXE[" build/bin/sage-venv conftest_venv && conftest_venv/bin/python3 -c "import ]REQUIRED_MODULES[" 2>& ]AS_MESSAGE_LOG_FD, [
AS_IF(["]PYTHON_EXE[" build/bin/sage-venv $conftest_venv_options conftest_venv && conftest_venv/bin/python3 -c "import $all_required_modules" 2>& ]AS_MESSAGE_LOG_FD, [
AS_VAR_SET([python3_result], [yes])
SAGE_PYTHON_CHECK_DISTUTILS([CC="$CC" CXX="$CXX" conftest_venv/bin/python3], [
SAGE_ARCHFLAGS="unset"
Expand All @@ -46,7 +58,7 @@ AC_DEFUN([SAGE_CHECK_PYTHON_FOR_VENV], [
AC_MSG_RESULT([$python3_result])
])
], [
AC_MSG_RESULT([no, the version is in the supported range but cannot import one of the required modules: ]REQUIRED_MODULES)
AC_MSG_RESULT([no, the version is in the supported range but cannot import one of the required modules: $all_required_modules])
])
], [
AC_MSG_RESULT([no, $python3_version is too recent])
Expand Down Expand Up @@ -123,8 +135,9 @@ PyInit_spam(void)
])
AC_LANG_POP([C])
cat > conftest.py <<EOF
from distutils.core import setup
from distutils.extension import Extension

from $distutils_core import setup
from $distutils_extension import Extension
from sys import exit
modules = list((Extension("config_check_distutils", list(("conftest.c",))),))
setup(name="config_check_distutils", ext_modules=modules)
Expand Down Expand Up @@ -176,8 +189,9 @@ PyInit_spam(void)
])
AC_LANG_POP([C++])
cat > conftest.py <<EOF
from distutils.core import setup
from distutils.extension import Extension

from $distutils_core import setup
from $distutils_extension import Extension
from sys import exit
modules = list((Extension("config_check_distutils_cxx", list(("conftest.cpp",)),
extra_compile_args=list(("-std=c++11",)), language="c++"),))
Expand Down
Loading