Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental support for Python 3.10 in Sage 9.5 #30766

Closed
slel opened this issue Oct 13, 2020 · 137 comments
Closed

Experimental support for Python 3.10 in Sage 9.5 #30766

slel opened this issue Oct 13, 2020 · 137 comments

Comments

@slel
Copy link
Member

slel commented Oct 13, 2020

This is to support Python 3.10, which was released on 2021-10-04.
https://docs.python.org/3.10/whatsnew/changelog.html#changelog

Thanks to the following tickets, Sage is mostly ready for
Python 3.10 in addition to 3.9, 3.8, 3.7.

In this ticket, we enable use of system Python 3.10 but issue a warning regarding its experimental status for Sage.

The ticket does not include the upgrade of the python3 SPKG to Python 3.10. Many systems already ship Python 3.10. To test, thanks to https://wiki.sagemath.org/ReleaseTours/sage-9.5#Separate_virtual_environment_for_Python_packages, a full rebuild of the Sage distribution is not necessary - only the Python packages will be built from scratch in a separate venv.

Related:

Depends on #33013
Depends on #32930
Depends on #33040

CC: @mkoeppe @slel @kiwifb @tornaria @antonio-rojas

Component: packages: standard

Keywords: upgrade, python

Author: Matthias Koeppe, Gonzalo Tornaría

Branch: df9f1d4

Reviewer: Antonio Rojas, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/30766

@slel
Copy link
Member Author

slel commented Oct 13, 2020

Changed keywords from none to upgrade, python

@slel

This comment has been minimized.

@slel
Copy link
Member Author

slel commented Oct 15, 2020

Dependencies: #30768

@slel

This comment has been minimized.

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor

mkoeppe commented Apr 14, 2021

Changed dependencies from #30768 to #30768, #31445

@mkoeppe mkoeppe added this to the sage-9.4 milestone Apr 14, 2021
@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor

mkoeppe commented May 24, 2021

Branch: u/mkoeppe/support_python_3_10

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor

mkoeppe commented May 24, 2021

New commits:

6ca849fbuild/pkgs/cython: Update to 0.29.23
4e4fd58Merge #31445

@mkoeppe
Copy link
Contributor

mkoeppe commented May 24, 2021

Commit: 4e4fd58

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor

mkoeppe commented May 25, 2021

Changed dependencies from #30768, #31445 to #30768, #31445, #31855

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Changed commit from 4e4fd58 to 2d7d715

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

8cd9ecebuild/pkgs/gmpy2/patches: Add patches from Fedora @python/python3.10
2d7d715Merge #31855

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

3321a59build/pkgs/python3/spkg-configure.m4: Accept system python 3.10.x

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Changed commit from 2d7d715 to 3321a59

@mkoeppe

This comment has been minimized.

@mkoeppe

This comment has been minimized.

@mkoeppe
Copy link
Contributor

mkoeppe commented May 25, 2021

Changed dependencies from #30768, #31445, #31855 to #30768, #31445, #31855, #31856

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Changed commit from 3321a59 to 65355f9

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

2a7d940build/pkgs/pyzmq: Update to 22.0.3
65355f9Merge #31856

@mkoeppe
Copy link
Contributor

mkoeppe commented May 25, 2021

Changed dependencies from #30768, #31445, #31855, #31856 to #30768, #31445, #31855, #31856, #31008

@mkoeppe

This comment has been minimized.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented May 25, 2021

Changed commit from 65355f9 to a661b6a

@tornaria
Copy link
Contributor

comment:95

I change the workarounds by an attempt at a "proper" fix.

With this all the doctests in src/sage/cpython/atexit.pyx pass, both with bundled python 3.9.7 and with my system python 3.10.

I'm still to run the whole testsuite; I am still getting failures in src/sage_docbuild/utils.py but I think they are due to the deprecation warnings caused by matplotlib and should go away when I apply #33040 which I forgot to do.

Should #33040 be merged into this ticket as a dependency? Am I the only one having problems caused by old matplotlib?


New commits:

249357dsage.cpython.atexit: fix for python 3.10 (#30766)

@tornaria
Copy link
Contributor

comment:96

Notes on the implementation:

Everything is in the cdef extern from * block where I moved the definition of struct atexit_callback and a new function _atexit_callbacks().

The C implementation for python 3.9 is short and should be equivalent to what was before. The implementation for 3.10 is even shorter since structs are now defined in (internal) python headers.

The argument to _atexit_callbacks() should be the atexit module as in the body of _get_exithandlers(). This is not necessary for python 3.10 but it seemed simpler to have the same declaration for both cases so there are no conditionals outside the cdef extern from * block.

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 19, 2021

Changed dependencies from #33013, #32930 to #33013, #32930, #33040

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 19, 2021

comment:98

Looks good to me but it needs to be tested also on Python < 3.10

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 19, 2021

Changed author from Matthias Koeppe to Matthias Koeppe, Gonzalo Tornaría

@tornaria
Copy link
Contributor

comment:100

Replying to @mkoeppe:

Looks good to me but it needs to be tested also on Python < 3.10

Absolutely! I did one test by passing --without-system-python3 to configure but more testing would be desirable.

Although I think the changes should really be nil for python < 3.10; would it be useful for review if I split the patch in two parts where the first part just refactors the existing code and the second part adds the new code for python 3.10? With some luck the first part visibly does nothing, and the second part will only add code conditional to python >= 3.10.

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 19, 2021

comment:101

Replying to @tornaria:

would it be useful for review if I split the patch in two parts where the first part just refactors the existing code and the second part adds the new code for python 3.10?

No, it's fine as is

@tornaria
Copy link
Contributor

comment:102

My full clean build with python 3.10 finished with the following doctest failure:

sage -t --random-seed=262548395979960264449538514011341238171 src/sage_docbuild/utils.py
**********************************************************************
File "src/sage_docbuild/utils.py", line 91, in sage_docbuild.utils
Failed example:
    build_many(target, range(8), processes=8)
Expected:
    Traceback (most recent call last):
    ...
    ZeroDivisionError: rational division by zero
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_docbuild.utils[6]>", line 1, in <module>
        build_many(target, range(Integer(8)), processes=Integer(8))
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_docbuild/utils.py", line 289, in build_many
        raise worker_exc.original_exception
    TypeError: 'sage.rings.real_mpfr.RealLiteral' object cannot be interpreted as an integer
**********************************************************************
File "src/sage_docbuild/utils.py", line 110, in sage_docbuild.utils
Failed example:
    build_many(target, range(8), processes=8)
Expected:
    Traceback (most recent call last):
    ...
    WorkerDiedException: worker for 4 died with non-zero exit code -9
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_docbuild.utils[8]>", line 1, in <module>
        build_many(target, range(Integer(8)), processes=Integer(8))
      File "/usr/lib/sage-9.5.beta8/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_docbuild/utils.py", line 289, in build_many
        raise worker_exc.original_exception
    TypeError: 'sage.rings.real_mpfr.RealLiteral' object cannot be interpreted as an integer
**********************************************************************
1 item had failures:
   2 of  10 in sage_docbuild.utils
    [9 tests, 2 failures, 1.43 s]

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 20, 2021

Branch pushed to git repo; I updated commit sha1. New commits:

df9f1d4Pass float(0.5) to time.sleep

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 20, 2021

Changed commit from 249357d to df9f1d4

@tornaria
Copy link
Contributor

comment:104

This seems to be the root cause:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.5.beta8, Release Date: 2021-12-12               │
│ Using Python 3.10.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: import time
sage: time.sleep(0.5)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-b448069dbb3c> in <module>
----> 1 time.sleep(RealNumber('0.5'))

TypeError: 'sage.rings.real_mpfr.RealLiteral' object cannot be interpreted as an integer

Now the same thing --without-system-python:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.5.beta8, Release Date: 2021-12-12               │
│ Using Python 3.9.7. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: import time
sage: time.sleep(0.5)
<ipython-input-2-b448069dbb3c>:1: DeprecationWarning: an integer is required (got type sage.rings.real_mpfr.RealLiteral).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
  time.sleep(RealNumber('0.5'))

@tornaria
Copy link
Contributor

comment:105

Everything passes now for me using system python 3.10.1 on void linux:

$ ./sage -tp 36 --optional sage,optional,build --all
...
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 236.9 seconds
    cpu time: 7024.5 seconds
    cumulative wall time: 6565.8 seconds

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 20, 2021

Reviewer: ..., Matthias Koeppe

@antonio-rojas
Copy link
Contributor

comment:107

All tests passing here too, thanks!

@antonio-rojas
Copy link
Contributor

Changed reviewer from ..., Matthias Koeppe to Antonio Rojas, Matthias Koeppe

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 20, 2021

comment:108

Thanks!

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 20, 2021

comment:109

The dependency #33040 needs review so that we can get this ticket merged.

@vbraun
Copy link
Member

vbraun commented Jan 1, 2022

Changed branch from u/tornaria/support_python_3_10 to df9f1d4

@antonio-rojas
Copy link
Contributor

Changed commit from df9f1d4 to none

@antonio-rojas
Copy link
Contributor

comment:111

Something regressed in 9.5.rc0 which is causing distutils deprecation warnings

File "/usr/lib/python3.10/site-packages/sage/env.py", line 359, in sage.env.sage_include_directories
Failed example:
    sage.env.sage_include_directories()
Expected:
    ['.../include/python...',
    '.../python.../numpy/core/include']
Got:
    doctest:warning
      File "/usr/bin/sage-runtests", line 155, in <module>
        err = DC.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1256, in run
        self.run_doctests()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 957, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2004, in dispatch
        self.parallel_dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1899, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2173, in start
        super(DocTestWorker, self).start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2145, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2475, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2527, in _run
        result = runner.run(test)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 861, in run
        return self._run(test, compileflags, out)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.env.sage_include_directories[1]>", line 1, in <module>
        sage.env.sage_include_directories()
      File "/usr/lib/python3.10/site-packages/sage/env.py", line 375, in sage_include_directories
        import distutils.sysconfig
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/distutils/__init__.py", line 19, in <module>
        warnings.warn(_DEPRECATION_MESSAGE,
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    doctest:warning
      File "/usr/bin/sage-runtests", line 155, in <module>
        err = DC.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1256, in run
        self.run_doctests()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 957, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2004, in dispatch
        self.parallel_dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1899, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2173, in start
        super(DocTestWorker, self).start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2145, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2475, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2527, in _run
        result = runner.run(test)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 861, in run
        return self._run(test, compileflags, out)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.env.sage_include_directories[1]>", line 1, in <module>
        sage.env.sage_include_directories()
      File "/usr/lib/python3.10/site-packages/sage/env.py", line 375, in sage_include_directories
        import distutils.sysconfig
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/distutils/sysconfig.py", line 58, in <module>
        warnings.warn(
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
    ['/usr/lib/python3.10/site-packages',
     '/usr/include/python3.10',
     '/usr/lib/python3.10/site-packages/numpy/core/include']
**********************************************************************

These didn't happen with 9.5.beta9 + this branch.

@antonio-rojas
Copy link
Contributor

comment:112

Replying to @antonio-rojas:

Something regressed in 9.5.rc0 which is causing distutils deprecation warnings

This comes from #32938, in particular form the removal of

from Cython.Build.Dependencies import strip_string_literals as cython_strip_string_literals

from sage.doctest.parsing. That import used to throw a distutils deprecation warning that for some reason was ignored during doctesting, and now that it's gone the warning is printed when doctesting any module that calls distutils.sysconfig.

@sheerluck
Copy link
Contributor

comment:113

adding internet to --optional= shows that all context=SSLContext() should be replaced with context=ssl.create_default_context()

@tornaria
Copy link
Contributor

tornaria commented Jan 9, 2022

comment:114

These three seem to fix all the warnings for me (numpy 1.22 from system via #29665)

--- a/src/sage/all.py	2022-01-09 15:34:29.833828362 -0300
+++ b/src/sage/all.py	2022-01-09 15:39:40.150091390 -0300
@@ -96,6 +96,15 @@
 warnings.filterwarnings('ignore', category=DeprecationWarning,
                         module='pythran')
 
+warnings.filterwarnings('ignore', category=DeprecationWarning,
+			module='numpy', message='.*distutils.*')
+
+warnings.filterwarnings('ignore', category=DeprecationWarning,
+			module='sage.env', message='.*distutils.*')
+
+warnings.filterwarnings('ignore', category=DeprecationWarning,
+			module='sage.features', message='.*distutils.*')
+
 ################ end setup warnings ###############################
 
 

Edit: fixed paths in the patch.

@tornaria
Copy link
Contributor

tornaria commented Jan 9, 2022

comment:115

Please see and review #33135.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants