Upgrade: Python 3.8.5

[slel]

This is to upgrade to Python 3.8.x.

• PEP569 -- Python 3.8 Release Schedule
• What's new in Python 3.8

Tarball: see checksums.ini

Previous and next Python upgrades:

• Upgrade to Python 3.7.x #25680 - Python 3.7.3
• Upgrade Python to 3.7.8 #30210 - Python 3.7.8
• Support Python 3.9 #30184 - Python 3.9.x

See also:

• Adopt the “time window-based” policy for support of Python versions from NEP 29 #30384 Adopt the “time window-based” policy for support of Python versions from NEP 29

Depends on #28197
Depends on #29042
Depends on #29861
Depends on #30162
Depends on #30225
Depends on #30127
Depends on #30144
Depends on #30345
Depends on #30365
Depends on #30140
Depends on #30351

Upstream: Reported upstream. Developers acknowledge bug.

CC: @fchapoton @embray @kiwifb @jdemeyer @mkoeppe @slel @timokau @isuruf @kliem @tscrim @heluani @sheerluck

Component: packages: standard

Keywords: days101

Author: Jeroen Demeyer, John Palmieri, Matthias Koeppe, Antonio Rojas

Branch: f1cda12

Reviewer: Matthias Koeppe, François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/27754

[slel]

comment:1

If people want to start testing, Python 3.8.0a3 is out:

• Python 3.8.0a3

[jdemeyer]

comment:2

I think it's a bit early for testing. I'd rather wait for the first beta, which is supposed to be released later this month.

[slel]

comment:3

Python 3.8.0b1 is out:

• Python 3.8.0b1

[jdemeyer]

comment:4

Note: this requires a Cython upgrade.

[slel]

comment:5

Cython upgrade needs review at #27886.

[slel]

Changed keywords from none to days101

[slel]

Dependencies: #28012

[jdemeyer]

comment:7

I'll have a look at this today.

[jdemeyer]

Changed dependencies from #28012 to #27886

[jdemeyer]

Author: Jeroen Demeyer

[jdemeyer]

Branch: u/jdemeyer/upgrade__python_3_8_x

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

46837b7

Python 3.8.0

[sagetrac-git]

Commit: 46837b7

[jdemeyer]

Changed dependencies from #27886 to none

[jdemeyer]

comment:12

Actually, thanks to two patches which are likely going to be accepted upstream, it no longer depends on the Cython upgrade.

[jdemeyer]

Upstream: Reported upstream. Developers acknowledge bug.

[jdemeyer]

comment:14

There is a mysterious build failure of matplotlib:

Processing /home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
  Created temporary directory: /tmp/pip-req-build-vq4okaxf
  Added file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/tmp/pip-req-tracker-xcv165b_'
  Running setup.py (path:/tmp/pip-req-build-vq4okaxf/setup.py) egg_info for package from file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-vq4okaxf/setup.py", line 172, in <module>
        result = package.check()
      File "/tmp/pip-req-build-vq4okaxf/setupext.py", line 963, in check
        import numpy
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/__init__.py", line 16, in <module>
        from . import multiarray
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
        from . import overrides
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/overrides.py", line 8, in <module>
        from numpy.compat._inspect import getargspec
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/__init__.py", line 14, in <module>
        from . import py3k
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/py3k.py", line 15, in <module>
        from pathlib import Path, PurePath
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 391, in <module>
        class _NormalAccessor(_Accessor):
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 415, in _NormalAccessor
        link_to = os.link
    AttributeError: module 'os' has no attribute 'link'

But when running Python normally, os.link does exist:

jdemeyer@sage4:~/sage-python3$ ./sage --python3
Python 3.8.0b1 (default, Jun 19 2019, 16:54:26) 
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.link
<built-in function link>

I have no clue how this could happen.

[jhpalmieri]

comment:15

I tried this with OS X and ran into a problem with Pillow:

  File "setup.py", line 316, in build_extensions
    _add_directory(library_dirs, sysroot+"/usr/lib")
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

Python 3.7.3 does this:

Python 3.7.3 (default, Jun 17 2019, 15:38:22) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk'
>>>

The new Python does this:

Python 3.8.0b1 (default, Jun 19 2019, 11:09:54) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
>>>

[jhpalmieri]

comment:16

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

If I run make -k, I also see problems building the Sage library, by the way: lots of errors of the form

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

[jdemeyer]

comment:17

Replying to @jhpalmieri:

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

That makes no sense. Somebody is fooling with us...

Could you try also print(os)? Maybe there is a different os module shadowing the standard one or something...

Just in case, I checked that the problem unfortunately persists after make distclean.

And a Google search for this error doesn't find anything.

[jhpalmieri]

comment:18

I'm guessing the problem is coming from matplotlib's setup.py:

import os
try:
    del os.link
except AttributeError:
    pass

[jhpalmieri]

comment:19

My question is why it worked with Python 3.7.3 but not 3.8.0.

[jhpalmieri]

comment:209

Okay, this works for me on OS X with the system Python and Sage's Python 3.8, and the same on a virtual Debian 32-bit machine.

[mkoeppe]

comment:210

Great!

[vbraun]

comment:211

Merge conflict

[sagetrac-git]

Changed commit from b9d451c to a9b6965

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

488bba4	on macOS set OMP_NUM_THREADS to 1 for docbuild and tests
4b11a70	use the correct import location, not six.*
a9b6965	Merge commit '4b11a706e1267b385099d8691b4468efd974f94d' of git://trac.sagemath.org/sage into t/27754/public/python3.8

[mkoeppe]

Changed dependencies from #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140 to #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140, #30351

[sagetrac-git]

Changed commit from a9b6965 to f1cda12

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

f1cda12

Fixup merge

[EmmanuelCharpentier]

comment:218

FWIW, worked for MeToo(TM) :

Rebuilding 2.9.beta11+#27754 on Debian sesting running on core i7 + 16 GB RAM gives results identical to those obtained on the same machine without #27754 (see sage-release). And $sSAGE_LOCAL\bin\python* are symlinks to \usr\bin\...

[dimpase]

comment:219

it even works on Rasberry Pi 4 (armv7l).

[vbraun]

Changed branch from public/python3.8 to f1cda12

[slel]

comment:221

Wonderful to have this in Sage 9.2.

Next episode:

• Support Python 3.9 #30184: Support Python 3.9

(Python 3.9.rc2 is expected on 2020-09-14
and Python 3.9 final on 2020-10-05.)

[slel]

Changed commit from f1cda12 to none