macos-11.0: scipy build fails #31326

mkoeppe · 2021-02-02T17:32:25Z

Affected configurations (From https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/GXfB1rW1AgAJ)

local-macos (homebrew-macos, minimal, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782309 -- Sage installs python3 from spkg
local-macos (conda-forge-macos, standard, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782333 - uses python3 from conda
local-macos (homebrew-macos-python3_pythonorg, standard, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782375 - uses somewhat dated python3 from python.org - https://www.python.org/ftp/python/3.7.7/python-3.7.7-macosx10.9.pkg
another failure report, for a configuration possibly similar to local-macos (homebrew-macos, minimal, default, macos-11.0) , in Meta-ticket: support macOS 11 Big Sur #30651:

      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 90, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 657, in CCompiler_get_version
        version = matcher(output)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/fcompiler/gnu.py", line 278, in version_match
        v = self.gnu_version_match(version_string)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.9/site-packages/numpy/distutils/fcompiler/gnu.py", line 80, in gnu_version_match
        raise ValueError(err + version_string)
    ValueError: A valid Fortran version was not found in this string:

see also https://github.com/mkoeppe/sage/runs/2103782381

CC: @jhpalmieri @zlscherr @dimpase @kliem @isuruf

Component: porting

Author: Zachary Scherr

Branch/Commit: 4cc8c4d

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/31326

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-03-15T00:10:41Z

comment:3

It seems like numpy.distutils gets confused by extra whitespace in the result of gfortran -dumpversion. This might be related to the use of zsh as the shell.

This will need investigation by someone who has already made the switch to Big Sur...

zlscherr · 2021-03-15T02:04:32Z

comment:4

just want to point out that this problem exists on bash as well. For anyone wanting to test you can, after configuring, do:

make toolchain
make gfortran
FC="local/bin/gfortran" make numpy

to see the error.

zlscherr · 2021-03-15T03:37:00Z

comment:5

I haven't tested the full homebrew-macos-11.0-xcode-minimal build, but I'm guessing it's related to MACOSX_DEPLOYMENT_TARGET.

As I noted above, I can recreate the error with numpy by using sage's gfortran 9.2.0 on my machine with homebrew. As per usual I think this is because python was built with MACOSX_DEPLOYMENT_TARGET 11.0 and gfortran 9.2.0 can't recognize this value.

Now I know that at some point scipy's spkg-install.in file was changed to contain the lines:

if [ $MACOSX_VERSION -ge 20 ]; then
     export MACOSX_DEPLOYMENT_TARGET=11.0
fi

I'm not exactly sure what is in homebrew-macos-11.0-xcode-minimal, but if sage is building its own gfortran and its own python using a different MACOSX_DEPLOYMENT_TARGET then I would guess that this line might be causing the problems.

It's been a while since I've thought about this problem, but it's possible that the above export might not be needed anymore since homebrew patched GCC.

zlscherr · 2021-03-15T03:39:53Z

comment:6

Actually it probably shouldn't matter whether or not sage is building its own python. gfortran 9.2.0 just can't recognize MACOSX_DEPLOYMENT_TARGET=11.0.

zlscherr · 2021-03-15T03:41:25Z

comment:7

Sorry for the spam, but the ticket that introduced that if statement is #31183

mkoeppe · 2021-03-15T03:43:28Z

comment:8

Could you check if the gcc upgrade in #29827 fixes this issue?

mkoeppe · 2021-03-15T03:46:07Z

comment:9

Replying to @zlscherr:

I'm not exactly sure what is in homebrew-macos-11.0-xcode-minimal, but if sage is building its own gfortran

Yes, that's right. This configuration uses a minimal installation of homebrew, plus the packages listed in build/pkgs/_bootstrap/distros/homebrew.txt, namely gettext autoconf automake libtool pkg-config

zlscherr · 2021-03-15T03:54:55Z

comment:10

I also just saw that the error reported on top is in numpy and not in scipy. I'll take a look at #29827 when I get a chance, but I know that homebrew had to specifically patch GCC to avoid this sort of problem. A viable solution might be to track the legacy option to see if sage is building its own Fortran, and if so we set MACOSX_DEPLOYMENT_TARGET to 10.something

mkoeppe · 2021-03-15T03:56:01Z

comment:11

No, it's an error building scipy -- which uses numpy.distutils, which you see in the lines of the log

zlscherr · 2021-03-15T04:11:36Z

comment:12

Ah okay. Also, do you know if the homebrew minimal test uses system for python? That might be responsible for why the deployment Target is set to 11, whereas when sage builds it's own python the deployment Target is set lower.

mkoeppe · 2021-03-15T04:18:59Z

comment:13

From https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/GXfB1rW1AgAJ: Runs in which scipy fails to build:

local-macos (homebrew-macos, minimal, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782309 -- Sage installs python3 from spkg
local-macos (conda-forge-macos, standard, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782333 - uses python3 from conda
local-macos (homebrew-macos-python3_pythonorg, standard, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782375 - uses somewhat dated python3 from python.org - https://www.python.org/ftp/python/3.7.7/python-3.7.7-macosx10.9.pkg

jhpalmieri · 2021-03-15T04:27:21Z

comment:14

For what it's worth, if I do

% make distclean && ./configure --with-system-gfortran=no && make

then the build fails on numpy, not scipy, but with the same sort of error:

ValueError: A valid Fortran version was not found in this string:

9.2.0

mkoeppe · 2021-03-15T04:35:18Z

comment:15

I think there are extra newlines in this version string, which confuses the numpy.distutils code

zlscherr · 2021-03-16T02:16:23Z

comment:16

It's actually worse than just extra newlines. I tried manually patching to strip all whitespace from both ends, but that didn't solve the problem. I dumped the contents of the version_string which is causing numpy to fail into a text file and it turned out to be

"gfortran: warning: couldn't understand version 11

9.3.0"

(I tested with gcc 9.3, but the same happens with 9.2)

I can get numpy and scipy to build by manually forcing MACOSX_DEPLOYMENT_TARGET to be 10.16 but it seems that 11 and 11.0 both cause problems.

Here is a proposal for how to deal with this:

Remove the if statements from scipy's spkg-install to set MACOSX_DEPLOYMENT_TARGET to 11.0. I don't think this is needed anymore since Homebrew patched GCC 10 but I can do some testing on that.
At some point in the build process we take a look at MACOSX_DEPLOYMENT_TARGET, which can be easily done with something like

python3 -c "import sysconfig; print(sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET'))"

if we get back a value of 11 and yet the version of gfortran we are using is less than 10.2.0 then we change MACOSX_DEPLOYMENT_TARGET to 10.16 or something like that.

zlscherr · 2021-03-16T02:18:27Z

comment:17

But probably the second step is not necessary, it would only be needed if the user has python3 from Homebrew and yet gfortran from sage. I think just fixing up scipy's spkg-install should fix everything.

zlscherr · 2021-03-16T02:31:07Z

Branch: u/gh-zlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET

sagetrac-git · 2021-03-16T02:32:30Z

Commit: 4cc8c4d

sagetrac-git · 2021-03-16T02:32:30Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4cc8c4d`	`Reverted changes in scipy's spkg-install to unset MACOSX_DEPLOYMENT_TARGET to 11 on Big Sur`

mkoeppe · 2021-03-16T02:33:47Z

comment:21

Thanks for investigating!

To test this in various configurations, it would be enough to merge the branch of #31492 and then push a tag to your github fork of sage.

zlscherr · 2021-03-16T02:45:06Z

comment:22

I think I did that correctly. I'll report on what happens in the CI on my repo.

zlscherr · 2021-03-16T03:23:27Z

comment:23

I'm not sure if there is something I need to do to get the CI working, but some of the homebrew's are already failing because of an error like:

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to homebrew.bintray.com:443

Is there something I need to do to avoid this?

mkoeppe · 2021-03-16T03:34:23Z

comment:24

There are various ways how these CI runs can intermittently fail. It's nothing that we can fix on our side as far as I can tell.

By the way, if you want, you can remove the unnecessary runs (for linux, cygwin) by editing files in .github/workflows/. Also max-parallel can be edited so that more macOS jobs are run in parallel.

mkoeppe · 2021-03-16T07:01:28Z

comment:25

Replying to @mkoeppe:

local-macos (conda-forge-macos, standard, default, macos-11.0) https://github.com/mkoeppe/sage/runs/2103782333 - uses python3 from conda

@isuruf: Also this build using conda-forge is affected

dimpase · 2021-03-20T20:59:50Z

comment:27

yes, my student also reports getting

[scipy-1.5.4]     clang: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=11.0'

zlscherr · 2021-03-20T21:22:12Z

comment:28

I tried running on github CI, but it quit after a few days and only got through 10.15. I just took out 10.15 and am trying again...

zlscherr · 2021-03-21T19:31:27Z

comment:29

I tried to run the CI tests on 11.0 but I got the following error message:

"No runner matching the specified labels was found: macos-11.0"

Is there a different way I should get the tests to run on Big Sur?

mkoeppe · 2021-03-21T19:45:33Z

comment:30

actions/runner-images#2486
"macOS 11.0 pools will be transited to private preview"

zlscherr · 2021-03-21T19:47:05Z

comment:31

gotcha. I've tried some local testing and the reversion seems to be fine, but I'll leave it to others to do more extensive testing.

mkoeppe · 2021-03-21T19:48:05Z

comment:32

Some of my recent runs in https://github.com/mkoeppe/sage/actions/workflows/tox.yml have used branches that merged this ticket and may contain some useful results for 11.0

mkoeppe · 2021-03-21T21:43:44Z

Reviewer: Matthias Koeppe

mkoeppe · 2021-03-21T21:43:44Z

comment:33

local-macos (homebrew-macos, minimal, default, macos-11.0) -
https://github.com/mkoeppe/sage/runs/2156548349 - OK
local-macos (homebrew-macos, standard, default, macos-11.0) - https://github.com/mkoeppe/sage/runs/2156548388 - OK

mkoeppe · 2021-03-21T21:43:44Z

Author: Zachary Scherr

mkoeppe · 2021-03-21T21:47:19Z

comment:34

Also

local-macos (conda-forge-macos, standard, default, macos-11.0) - https://github.com/mkoeppe/sage/runs/2135024636 - OK (except for an unrelated error in docbuild)

jhpalmieri · 2021-03-22T02:57:34Z

comment:35

I agree it works with various Pythons on OS X 11. comment:14 has not been addressed, but I guess that's not for this ticket.

mkoeppe · 2021-03-22T03:08:33Z

comment:36

Replying to @jhpalmieri:

comment:14 has not been addressed, but I guess that's not for this ticket.

Yes, to fix that we need to upgrade our gcc/gfortran - that's #29703.

vbraun · 2021-03-23T23:49:36Z

Changed branch from u/gh-zlscherr/revert_scipy_MACOSX_DEPLOYMENT_TARGET to 4cc8c4d

mkoeppe added this to the sage-9.3 milestone Feb 2, 2021

mkoeppe added c: porting labels Feb 2, 2021

This comment has been minimized.

Sign in to view

mkoeppe added p: blocker / 1 and removed p: critical / 2 labels Mar 14, 2021

zlscherr added the s: needs review label Mar 16, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~homebrew-macos-11.0-xcode-minimal: scipy build fails~~ macos-11.0: scipy build fails Mar 16, 2021

mkoeppe added s: positive review and removed s: needs review labels Mar 21, 2021

vbraun removed the s: positive review label Mar 23, 2021

vbraun closed this as completed in 5e86bf2 Mar 23, 2021

bukzor mannequin mentioned this issue May 27, 2021

unset MACOSX_DEPLOYMENT_TARGET, rather than sometimes setting a bogus value #18272

Closed

slel mentioned this issue Jun 24, 2021

Meta-ticket: support macOS 11 Big Sur #30651

Closed

macos-11.0: scipy build fails #31326

macos-11.0: scipy build fails #31326

Comments

mkoeppe commented Feb 2, 2021

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Mar 15, 2021

zlscherr commented Mar 15, 2021

zlscherr commented Mar 15, 2021

zlscherr commented Mar 15, 2021

zlscherr commented Mar 15, 2021

mkoeppe commented Mar 15, 2021

mkoeppe commented Mar 15, 2021

zlscherr commented Mar 15, 2021

mkoeppe commented Mar 15, 2021

zlscherr commented Mar 15, 2021

mkoeppe commented Mar 15, 2021

jhpalmieri commented Mar 15, 2021

mkoeppe commented Mar 15, 2021

zlscherr commented Mar 16, 2021

zlscherr commented Mar 16, 2021

zlscherr commented Mar 16, 2021

sagetrac-git mannequin commented Mar 16, 2021

sagetrac-git mannequin commented Mar 16, 2021

mkoeppe commented Mar 16, 2021

zlscherr commented Mar 16, 2021

zlscherr commented Mar 16, 2021

mkoeppe commented Mar 16, 2021

mkoeppe commented Mar 16, 2021

This comment has been minimized.

dimpase commented Mar 20, 2021

zlscherr commented Mar 20, 2021

zlscherr commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

zlscherr commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

mkoeppe commented Mar 21, 2021

jhpalmieri commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

vbraun commented Mar 23, 2021