Fix build errors with pplpy, scipy, kiwisolver, fpylll on macos with system python3

[mkoeppe]

On the platform local-homebrew-macos defined by tox.ini, running on a macos-latest system as defined in https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources, a branch that includes #27824 (spkg-configure.m4 for python3) leads to the following error with pplpy, as seen in https://github.com/mkoeppe/sage/runs/538432631:

    g++ -sdk macosx clang -bundle -undefined dynamic_lookup -L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/local/lib -Wl,-rpath,/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/local/lib -L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/homebrew/opt/readline/lib -L/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/homebrew/lib -I/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/homebrew/opt/readline/include -I/Users/runner/runners/2.165.2/work/sage/sage/.tox/local-homebrew-macos-standard/homebrew/include build/temp.macosx-10.14-x86_64-3.7/ppl/linear_algebra.o build/temp.macosx-10.14-x86_64-3.7/ppl/ppl_shim.o -lgmp -lgmpxx -lppl -lm -o build/lib.macosx-10.14-x86_64-3.7/ppl/linear_algebra.cpython-37m-darwin.so
    clang: error: unknown argument: '-sdk'
    clang: error: no such file or directory: 'macosx'
    clang: error: no such file or directory: 'clang'
    error: command 'g++' failed with exit status 1
    Running setup.py install for pplpy: finished with status 'error'
Cleaning up...

Same kind of error with scipy, kiwisolver, fpylll.

Full logs can be downloaded at the above link.

To test locally, using #29104 + #27824, remove all python3 from the homebrew in /usr/local
and run

tox -e local-homebrew-macos-standard -- V=0 pplpy

With #29417, it is:

tox -e local-homebrew-macos-standard-python3_xcode -- V=0 pplpy

(broken out from #29404)

CC: @dimpase @videlec @kiwifb @mwageringel @embray @jhpalmieri @vbraun

Component: packages: standard

Author: Matthias Koeppe

Branch/Commit: 8317c3e

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29408

[dimpase]

comment:1

this needs details on the Python (system MacOS? Homebrew? CPython package?) and on clang (XCode? Homebrew? Other (e.g. conda?))

[dimpase]

comment:3

what I gather from these links is that Pythons in MacOS are not Homebrew ones.

[mkoeppe]

comment:5

Replying to @dimpase:

what I gather from these links is that Pythons in MacOS are not Homebrew ones.

That's right, this is system python3 from /usr/bin

[mkoeppe]

comment:7

Something seems to get very confused about values from sysconfig as it tries to replace xcrun by g++.

>>> for k, v in sysconfig.get_config_vars().items():
...     if isinstance(v, str) and 'sdk' in v:
...         print(k, v)
... 
BLDSHARED xcrun -sdk macosx clang     -bundle -undefined dynamic_lookup 
CC xcrun -sdk macosx clang    -arch x86_64 
CONFIG_ARGS '-C' '--host=x86_64-apple-darwin' '--build=x86_64-apple-darwin' '--enable-framework=/Applications/Xcode.app/Contents/Developer/Library/Frameworks' '--with-framework-name=Python3' '--with-openssl=/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Root/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7' '--enable-ipv6' '--prefix=/Applications/Xcode.app/Contents/Developer/usr' '--with-pymalloc' 'PYTHON_FOR_BUILD=DYLD_FRAMEWORK_PATH=/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects/host /BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects/host/python.exe' 'CC=xcrun -sdk macosx clang    -arch x86_64' 'CXX=xcrun -sdk macosx clang++  -arch x86_64' 'CPP=xcrun -sdk macosx clang -E -arch x86_64' 'CFLAGS=-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/Headers' 'CPPFLAGS=-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/Headers' 'LIBS=-lSystem' 'LDFLAGS=' 'OBJROOT=/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects' 'SDKROOT=/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.Internal.sdk' 'build_alias=x86_64-apple-darwin' 'host_alias=x86_64-apple-darwin'
CXX xcrun -sdk macosx clang++  -arch x86_64 
LDCXXSHARED xcrun -sdk macosx clang++  -arch x86_64 -bundle -undefined dynamic_lookup
LDSHARED xcrun -sdk macosx clang     -bundle -undefined dynamic_lookup 
LINKCC xcrun -sdk macosx clang    -arch x86_64
MAINCC xcrun -sdk macosx clang    -arch x86_64
SDKROOT /BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.Internal.sdk
_OSX_SUPPORT_INITIAL_BLDSHARED xcrun -sdk macosx clang    -arch x86_64 -bundle -undefined dynamic_lookup
_OSX_SUPPORT_INITIAL_LDSHARED xcrun -sdk macosx clang    -arch x86_64 -bundle -undefined dynamic_lookup
_OSX_SUPPORT_INITIAL_CC xcrun -sdk macosx clang    -arch x86_64
_OSX_SUPPORT_INITIAL_CXX xcrun -sdk macosx clang++  -arch x86_64

[mkoeppe]

comment:8

It is caused by bizarre code in distutils.unixccompiler.UnixCCompiler.link.

[mkoeppe]

comment:9

https://github.com/python/cpython/blob/master/Lib/distutils/unixccompiler.py#L180
with

(Pdb) p self.linker_so
['xcrun', '-sdk', 'macosx', 'clang', '-bundle', '-undefined', 'dynamic_lookup', '-L/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib', '-Wl,-rpath,/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib', '-L/usr/local/opt/readline/lib', '-L/usr/local/lib', '-I/usr/local/opt/readline/include', '-I/usr/local/include']
(Pdb) p self.compiler_cxx
['g++', '-std=gnu++11']

[kiwifb]

comment:10

Yes, my head hurts when I look at that file. I am fairly sure distros all over have patches for some aspects of it.

From the line it expect some behavior from xcode.

[kiwifb]

comment:11

Hum

fbissey@itsv-frb15 ~ % xcrun --help
Usage: xcrun [options] <tool name> ... arguments ...

Find and execute the named command line tool from the active developer
directory.

The active developer directory can be set using `xcode-select`, or via the
DEVELOPER_DIR environment variable. See the xcrun and xcode-select manual
pages for more information.

Options:
  -h, --help                  show this help message and exit
  --version                   show the xcrun version
  -v, --verbose               show verbose logging output
  --sdk <sdk name>            find the tool for the given SDK name
  --toolchain <name>          find the tool for the given toolchain
  -l, --log                   show commands to be executed (with --run)
  -f, --find                  only find and print the tool path
  -r, --run                   find and execute the tool (the default behavior)
  -n, --no-cache              do not use the lookup cache
  -k, --kill-cache            invalidate all existing cache entries
  --show-sdk-path             show selected SDK install path
  --show-sdk-version          show selected SDK version
  --show-sdk-build-version    show selected SDK build version
  --show-sdk-platform-path    show selected SDK platform path
  --show-sdk-platform-version show selected SDK platform version

Is it just a stupid -- versus -?

[mkoeppe]

comment:12

No, the linked code is really replacing "xcrun" by "g++".

[mkoeppe]

comment:13

More context, more fun at https://github.com/python/cpython/blob/master/Lib/_osx_support.py

[mkoeppe]

comment:14

Each of xcrun -sdk macosx clang, g++, and xcrun -sdk macosx clang++ make sense.
But the code tries to mix the first two to get the third, and gets g++ -sdk macosx clang, which makes no sense, obviously.

[mkoeppe]

comment:15

Google search for LDCXXSHARED distutils brings up various things

https://www.google.com/search?safe=off&sxsrf=ALeKk01RyL6O3kSKx0-W2ZRb3O8N8Pdg-A:1585342860000&q=LDCXXSHARED+distutils&nfpr=1&sa=X&ved=2ahUKEwiSrZGwxrvoAhWkl3IEHZtFCmEQvgUoAXoECAwQKA&biw=875&bih=862

[mkoeppe]

comment:16

It's an existing Python issue: https://bugs.python.org/issue8027

[mkoeppe]

comment:18

More precisely, this one: https://bugs.python.org/issue1222585

[mkoeppe]

comment:19

... open since 2005

[mkoeppe]

comment:20

Related previous issue: #17484

[kiwifb]

comment:21

Replying to @mkoeppe:

... open since 2005

Upstream python does have a bad history of compiler/linker support (especially differentiation between C and C++ actually).

[mkoeppe]

comment:22

A simple workaround is to clear the CXX environment variable before we call the installation.
Then it will build the extension using the compiler configured in python's sysconfig.

[kiwifb]

comment:23

Hum, OK but only for OSX. The problem is that we set CXX as a c++11 compiler which means CXX may actually be g++ -std=c++11 (or gnu++11 actually) in most case. On OS X, clang++ is c++11 at least by default so we can afford that. On other platforms, we would run into troubles.

Why does CXX includes -std=c++11 rather than it being a separate CPPFLAGS? Because CXX as a whole is defined as a c++11 compiler.

[mkoeppe]

comment:24

Our own python3 has g++ -std=c++11 in its sysconfig CXX in that case.

But you are right, for a system python3, we would not be able to assume that the sysconfig CXX is a C++11 compiler.

[mkoeppe]

comment:25

So I guess this needs to be figured out at configure time.

[mkoeppe]

comment:26

A different workaround is to set LDSHARED="$CXX -bundle -undefined dynamic_lookup".

[mkoeppe]

Changed branch from u/isuruf/fix_build_errors_with_pplpy__scipy__kiwisolver__fpylll_on_macos_with_system_python3 to u/mkoeppe/fix_build_errors_with_pplpy__scipy__kiwisolver__fpylll_on_macos_with_system_python3

[mkoeppe]

comment:62

rebased on 9.1.beta9

---

New commits:

7a6238e	src/bin/sage-env: Set LDSHARED when CXX is set
cbe7a1e	build/pkgs/numpy/spkg-install.in: Unset LDSHARED
089d2a2	Add -pthread to LDSHARED
96a599c	Set LDSHARED only on OSX

[mkoeppe]

Changed commit from e4056f0 to 96a599c

[mkoeppe]

comment:63

On local-homebrew-macos-python3_pythonorg-minimal(https://github.com/mkoeppe/sage/runs/557674691):

sage -t src/sage/repl/ipython_extension.py
**********************************************************************
File "src/sage/repl/ipython_extension.py", line 385, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
...
RuntimeError: failed to compile Fortran code:
    b'Could not locate executable g++ -std=gnu++11 -pthread -bundle -undefined dynamic_lookup\nIn file included from 

[mkoeppe]

comment:65

Also

sage -t src/sage/interfaces/gap.py  # 1 doctest failed
sage -t src/sage/misc/inline_fortran.py  # 3 doctests failed
sage -t src/sage/misc/persist.pyx  # 2 doctests failed

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f721740	build/pkgs/python3/spkg-configure.m4: Also check whether C++ extensions can be built
8317c3e	Undo #21175: Set ARCHFLAGS only when compiling Perl modules

[sagetrac-git]

Changed commit from 96a599c to 8317c3e

[mkoeppe]

comment:67

Started again from scratch. Turns out I broke it 4 years ago in #21175 by setting ARCHFLAGS. This interacts in a bizarre way with the bizarre code discussed above.

[mkoeppe]

Changed author from Matthias Koeppe, Isuru Fernando to Matthias Koeppe

[mkoeppe]

comment:69

Tests run in https://github.com/mkoeppe/sage/actions/runs/73955490

[jhpalmieri]

comment:70

This works for me, but I don't understand the changes in build/pkgs/python3/spkg-configure.m4.

[jhpalmieri]

comment:71

So if someone else wants to review it, I am happy to give some fraction of a positive review.

[mkoeppe]

comment:72

Replying to @jhpalmieri:

I don't understand the changes in build/pkgs/python3/spkg-configure.m4.

There was already a test that the system python (in the venv) is able to build a C extension.
The change tightens this check by imposing the configured CC (and CXX) values -- because this is what the actual build will be running.
It also adds the same test for building a C++ 11 extension.

[mkoeppe]

comment:73

Replying to @jhpalmieri:

some fraction of a positive review.

100% thanks already

[mkoeppe]

comment:74

Let's also get this one into the next beta please

[jhpalmieri]

comment:77

Okay, let's merge it. I can't get polymake to build with or without these changes, and the changes make sense.

Re polymake, here is the error:

ninja: Entering directory `build/Opt'
[1/1] GENERATE /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/build/targets.ninja
FAILED: /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/build/targets.ninja 
/usr/bin/perl /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/support/generate_ninja_targets.pl /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/build/targets.ninja /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/build/config.ninja
Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/lib/perl5/darwin-thread-multi-2level /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/lib/perl5 /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.4 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/support/generate_cpperl_modules.pl line 22.
BEGIN failed--compilation aborted at /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-9.1.beta9/local/var/tmp/sage/build/polymake-3.4/src/support/generate_cpperl_modules.pl line 22.
ninja: error: rebuilding 'build.ninja': subcommand failed
make[2]: *** [all] Error 1

It's probably the old problem of not having the right perl modules present and Sage's polymake build system failing to recognize that. The instructions in polymake's SPKG.txt are no longer valid.

In any case, the polymake problems should be dealt with at #29054.

[jhpalmieri]

Reviewer: John Palmieri

[mkoeppe]

comment:78

Thanks. Yes, polymake will take more work.

[vbraun]

Changed branch from u/mkoeppe/fix_build_errors_with_pplpy__scipy__kiwisolver__fpylll_on_macos_with_system_python3 to 8317c3e