OS X Mojave without /usr/include: a range of issues

[jhpalmieri]

On OS X Mojave, it is possible to have Xcode installed but to have no directory /usr/include.

• The configure check from more system package checks #26286 finds zlib somewhere, but the Python build does not: Python reports that it has built without zlib support. Without zlib, pip will not build.

• gcc/gfortran needs the location of headers to be explicitly passed via --with-native-system-header-dir= option. (untested)

Upstream: Reported upstream. No feedback yet.

CC: @embray

Component: packages: standard

Author: Dima Pasechnik, Volker Braun

Branch: db50d09

Reviewer: Dima Pasechnik, Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/26899

[dimpase]

comment:1

See #26713. I'd close this one, but yes, what's offered on #26713 is only a temporary solution. We need to think what to do with OSX (Homebrew? (Ana)conda?), it's getting too hard to jump through new and new hoops built up by Apple.

[jhpalmieri]

comment:2

On this machine, Sage 8.5.beta4 builds. I am now testing. So it's probably not the same issue as #26713 which was opened before 8.5.rc0 was released. Or it's not necessarily the same issue. In particular, I believe that the problem on this machine can be traced directly to #26286.

I believe that on this machine, I once ran

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg 

as advised at #26713, and then I believe a subsequent Xcode upgrade removed /usr/include. That upgrade apparently left the files necessary to build Sage.

[jhpalmieri]

comment:3

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

[jhpalmieri]

comment:4

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

[dimpase]

comment:5

Replying to @jhpalmieri:

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

Proper, i.e. only known to Xcode? I don't know. It's not really a zlib problem, it's an Apple bug, if you ask me. They really, really want you to do everything via Xcode.

[dimpase]

comment:6

Replying to @jhpalmieri:

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

./sage -i zlib

will build zlib from scratch and install it in SAGE_LOCAL, as always.

[dimpase]

comment:7

Once again, on the machine with this problem, do you have /usr/include at all?

I guess it might be that you need to run that open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg etc. after you upgraded XCode, as the latter could have left everything in a slightly out of sync state.

If this does not help then we need to find out whether it's that you actually have non-functioning zlib installation before you start building python, or it's a python issue
(I suppose you can look at the logs of the python build and see why zlib extension is not built).

[jhpalmieri]

comment:8

Once again, no I do not have /usr/include on this machine, and once again, Sage 8.5.beta6 builds on it just fine. The problem is only zlib.

I can't tell whether it's something out of sync with this machine, as you suggest, or whether it could routinely happen to other people. If we think it could affect other machines, which is the more conservative point of view, then we should roll back the zlib part of #26286 to fix it. Assuming it's just my machine is riskier and could mean that others will not be able to build Sage.

[dimpase]

comment:9

Rather than blindly rolling back, we'd like to know the cause of the problem, no?
This is why I asked you for the log...

[vbraun]

comment:10

I don't think we are going to be able to do anything in the short run, for now you just have to install macOS_SDK_headers_for_macOS_10.14

[jhpalmieri]

comment:11

From the main config.log:

configure:8396: === checking whether to install the zlib SPKG ===
configure:8410: checking for inflateValidate in -lz
configure:8435: g++ -o conftest -g -O2 -std=gnu++11    conftest.cpp -lz  -lm  >&5
configure:8435: $? = 0
configure:8444: result: yes
configure:8451: checking if zlib is wanted
configure:8473: result: yes
configure:8503: checking for inflateEnd in -lz
configure:8528: gcc -o conftest -g -O2    conftest.c -lz  -lm  >&5
configure:8528: $? = 0
configure:8537: result: yes
configure:8545: checking zlib.h usability
configure:8545: gcc -c -g -O2   conftest.c >&5
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking zlib.h presence
configure:8545: gcc -E   conftest.c
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking for zlib.h
configure:8545: result: yes

I'm rebuilding now (I deleted the old build to test whether removing build/pkgs/zlib/spkg-configure.m4 would work, and it did). I'll post the Python log when it's available, although I didn't see anything helpful there before, just a comment that the zlib component was not built. I think that Python doesn't know all of the places to look for zlib.h.

[jhpalmieri]

comment:12

Volker: a short term fix that doesn't hurt anything is just to remove build/pkgs/zlib/spkg-configure.m4. That is completely safe, so I think we could easily do it for rc1 (if it's decided that it's the right approach).

[jhpalmieri]

comment:13

Attachment: python2-2.7.15.p0.log

Here is the Python log. I don't see anything helpful in there. By the way, why is Python built twice? Is this a recent thing?

[dimpase]

comment:14

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3?
(and if not, how is it failing)
There is a sea of difference between Python2 and 3 in regard of how configuration is done, so it could be a bug of sorts in Python2.

Another question is where that zlib.h is found by spkg-configure.m4
Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

[dimpase]

comment:15

In Python2/3 configure.ac, one sees that OSX is treated in a special way:

dnl Check if system zlib has *Copy() functions
dnl
dnl On MacOSX the linker will search for dylibs on the entire linker path
dnl before searching for static libraries. setup.py adds -Wl,-search_paths_firs
t
dnl to revert to a more traditional unix behaviour and make it possible to
dnl override the system libz with a local static library of libz. Temporarily
dnl add that flag to our CFLAGS as well to ensure that we check the version
dnl of libz that will be used by setup.py.
dnl The -L/usr/local/lib is needed as wel to get the same compilation
dnl environment as setup.py (and leaving it out can cause configure to use the
dnl wrong version of the library)
case $ac_sys_system/$ac_sys_release in
Darwin/*)
        _CUR_CFLAGS="${CFLAGS}"
        _CUR_LDFLAGS="${LDFLAGS}"
        CFLAGS="${CFLAGS} -Wl,-search_paths_first"
        LDFLAGS="${LDFLAGS} -Wl,-search_paths_first -L/usr/local/lib"
        ;;
esac

AC_CHECK_LIB(z, inflateCopy, AC_DEFINE(HAVE_ZLIB_COPY, 1, [Define if the zlib library has inflateCopy]))

case $ac_sys_system/$ac_sys_release in
Darwin/*)
        CFLAGS="${_CUR_CFLAGS}"
        LDFLAGS="${_CUR_LDFLAGS}"
        ;;
esac

One may wonder whether it could pick up wrong old zlib from /usr/local, say, and then fail on testing it being recent enough).

By right, Python should have a configure option to tell it where zlib is, and do not fool around the system looking for one...

[vbraun]

comment:16

Python should be built only once. Our logging concatenates output from different builds.

[vbraun]

comment:17

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent, and probably only search zlib in /usr. Right now there is

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

[jhpalmieri]

comment:18

Replying to @dimpase:

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3?
(and if not, how is it failing)

Same for Python 3. As you note in your later comment, zlib is treated differently on OS X in configure.ac and in Modules/Setup.dist.

Another question is where that zlib.h is found by spkg-configure.m4
Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

$ locate zlib.h prints the following (along with some matches for files containing "zlib.h", like "nazlib.htf", which I have omitted):

/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/zlib.h

Perhaps the relevant ones are

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h

In any case, there is nothing relevant in /usr/local. I also don't have /sw or /opt/local/.

[jhpalmieri]

comment:19

Replying to @vbraun:

Python should be built only once. Our logging concatenates output from different builds.

I did actually know that second part, which is what led me to ask the question in the first place.

If I run make distclean; make, I see this in the Python 2 and 3 log files:

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
readline module imported OK
socket module imported OK
_scproxy module imported OK

real	2m28.119s
user	1m17.674s
sys	0m21.064s
Installing python2-2.7.15.p0
make[4]: warning: -jN forced in submake: disabling jobserver mode.

The "real[TAB][time]" string is typically only printed when the installation is complete. Oh, I see, it's running spkg-build and spkg-install and reporting timings for each. That's a little confusing. The first one should say "Successfully ran spkg-build" at the end.

[dimpase]

comment:20

Replying to @vbraun:

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent,

The requirements are needed to make libpg happy, the version we use needs these checks to pass.

and probably only search zlib in /usr.

Homebrew and freebsd have most things in /usr/local.

What is needed for conda?

Right now there is

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

[dimpase]

comment:21

My current theory is that our macro to recognise/test zlib resorts to calling C complier/linker without extra flags, and everything works, whereas Python's one does something less clever, and fails.

That is we can patch Python's configure.ac and get everything working.

[vbraun]

comment:22

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

PS: Conda is not installed at any fixed path, users can place it whereever they want in their home directory.

[dimpase]

comment:23

Replying to @vbraun:

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

Assuming that mixing system's zlib and Sage's zlib is not a good idea, note that
zlib is a dependence (1st or 2nd order) of several other libraries: boost, cbc, glpk, libpng, brial,
freetype, giac, libgd, matplotlib, python2/3, etc

that means not using system's zlib forces building of a lot of other Sage libs...

[vbraun]

comment:24

Since its a common dependency its even more important that we err on the safe side

IMHO either

1. the user has libraries installed in the standard location /usr, or
2. the user explicitly configures the libray location, or
3. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

[dimpase]

comment:25

Replying to @vbraun:

Since its a common dependency its even more important that we err on the safe side

IMHO either

1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

2. the user explicitly configures the libray location, or

3. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

[vbraun]

comment:26

Replying to @dimpase:

1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

Its pretty much on every unix-like OS, including OSX. The only thing that OSX lacks here is a sane way to distribute software that is not an self-contained opaque application bundle.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

And the sane way is to let the users explicitly pick their dependencies, not rummage around and see if we can randomly find something that isn't part of the OS/distribution. It should just be a matter of ./configure CPPFLAGS=-I/the/location/include LDFLAGS=-L/the/location/lib

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

Surely there are many more homebrew developers than sage developers. Its also going to be much easier to maintain a homebrew package when there is a simple way to configure the sage dependencies, than to fix all possible Sage build pitfalls with whatever broken bits and pieces we may find in the filesystem.

[vbraun]

comment:78

PS: the osx buildbot worker had his /usr/include wiped in the last osx update

[dimpase]

comment:79

I'm seting up a remote paid-for (from the grant, hopefully) Mac mini with 10.14, hopefully I'd sort these issues out.

[jhpalmieri]

comment:80

Replying to @dimpase:

Replying to @vbraun:

gfortran still dies with

The directory that should contain system headers does not exist:
  /usr/include

gcc has --with-native-system-header-dir= option (not shown in the output of ./configure -h, but present in docs and in configure itself. One can use already mentioned.
xcrun --show-sdk-path call to compute the value to be passed.

Using GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE" worked for me. (I didn't see this earlier because I've installed my own gfortran on each of my OS X machines so I don't have to wait for Sage to build it. But when I hid my copy, I get the same error.)

[dimpase]

comment:81

Replying to @jhpalmieri:

Using GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE" worked for me. (I didn't see this earlier because I've installed my own gfortran on each of my OS X machines so I don't have to wait for Sage to build it. But when I hid my copy, I get the same error.)

yes, this works (checking on a very bare MacOS 10.14, with nothing except Command Line Tools for Xcode 10.2 installed), thanks:

--- a/build/pkgs/gcc/build-gcc
+++ b/build/pkgs/gcc/build-gcc
@@ -25,6 +25,7 @@ if [ "$UNAME" = "Darwin" ]; then
     # files prior to comparison during bootstrap (broken by Xcode 6.3).
     # See #18156
     GCC_CONFIGURE="--with-build-config=bootstrap-debug $GCC_CONFIGURE"
+    GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE"
 fi
 
 # Let Gfortran build on Raspberry Pi using hard floats.

I suppose it should be done conditionally, only if there is no /usr/include present.

[vbraun]

comment:82

I've added that to the branch, now gfortran builds...

[sagetrac-git]

Changed commit from 3d49fc2 to 758e699

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

758e699

Add Xcode sdk path

[vbraun]

comment:84

Passes so I'd suggest we merge this to get the OSX buildbot up and running again. Feel free to reintroduce the zlib configure in a later ticket, then we'll be able to test it against a buildbot without the extra headers installed.

[dimpase]

comment:85

If you are really in rush we can add a condition disabling spkg-configure.m4 for zlib in the case there is no /usr/include present.
Why is this a problem, it's one if block, right?

I hopefully soon will have a Python fix ready that would make this not necessary.

[dimpase]

Reviewer: Dima Pasechnik

[dimpase]

comment:86

Instead of removing spkg-configure.m4, apply

--- a/build/pkgs/zlib/spkg-configure.m4
+++ b/build/pkgs/zlib/spkg-configure.m4
@@ -5,4 +5,5 @@ SAGE_SPKG_CONFIGURE([zlib], [
         AX_CHECK_ZLIB([], [zlib_cv_libz=no])
     ])
     AS_IF([test "x$zlib_cv_libz" != "xyes"], [sage_spkg_install_zlib=yes])
+    AC_CHECK_FILE([/usr/include/],[],[sage_spkg_install_zlib=yes])
 ])

this disables the usage of zlib from the system if /usr/include/ is absent.
I just tested this on MacOS.
With this change, please feel free to give it positive review.

[sagetrac-git]

Changed commit from 758e699 to db50d09

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

25761b7	Add Xcode sdk path
db50d09	Disable the usage of zlib from the system if /usr/include/ is absent

[vbraun]

comment:88

You can push commits instead of pasting diffs to the chat, you know ;-)

New commits:

25761b7	Add Xcode sdk path
db50d09	Disable the usage of zlib from the system if /usr/include/ is absent

[vbraun]

Changed reviewer from Dima Pasechnik to Dima Pasechnik, Volker Braun

[vbraun]

Author: Dima Pasechnik, Volker Braun

[vbraun]

Changed branch from u/vbraun/os_x_mojave_without__usr_include__python_builds_without_zlib_support to db50d09

[dimpase]

Changed commit from db50d09 to none

[dimpase]

comment:91

Replying to @dimpase:

If you are really in rush we can add a condition disabling spkg-configure.m4 for zlib in the case there is no /usr/include present.
Why is this a problem, it's one if block, right?

I hopefully soon will have a Python fix ready that would make this not necessary.

Patching Python 3 to do this is easy,
Erik's suggestion on https://bugs.python.org/issue36231 really makes it
a 1-line addition to Python's configure.ac and 2 more lines to setup.py,
but backporting this to Python 2 is not straightforward, as it has more brain-dead setup.py,
which seems to requires manual adjustments to a bunch of module configurations.

[embray]

comment:92

IMO this is kinda weak, albeit harmless for the time being (though the line added to spkg-configure.m4 should at least have had a comment, but at least we can git blame it easily...). I think Python should be patched instead.

[dimpase]

comment:93

Replying to @embray:

IMO this is kinda weak, albeit harmless for the time being (though the line added to spkg-configure.m4 should at least have had a comment, but at least we can git blame it easily...). I think Python should be patched instead.

to be done on #27631 - almost there...