Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix build failures of pynac on macos with system python3 #29404

Closed
mkoeppe opened this issue Mar 25, 2020 · 79 comments
Closed

Fix build failures of pynac on macos with system python3 #29404

mkoeppe opened this issue Mar 25, 2020 · 79 comments
Labels
c: packages: standard p: blocker / 1 t: bug
Milestone
sage-9.1

Comments

@mkoeppe
Copy link
Contributor

mkoeppe commented Mar 25, 2020

https://github.com/mkoeppe/sage/runs/531095486

pynac:

checking whether sage-python23 version is >= 2.7... yes
checking for sage-python23 version... 3.7
checking for sage-python23 platform... darwin
checking for sage-python23 script directory... ${prefix}/lib/python3.7/site-packages
checking for sage-python23 extension module directory... ${exec_prefix}/lib/python3.7/site-packages
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking for python3.7... no
configure: error: Cannot find python3.7 in your system path

Fix in pynac/pynac#351, https://github.com/mkoeppe/pynac/releases/tag/pynac-0.7.26.sage-2020-03-28

The "temporary upstream" tarball URL is in checksums.ini. To test the update on this branch: Use make SAGE_SPKG="sage-spkg -o" pynac-clean pynac; this will download the new upstream package automatically.

Upstream: Completely fixed; Fix reported upstream

CC: @dimpase @kiwifb @videlec @saraedum @isuruf @embray

Component: packages: standard

Author: Matthias Koeppe

Branch: 7344e2d

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29404
@mkoeppe mkoeppe added this to the sage-9.1 milestone Mar 25, 2020
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2020

comment:1

pynac's python discovery should be fixed, this looks completely broken

@mkoeppe mkoeppe changed the title Fix build failures of pplpy on macos with system python3 Fix build failures of pplpy and pynac on macos with system python3 Mar 25, 2020
@dimpase
Copy link
Member

dimpase commented Mar 25, 2020

comment:3

This should be probably split into two tickets.

pplpy upstream is https://gitlab.com/videlec/pplpy - also in CC:

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:4

Yes for the split. I am surprised that problem shows now in pynac. I would have expected it to happen much earlier. The AX_PYTHON_DEVEL macro is 5 years old - which probably doesn't help.

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:5

Note that I don't think the problem is in the macro, but something in there may trigger. I suspect the problem is in AM_PATH_PYTHON and that PYTHON needs to be defined when calling configure in spkg-install.

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:6

Which is already done. So I am not sure what happens to override it. Access to config.log would be helpful.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2020

comment:7

Yes, please go ahead with splitting the ticket.

@dimpase
Copy link
Member

dimpase commented Mar 25, 2020

comment:8

It looks from the log as here could be some mixing up happening between system python3 and Homebrew's python3 (unless Homebrew's python3 is using system's python3 headers by design).

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:9

Replying to @dimpase:

It looks from the log as here could be some mixing up happening between system python3 and Homebrew's python3 (unless Homebrew's python3 is using system's python3 headers by design).

Shouldn't happen. We pass PYTHON=sage-python23 explicitly to configure. That's what should be tested.
https://github.com/sagemath/sage-prod/blob/develop/build/pkgs/pynac/spkg-install.in#L14

My gut feeling is that something funny happened to that line during processing. But I need config.log to establish it.

@dimpase
Copy link
Member

dimpase commented Mar 25, 2020

comment:10

I don't have MacOS machine, but anyone with it may install Homebrew and try.

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:11

It is more subtle than my implied gut feeling. The macro AX_PYTHON_DEVEL gets the right value for PYTHON and does its stuff based on that. I think AM_PATH_PYTHON doesn't like python with exotic names because it looks like it goes through a pre-defined list.

@dimpase
Copy link
Member

dimpase commented Mar 25, 2020

comment:12

isn't the location of Sage's Python exotic enough to choke it, if it were the case?

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:13

It would be :)

Code inspection says I have it backwards. AM_PATH_PYTHON works and AX_PYTHON_DEVEL is the one failing.

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:14

Toxic. This is with an old sage I have around

[pynac-0.7.14] checking whether sage-python23 version is >= 2.7... yes
[pynac-0.7.14] checking for sage-python23 version... 2.7
[pynac-0.7.14] checking for sage-python23 platform... linux2
[pynac-0.7.14] checking for sage-python23 script directory... ${prefix}/lib/python2.7/site-packages
[pynac-0.7.14] checking for sage-python23 extension module directory... ${exec_prefix}/lib/python2.7/site-packages
[pynac-0.7.14] checking for style of include used by make... GNU
[pynac-0.7.14] checking for gcc... gcc
[pynac-0.7.14] checking whether the C compiler works... yes
[pynac-0.7.14] checking for C compiler default output file name... a.out
[pynac-0.7.14] checking for suffix of executables... 
[pynac-0.7.14] checking whether we are cross compiling... no
[pynac-0.7.14] checking for suffix of object files... o
[pynac-0.7.14] checking whether we are using the GNU C compiler... yes
[pynac-0.7.14] checking whether gcc accepts -g... yes
[pynac-0.7.14] checking for gcc option to accept ISO C89... none needed
[pynac-0.7.14] checking whether gcc understands -c and -o together... yes
[pynac-0.7.14] checking dependency style of gcc... gcc3
[pynac-0.7.14] checking for python2.7... /home/fbissey/sagemath/sage/local/bin/python2.7
[pynac-0.7.14] checking for a version of Python >= '2.1.0'... yes
[pynac-0.7.14] checking for the distutils Python package... yes
[pynac-0.7.14] checking for Python include path... -I/home/fbissey/sagemath/sage/local/include/python2.7
[pynac-0.7.14] checking for Python library path... -L/home/fbissey/sagemath/sage/local/lib -lpython2.7
[pynac-0.7.14] checking for Python site-packages path... /home/fbissey/sagemath/sage/local/lib/python2.7/site-packages

As you can see, the first test (AM_PATH_PYTHON) use sage-python23 properly. But not the second that goes directly for pythonX.Y. Worse, that's by design of the macro.

	AC_PATH_PROG([PYTHON],[python[$PYTHON_VERSION]])
	if test -z "$PYTHON"; then
	   AC_MSG_ERROR([Cannot find python$PYTHON_VERSION in your system path])
	   PYTHON_VERSION=""
	fi

And for the cherry on the top of the cake, I believe PYTHON_VERSION is set by AM_PATH_PYTHON. Conclusion AX_PYTHON_DEVEL doesn't do exotic python name, and you better have a regular python executable in the PATH. Now, I'd like to know more about that setup that has sage-python23 in the PATH but not the matching pythonX.Y. Did something happen while I wasn't looking?

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:15

Updating AX_PYTHON_DEVEL would still give the same result. There is probably something smarter to do.

@kiwifb
Copy link
Member

kiwifb commented Mar 25, 2020

comment:16

All the stuff found by AX_PYTHON_DEVEL could be found by snipping bits of it and using the results from AM_PATH_PYTHON. This is a substantial rewrite of upstream's configure.ac that we are looking at. We only need to lift the bits that figure PYTHON_CPPFLAGS and PYTHON_LDFLAGS (PYTHON_LIBS if lifted from the latest version of AX_PYTHON_DEVEL). The rest is provided by AM_PATH_PYTHON.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2020

comment:17

Replying to @kiwifb:

Yes for the split. I am surprised that problem shows now in pynac. I would have expected it to happen much earlier. The AX_PYTHON_DEVEL macro is 5 years old - which probably doesn't help.

It shows now because this is a test with system python3 (#27824) instead of building sage's own python3. On macOS, there is /usr/bin/python3 (which is Python 3.7) but no /usr/bin/python3.7.

@dimpase
Copy link
Member

dimpase commented Mar 25, 2020

comment:18

would it work with Homebrew's python3 ?
https://formulae.brew.sh/formula/python

Python shipped with MacOS is dodgy, IMHO.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2020

comment:19

Replying to @kiwifb:

All the stuff found by AX_PYTHON_DEVEL could be found by snipping bits of it and using the results from AM_PATH_PYTHON. This is a substantial rewrite of upstream's configure.ac that we are looking at. We only need to lift the bits that figure PYTHON_CPPFLAGS and PYTHON_LDFLAGS (PYTHON_LIBS if lifted from the latest version of AX_PYTHON_DEVEL). The rest is provided by AM_PATH_PYTHON.

Yes, this sounds like the correct approach.

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe changed the title Fix build failures of pplpy and pynac on macos with system python3 Fix build failures of pynac on macos with system python3 Mar 26, 2020
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 26, 2020

comment:21

Replying to @dimpase:

This should be probably split into two tickets.

pplpy upstream is https://gitlab.com/videlec/pplpy - also in CC:

pplpy is now #29408

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 26, 2020

comment:22

Some links for AX_PYTHON_DEVEL:

@dimpase
Copy link
Member

dimpase commented Mar 26, 2020

comment:23

probably we should try replacement using python-config- except that the name should not start with AC_...

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 26, 2020

comment:24

... and except that we do not have sage-python23-config

@kiwifb
Copy link
Member

kiwifb commented Mar 26, 2020

comment:25

In the same bag, you could be tempted to use python-X.Y.pc, AM_PATH_PYTHON identifies the python version correctly. I eliminated it from my solutions yesterday but that may be viable if it is shipped consistently. If not, the AX_PYTHON_DEVEL macro just runs python commands to get at sysconfig. It is a fairly valid strategy.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 3, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

cb15a59Use pynac-0.7.26.sage-2020-04-02
deb717fuse pynac-0.7.26.sage-2020-04-03

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 3, 2020

Changed commit from db5074a to deb717f

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 3, 2020

comment:62

This fixes the build on cygwin.

@dimpase
Copy link
Member

dimpase commented Apr 4, 2020

comment:63

please squash the commits

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 4, 2020

Changed commit from deb717f to 7344e2d

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 4, 2020

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7344e2dUse temporary upstream version pynac-0.7.26.sage-2020-04-03

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 5, 2020

comment:65

squashed, needs review

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 10, 2020

comment:66

Correct builds can be seen at https://github.com/mkoeppe/sage/actions/runs/74117913

@jhpalmieri
Copy link
Member

comment:67

It's good on OS X, built with various different Pythons (homebrew, system, Sage).

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 12, 2020

comment:68

Let's get this one into the next beta please

@dimpase
Copy link
Member

dimpase commented Apr 12, 2020

comment:69

looks good.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Apr 12, 2020

comment:70

Thanks!

@DaveWitteMorris
Copy link
Member

Attachment: pynac-0.7.26.sage-2020-04-03.log

pynac did not download on Mac OS 10.15.3

@DaveWitteMorris
Copy link
Member

comment:72

I apologize if this is noise (I may have messed up my installation somehow), but I thought I should report that I had trouble building pynac in 9.1rc0 on Mac OS 10.15.3, even with this ticket:

Attempting to download from https://github.com/mkoeppe/pynac/releases/download/pynac-0.7.26.sage-2020-04-03/pynac-0.7.26.sage-2020-04-03.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
************************************************************************

I merged #29408 and this ticket, and ran make build, then realized that the description of this ticket has a different make command, so I tried that, too. But pynac did not download. I am attaching the pynac-0.7.26.sage-2020-04-03.log log file from the more complicated build command, in case it is of use.

The build went smoothly after I manually downloaded pynac and put it in the upstream directory, so I want to say thanks for these patches! I never would have figured out how to fix the build errors on my own.

@dimpase
Copy link
Member

dimpase commented Apr 15, 2020

comment:73

this might be due to (system) python, more precisely its urllib module, being unable to provide correct certs...

does installing python 3.7 from python.org and also making sure it installs certs (there is a 2nd step in installation, iirc) fixes this?

@DaveWitteMorris
Copy link
Member

comment:74

Yes, that solved the problem. After installing python and certifi, pynac downloaded from mkoeppe's directory (and the build succeeded) when I used make SAGE_SPKG="sage-spkg -o" pynac-clean pynac. Thanks!

@vbraun
Copy link
Member

vbraun commented Apr 16, 2020

Changed branch from u/mkoeppe/fix_build_failures_of_pynac_on_macos_with_system_python3 to 7344e2d

@vbraun vbraun closed this as completed in bf8c781 Apr 16, 2020
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 16, 2020

Changed commit from 7344e2d to none

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jun 16, 2020

comment:76

Follow up: #29872

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: packages: standard p: blocker / 1 t: bug
Projects
None yet
Milestone
sage-9.1
Development

No branches or pull requests
7 participants
@kiwifb @vbraun @dimpase @jhpalmieri @isuruf @mkoeppe @DaveWitteMorris