Removal of pyprocessing causing problems as _multiprocessing not building on Solaris

[sagetrac-drkirkby]

Since #6503 removed pyprocessing from Sage, multiprocessing (or perhaps _multiprocessing) is needed, but this is not building on Solaris 10 SPARC.

gcc -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I/export
/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/./Include -I. -IInclude -I./Include -I/export/home/drkirkby/32/sage-4.3.4.alpha0/local/include -I/usr/local/include
-I/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Include -I/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src -c /export/home/drkirkb
y/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c -o build/temp.solaris-2.10-sun4u-2.6/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/bu
ild/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.o
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c: In function 'multiprocessing_sendfd':
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:100: warning: implicit declaration of function 'CMSG_SPACE'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:110: error: 'struct msghdr' has no member named 'msg_control'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:111: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:114: warning: implicit declaration of function 'CMSG_FIRSTHDR'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:114: warning: assignment makes pointer from integer without a cast
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:117: warning: implicit declaration of function 'CMSG_LEN'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:118: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:119: warning: implicit declaration of function 'CMSG_DATA'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c: In function 'multiprocessing_recvfd':
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:145: error: 'struct msghdr' has no member named 'msg_control'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:146: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:149: warning: assignment makes pointer from integer without a cast
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:153: error: 'struct msghdr' has no member named 'msg_controllen'

Further down the log, the fact the _multiprocessing module has failed to build is clearly stated:

Failed to find the necessary bits to build these modules:
_bsddb		   _hashlib		_ssl
_tkinter	   bsddb185		gdbm
linuxaudiodev	   ossaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_curses		   _curses_panel	_multiprocessing

Upstream: Not yet reported upstream; Will do shortly.

CC: @mwhansen @jhpalmieri @williamstein @sagetrac-mvngu @jaapspies

Component: porting: Solaris

Author: David Kirkby

Reviewer: Minh Van Nguyen

Merged: sage-4.3.4.alpha1

Issue created by migration from https://trac.sagemath.org/ticket/8440

[sagetrac-drkirkby]

Author: David Kirkby

[sagetrac-drkirkby]

Changed upstream from Fixed upstream, but not in a stable release. to Not yet reported upstream; Will do shortly.

[sagetrac-drkirkby]

comment:2

The solution to this problem is found in the old pyprocessing package, where it says:

#   HAVE_FD_TRANSFER
#     Set this to 1 to compile functions for transferring file
#     descriptors between processes over an AF_UNIX socket using a
#     control message with type SCM_RIGHTS.  On Unix the pickling of 
#     of socket and connection objects depends on this feature.
#
#     If you get errors about missing CMSG_* macros then you should
#     set this to 0.

This is not as well documented in Python's setup.py, but by setting

HAVE_FD_TRANSFER=0

in Python's top-level setup.py, the problem goes away. Python's build then reports:

Failed to build these modules:
_curses		_curses_panel

The _multiprocessing module is now building ok.

The new setup.py has a Solaris-specific section, which I added. However, so reviewers can be even more confident this will only affect Solaris, the patch is only applied on Solaris.

I also took time to address a minor issue at #8356, where '--without-libpng' was used, despite the fact the option is no longer recognised.

== Notes for Release Manager ==
Prerequisites:

• GCC reports incorrect flags compiling descent_two_isogeny.c on Solaris 10 #7867 (This already has positive review)

This patch should be applied on top of the changes at #7867

Once this ticket is closed, #8356 may be closed too.

It would be appreciated if #8374, #8375, 8391 and #8404 could also be integrated into the next alpha, as that would have a high probability of allowing all doc tests to pass. All these tickets have positive review.

[sagetrac-drkirkby]

comment:3

I forgot to put the location of the package

http://boxen.math.washington.edu/home/kirkby/portability/python-2.6.4.p7/python-2.6.4.p7.spkg

Patch will be attached in a minute

[sagetrac-drkirkby]

Attachment: 8440-python.patch.gz

Mercurial patch

[sagetrac-mvngu]

comment:4

The updated package python-2.6.4.p7.spkg needs a minor clean up:

[mvngu@sage python-2.6.4.p7]$ hg status
? patches/locale.pyc

That is, we don't place Python bytecode under revision control. Nor do we put any binary or Python bytecode under the directory "patches/".

[sagetrac-drkirkby]

comment:5

Hi Minh,
I need to go out very soon and get a train, so don't have time to fully investigate this, but there appears there may be something wrong on the math's computer setup, as what I am putting there, and what I can see in the browser are not the same. (This could be the fact the ZIL log is disabled - complex story, and one I need to discuss again with William).

There should be a .spkg in the directory http://boxen.math.washington.edu/home/kirkby/portability/python-2.6.4.p7/

which does (for me at least) not have any such file or problem. There should be a directory below that where I extracted the file. But when I look with the browser, I can not see any of it! So in summary.

• I don't see the problem you do.
• Changes I am making on the file system do not appear to be reflected in what I can see from my browser.

If you look at this location, you might find the package, but I'm totally confused. I think the file system might be messed up.

kirkby@t2:[/home/kirkby/portability/python-2.6.4.p7] $ ls 
python-2.6.4.p7       python-2.6.4.p7.spkg
kirkby@t2:[/home/kirkby/portability/python-2.6.4.p7] $ 

I will have to look later, as I need to go out now.

Dave

[sagetrac-drkirkby]

comment:6

I managed to look at this in more detail today. The reason the directory was not visable was a permissions problem, and nothing to do with file system errors.

I still can't understand why you see this odd file, as I don't:

kirkby@sage:~/portability/python-2.6.4.p7/python-2.6.4.p7$ ls patches/locale.pyc
ls: cannot access patches/locale.pyc: No such file or directory
kirkby@sage:~/portability/python-2.6.4.p7/python-2.6.4.p7$ hg status

[sagetrac-drkirkby]

comment:7

Minh,

could you double- check you are using the same package as me, since I simply can't see this spurious file.

[sagetrac-mvngu]

Reviewer: Minh Van Nguyen

[sagetrac-mvngu]

comment:8

Replying to @sagetrac-drkirkby:

could you double- check you are using the same package as me, since I simply can't see this spurious file.

I have re-checked python-2.6.4.p7.spkg and indeed it's OK by me. I have no idea why I received the warning about a bytecode file under the directory patches/. What I would usually do is take an spkg that has been compressed or just tarball'd, and unpack it. Then I would go through that unpacked spkg with Mercurial to make sure that Mercurial is happy with the repository under consideration. I have built python-2.6.4.p7.spkg on the following:

• bsd.math
• Cygwin (winxp1 on boxen.math)
• rosemary.math
• sage.math
• t2.math

On all systems/platforms that I tested, the said Python package builds as claimed. Where relevant (i.e. bsd.math, rosemary.math, sage.math), all doctests passed.

[mwhansen]

Merged: sage-4.3.4.alpha1

[sagetrac-drkirkby]

comment:10

#8356 can be closed too following the inclusion of this updated python package.

dave