port Sage to FreeBSD 12

[dimpase]

In FreeBSD 12 the long-standing incompleteness of its libm has been helped. This simplifies a lot, compared to #22679.

We will not try to work around the toolchain problem related to gfortran/libgcc_s. Instead, we just add a line at the end of /etc/libmap.conf as
follows:

libgcc_s.so.1 /usr/local/lib/gcc7/libgcc_s.so.1

Also, we create the following symbolic links:

$ ls -l ~/bin
total 0
lrwxr-xr-x  1 dima  dima  20 Apr 27 12:49 make -> /usr/local/bin/gmake
lrwxr-xr-x  1 dima  dima  21 Apr 27 13:46 patch -> /usr/local/bin/gpatch

and use the standard shell, /bin/sh. In ~/.profile we put

export LDFLAGS="-L/usr/local/lib"
export CFLAGS="-I/usr/local/include"
export CXXFLAGS=$CFLAGS
export CPPFLAGS=$CFLAGS
export FCFLAGS=$CFLAGS
export FFLAGS=$CFLAGS

to account for clang not looking into /usr/local by default.

Things to fix:

• rubiks makefile should use MAKE #26250 (fixes to rubiks)
• configure threading in flint in sync with NTL #27764 (a fix to flint)
• Upgrade: psutil 5.8.0 #26253 (update of psutils)
• yasm check is broken on FreeBSD (it's always YES!)
• remove cephes package #26602 (cephes removed)
• Upgrade to gc 8.0.4 and libatomic 7.6.10 #26287 (update to gc, to the FreeBSD's version)
• upgrade matplotlib to 2.2.3 #26288 (update to matplotlib, fixes compilation errors)
• Upgrade latte_int to 1.7.5 #26190 (update to latte_int---which is an optional package)
• update openblas to version 0.3.3 #26733 (update to openblas)
• Upgrade to Giac 1.5 #26315 (update to giac, to be able to use clang 6.0.1)
• remove "Replacing library search directory in linker " in src/setup.py #27262 (needed to link sagelib to libs from /usr/local/lib)

---

There is some hope that the libgcc_s.so.1 problem will finally be [fixed in FreeBSD https://lists.freebsd.org/pipermail/freebsd-ports/2019-April/115972.html]

The strategy which would pay off to make Sage an "easy" FreeBSD port is to add/enhance as many needed by Sage external libraries as separate ports into FreeBSD, so that Sage does not need to vendor this. To this end, #27330 is very relevant ticket.

A first foray into this is enhancing FreeBSD's port of flint2 to be linkable with NTL (something that is a requirement for flint2 in Sage), see here. The necessary work on the flint2 side has been done in the meantime, see flintlib/flint#543 - so now it's only the FreeBSD part that needs to be done.

Few obvious other candidates for a port in FreeBSD are gf2x and ecm.

Depends on #26253
Depends on #27764

CC: @mkoeppe

Component: porting: BSD

Author: Dima Pasechnik, Li-Wen Hsu

Issue created by migration from https://trac.sagemath.org/ticket/26249

[lwhsu]

Author: Dima Pasechnik, Li-Wen Hsu

[dimpase]

comment:11

This scipy problem is still there.
(while this works with the system's python2)
It's a matter of comparing the patches used there with what we do in Sage. Perhaps they disable multithreading...

[embray]

comment:12

I'm currently overextended as always and don't have any deep personal motivation to work on FreeBSD support, but I would definitely like to have it if we can so if there's anything I can help with here I'll make a FreeBSD 12 VM so I can have it to play around on.

[dimpase]

comment:13

Replying to @embray:

I'm currently overextended as always and don't have any deep personal motivation to work on FreeBSD support, but I would definitely like to have it if we can so if there's anything. I can help with here I'll make a FreeBSD 12 VM so I can have it to play around on.

I can give you access to the FreeBSD 12.0 VM I am running on GCE, this would be easier, I guess. (12.0 is still in beta/RC cycle, so installation is not straightforward, you basically need to build it from source).

[dimpase]

comment:14

currently testing what we have here (with Sage 8.6) on the released FreeBSD 12.0

[dimpase]

comment:15

the next issue we hit is Python3 on Sage 8.7.beta3: https://bugs.python.org/issue6299
(pyexpat module does not build)

[dimpase]

Dependencies: #27254

[dimpase]

comment:16

Replying to @dimpase:

the next issue we hit is Python3 on Sage 8.7.beta3: https://bugs.python.org/issue6299
(pyexpat module does not build)

for the time being, working around this by not installing setuptools and pip for py3

[dimpase]

Changed dependencies from #27254 to #27254, #/26253

[dimpase]

Changed dependencies from #27254, #/26253 to #27254, #26253, #27262

[dimpase]

Changed dependencies from #27254, #26253, #27262 to #27254, #26253, #27262, #27764

[thierry-FreeBSD]

comment:38

Some news: after #29502 for suitesparse and #29024 for singular, only 13 libraries (+ of course the modules built around them!) cause problems with staging: libbraiding libbrial_groebner libbrial libecl libfflas libffpack libgap libgc libhomfly liblinbox libppl libpynac libzn_poly-0.

Among them, all are listed in #27330 (with or without a ticket), but libpynac. We do not even have pynac in the FreeBSD ports tree (but we have math/GiNaC) => it will be my next target.

[thierry-FreeBSD]

comment:39

After giac (#29541) and pynac (#29542), I tried to use a system package for ECL (lang/ecl) and Maxima (math/maxima), but without success for the moment: see #29617. To be tried again when ECL will be upgraded (#22191)!

Meanwhile, our ports must be patched:

• lang/ecl: add a symbolic link for $PREFIX/lib/ecl

• math/maxima: this one is more tricky, because now ECL is not even supported and must be added, and to get a useful package we need to replace the default from SBCL to ECL (or create a slave port?)

They are not submitted yet, but if someone want to work on #29617 the patches are available at

• https://people.freebsd.org/~thierry/lang_ecl.diff
• https://people.freebsd.org/~thierry/math_maxima.diff

[dimpase]

comment:40

Replying to @thierry-FreeBSD:

After giac (#29541) and pynac (#29542), I tried to use a system package for ECL (lang/ecl) and Maxima (math/maxima), but without success for the moment: see #29617. To be tried again when ECL will be upgraded (#22191)!

Meanwhile, our ports must be patched:

• lang/ecl: add a symbolic link for $PREFIX/lib/ecl

• math/maxima: this one is more tricky, because now ECL is not even supported and must be added, and to get a useful package we need to replace the default from SBCL to ECL (or create a slave port?)

there are many free Common Lisp implementations, I suppose there can be lang/cl as a "main" package with a number of slaves, perhaps SBCL as a standard one.

[mkoeppe]

comment:41

I'd recommend to add system package information for FreeBSD. See #29354 where this is done for Slackware (except for the bits that use docker)

[thierry-FreeBSD]

comment:42

Well, every package listed in #27330 have a corresponding package from the FreeBSD ports tree and a spkg-configure.m4 to use it!

Next step: try to solve the problem with bad staging (non-respect of $DESTDIR) for the libraries built by Sage.

[mkoeppe]

comment:43

Replying to @thierry-FreeBSD:

Well, every package listed in #27330 have a corresponding package from the FreeBSD ports tree and a spkg-configure.m4 to use it!

Yes, and the trick is to put the names of these packages in a freebsd.txt file to record the correspondence of spkg to FreeBSD port.

[thierry-FreeBSD]

comment:44

OK, I'll do it!

[thierry-FreeBSD]

comment:45

Replying to @thierry-FreeBSD:

OK, I'll do it!

Done in #29653.

[mkoeppe]

comment:46

Next, if you're serious about FreeBSD support, I would recommend setting up testing infrastructure with our tox.ini and virtualbox (https://www.freebsd.org/doc/handbook/virtualization-guest-virtualbox.html) or a similar solution so that Sage developers and our CI infrastructure have a chance of testing on that platform.

[thierry-FreeBSD]

comment:47

Replying to @mkoeppe:

I would recommend setting up testing infrastructure with our tox.ini and virtualbox (https://www.freebsd.org/doc/handbook/virtualization-guest-virtualbox.html) or a similar solution so that Sage developers and our CI infrastructure have a chance of testing on that platform.

Thanks, this is interesting - but at this point a bit premature: there is no compilation error, but Sage is not yet ready for FreeBSD, even if we are making progress.

[thierry-FreeBSD]

comment:48

Well, we are almost done!

It is not yet committable in the FreeBSD ports tree, because there are some pre-requisites (isl to be upgraded, and two possible forks for ECL and Maxima if no better solution is found).

See https://reviews.freebsd.org/D24195 for details.

[dimpase]

comment:49

for ecl I suggest to use a not yet merged
#22191

which looks good on Linux and OpenBSD.

also a new release of Flint is coming.

[thierry-FreeBSD]

comment:50

Thanks! I'm just in discussion with the maintainers of these two ports (ECL and Maxima). Let's check this version.

[dimpase]

comment:51

ecl obviously must be upgraded to the latest release (maxima currently packaged with Sage works well with it, too).

which bugs of maxima one prefers, is another question. :-)

our experience with updating maxima is that each new release has serious regressions.

[dimpase]

comment:52

here is Flint update ticket
#29719

it is awaiting upstream putting in
finishing touches here:
flintlib/flint#753

[thierry-FreeBSD]

comment:53

I guess that it is safer to get Sage 9.1 into our ports tree with the current dependencies.

We'll look to upgrade these afterwards!

[thierry-FreeBSD]

comment:54

We are done, 9.1 is in the tree:
https://www.freshports.org/math/sage/

and we are working on the upgrade of ECL to 20.4.24:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247283

[dimpase]

comment:55

Replying to @thierry-FreeBSD:

We are done, 9.1 is in the tree:
https://www.freshports.org/math/sage/

This is great!

I guess we should make the Sagemath source FreeBSD-friendly, so that one can build
Sage from source using the standard ./configure + make route, right? This would need

• adding data for the needed packages, that is, there should be
  build/pkgs/*/distros/freebsd.txt files added with package names.

• adding pre-reqs data: build/pkgs/freebsd.txt and build/pkgs/freebsd-bootstrap.txt

• import patches you added in FreeBSD to Sage (if any)

[dimpase]

Changed dependencies from #27254, #26253, #27262, #27764 to #26253, #27764

[thierry-FreeBSD]

comment:57

The most important patch concerns configure.ac: without it, you cannot even run configure!

I submitted it with the system package information in #29653, but it has been rejected.

The other patches are mostly specific and related to the ports system, caused by differences in directories, but shouldn't be necessary for those trying to build Sage by themselves with the bundled packages.

mkoeppe asked me to set up a testing infrastructure. I'm not familiar at all with that, but I'll try to do it (any pointers appreciated!).

[dimpase]

comment:58

I will look at #29653 - it needs a rebase over the latest 9.2.beta1

regarding testing - is docker ported to FreeBSD?

Anyway, one can set a patchbot, for this one just needs a FreeBSD host that will pull branches and build/test them.

[thierry-FreeBSD]

comment:59

There exists a port of docker: https://www.freshports.org/sysutils/docker (but I never used it!).

[lwhsu]

comment:60

I don't think docker port is production ready at the moment. I suggest the faster way would be having a github mirror and use hosted CI service: https://wiki.freebsd.org/HostedCI

[dimpase]

comment:61

Replying to @thierry-FreeBSD:

We are done, 9.1 is in the tree:
https://www.freshports.org/math/sage/

I've revived a VM with FreeBSD, it's running 12.1-RELEASE-p6

How do I get the new port in there? Do I need a system update (if yes, to what?)?
Or it's easier?
I've run portsnap, and still see old stuff for sage in /usr/ports/math/sage.

[thierry-FreeBSD]

comment:62

You did the right thing!

• 12.1-RELEASE-p6 is OK

• `portsnap fetch update' is a well supported method - but I do not know how long it takes to be refreshed. I guess that if you try again is 1h or 2 it should be fine.

[lwhsu]

comment:63

Replying to @dimpase:

Replying to @thierry-FreeBSD:

We are done, 9.1 is in the tree:
https://www.freshports.org/math/sage/

I've revived a VM with FreeBSD, it's running 12.1-RELEASE-p6

How do I get the new port in there? Do I need a system update (if yes, to what?)?
Or it's easier?
I've run portsnap, and still see old stuff for sage in /usr/ports/math/sage.

I guess there are two possibilities, one is portsnap is not updated yet (unlikely, IIRC it updates hourly), the other is it's tracking quarterly branch, and 9.1 is only in latest branch now. I haven't used portsnap for a while so I might also be wrong.

To get the latest ports tree in real-time, you can try to use svn to checkout from:

https://svn.freebsd.org/ports/head

or git from

https://github.com/freebsd/freebsd-ports

[dimpase]

comment:64

Replying to @thierry-FreeBSD:

You did the right thing!

• 12.1-RELEASE-p6 is OK

• `portsnap fetch update' is a well supported method - but I do not know how long it takes to be refreshed. I guess that if you try again is 1h or 2 it should be fine.

OK, thanks, seems to be in progress now, after I did portsnap fetch update (my first try was ``portsnap fetch` - probably they are not the same thing)

lwhsu - did you see my email about CI? (just checking)

[lwhsu]

comment:65

Replying to @dimpase:

Replying to @thierry-FreeBSD:

You did the right thing!

• 12.1-RELEASE-p6 is OK

• `portsnap fetch update' is a well supported method - but I do not know how long it takes to be refreshed. I guess that if you try again is 1h or 2 it should be fine.

OK, thanks, seems to be in progress now, after I did portsnap fetch update (my first try was ``portsnap fetch` - probably they are not the same thing)

Yes there are two steps. One if for fetching the content and the other one is for actually updating the ports tree. See portsnap(8)

lwhsu - did you see my email about CI? (just checking)

Yes, please forgive me for needing longer to response.