Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update ECL's spkg-install for building multiple spkgs in parallel #9187

Closed
qed777 mannequin opened this issue Jun 8, 2010 · 34 comments
Closed

Update ECL's spkg-install for building multiple spkgs in parallel #9187

qed777 mannequin opened this issue Jun 8, 2010 · 34 comments
Assignees
@sagetrac-drkirkby
Labels
c: packages: standard p: minor / 4 t: enhancement
Milestone
sage-4.5

Comments

@qed777
Copy link
Mannequin

qed777 mannequin commented Jun 8, 2010

To build ECL with SAGE_PARALLEL_SPKG_BUILD="yes" on Mac OS X, we need to add, e.g.,

MAKEFLAGS=
export MAKEFLAGS

to the package's spkg-install.

Please see #8306 about building spkgs in parallel. For MAKEFLAGS, see the GNU Make manual.

CC: @sagetrac-drkirkby @jhpalmieri @nexttime @jaapspies

Component: packages: standard

Author: Mitesh Patel

Reviewer: John Palmieri, John Cremona

Merged: sage-4.5.rc1

Issue created by migration from https://trac.sagemath.org/ticket/9187
@qed777 qed777 mannequin added this to the sage-5.0 milestone Jun 8, 2010
@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jun 8, 2010

comment:1

I think we should work from #8951.

@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jun 9, 2010

spkg patch. Set empty MAKEFLAGS.

@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jun 9, 2010

comment:2

Attachment: trac_9187-ecl_makeflags.patch.gz

I've put a new spkg at

(Or should it be p1?) This is based against the package mentioned in this comment at #8951.

@qed777 qed777 mannequin added the s: needs review label Jun 9, 2010
@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jun 18, 2010

comment:3

Unfortunately, there are two spkgs on #8951 and neither have the patches from #8089 and both claim in SPKG.txt to have things they do not.

I intend creating a 10.4.1 package which has this, and all other relevant fixes in the one package. Since no ecl-10.4.1.spkg has been merged, this should be called ecl-10.4.1.spkg. The patch levels start at 0, not 1, but in this case as nothing has been merged, there is no need to increment the patch level.

Please see #9264, which will have these changes, those which are listed as being in #8591 but are not, and also that at #8089

Dave

@sagetrac-drkirkby sagetrac-drkirkby mannequin self-assigned this Jun 18, 2010
@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jun 18, 2010

comment:4

The attached patch is ok and deserves positive review. However, the .spkg attached is not ideal, as it overlooks some other changes which need merging. I think a new .spkg needs to be created, which has all the minor changes together.

@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jun 21, 2010

comment:5

There is no need to merge this now.

#9264, which has positive review, includes this patch, along with several others related to ECL. All the relevant problems with ECL are solved by #9264, including this one.

Dave

@rlmill rlmill mannequin added r: duplicate and removed p: minor / 4 labels Jun 25, 2010
@rlmill rlmill mannequin closed this as completed Jun 25, 2010
@rlmill rlmill mannequin removed this from the sage-5.0 milestone Jun 25, 2010
@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jul 12, 2010

comment:8

Related tickets: #9264, #9460, #9474.

@qed777 qed777 mannequin removed the r: duplicate label Jul 12, 2010
@qed777 qed777 mannequin added this to the sage-4.5.1 milestone Jul 12, 2010
@qed777 qed777 mannequin reopened this Jul 12, 2010
@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jul 12, 2010

spkg patch rebased vs. #9474.

@qed777
Copy link
Mannequin Author

qed777 mannequin commented Jul 12, 2010

comment:9

Attachment: trac_9187-ecl_makeflags.2.patch.gz

Here's a new spkg based on #9474:

http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg

It works for me with 4.5.rc0 + #7379 on sage.math, but I have not yet had a chance to test it on bsd.math.

@jhpalmieri
Copy link
Member

comment:10

This works for me on sage.math and on an OS X box running 10.6, in both cases building spkg's in parallel. I'm building on t2 now; if there are problems, I'll report them.

@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jul 12, 2010

comment:11

Replying to @qed777:

Here's a new spkg based on #9474:

http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg

It works for me with 4.5.rc0 + #7379 on sage.math, but I have not yet had a chance to test it on bsd.math.

It works for me on bsd.math - at least ecl builds ok. I've not at this time completed the build, but ecl has built ok.

@jhpalmieri
Copy link
Member

comment:12

Replying to @sagetrac-drkirkby:

It works for me on bsd.math - at least ecl builds ok. I've not at this time completed the build, but ecl has built ok.

That's what I meant, too. The same is also true on t2 now: this version of ecl has built successfully as part of a parallel build, but the full build has several more hours to go...

@JohnCremona
Copy link
Member

comment:13

I am testing a new build (on 64-bit ubuntu) of 4.5.rc0 + http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg with SAGE_PARALLEL_SPKG_BUILD="yes" (which I have never tried before) and will report back. Though these ecl tickets are so confusing that I am not sure this will tell anyone anything useful, I thought it would be fun. (The other users of the machine may not agree, of course!)

@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jul 13, 2010

comment:14

Replying to @JohnCremona:

I am testing a new build (on 64-bit ubuntu) of 4.5.rc0 + http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg with SAGE_PARALLEL_SPKG_BUILD="yes" (which I have never tried before) and will report back. Though these ecl tickets are so confusing that I am not sure this will tell anyone anything useful, I thought it would be fun. (The other users of the machine may not agree, of course!)

I think you test results will be very useful, as there is a distinct lack of direct evidence for this working on Linux. There is evidence it is working on OS X and Solaris.

You also need to export MAKE, with something like

$ export MAKE="make -j6"

or however many threads you want. Do not put a space between the 'j' and the '6'.

Dave

@JohnCremona
Copy link
Member

comment:17

Replying to @sagetrac-drkirkby:

John,

I'm not sure if you mis-understood me there.

$ export MAKE="make -j 10"

will not work as well as it could do. The .spkg's will build in parallel, but when it gets to building the Sage library, that will only build serially. You should use

$ export MAKE="make -j10"

It's ok, I knew that...

I assume you are aware you also need a library patch

https://github.com/sagemath/sage-prod/files/10646697/trac-7379-decorator-defaults.patch.gz

... and that. 83 spkgs now installed and counting....

(only the library patch, not the sagenb patch on that ticket). Only part of #7379 got merged into 4.5.rc0

Dave

@jhpalmieri
Copy link
Member

Reviewer: John Palmieri, John Cremona

@jhpalmieri
Copy link
Member

comment:18

I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?

@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 13, 2010

comment:19

I think this ecl-10.2.1.p1.spkg can be merged into rc1, but could you MacOS X testers check which version of Boehm GC is used during install (i.e., the one shipped with Sage or the one included in ECL)?

$ grep -i boehm spkg/logs/ecl-10.2.1.p1.log | head -n 2

should show soemthing like

checking whether we can use the existing Boehm-Weiser library ... yes
checking if we need to copy GC private headers ... checking if we use Boehm-Demers-Weiser precise garbage collector... yes

or

checking if we use Boehm-Demers-Weiser precise garbage collector... yes
configure: Configuring included Boehm GC library:

(The latter is from Dave's install log on bsd.math/MacOS X 10.4.0.)

@JohnCremona
Copy link
Member

comment:20

Replying to @jhpalmieri:

I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?

It finished building, now building docs. Then I'll do a ptestlong and report back -- currently I don't know if what was built actually works!

@jhpalmieri
Copy link
Member

comment:21

On my OS X box, it says:

checking if we use Boehm-Demers-Weiser precise garbage collector... yes
...
configure: Configuring included Boehm GC library:

I'm not an expert in these things, but in src/src/configure, it says:

        darwin*)
                thehost='darwin'
                shared='yes'
                ...
                # The Boehm-Weiser GC library shipped with Fink does not work
                # well with our signal handler.
                enable_boehm=included

which leads me to believe that on OS X, it will be a little work to use Sage's installation of boehm_gc. I think this should go on a follow-up ticket, and it should probably be done in coordination with the ecl people.

@jhpalmieri
Copy link
Member

comment:22

(That is, on OS X, it looks like any setting to the --enable-boehm flag gets overridden.)

@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 13, 2010

comment:23

Replying to @jhpalmieri:

(That is, on OS X, it looks like any setting to the --enable-boehm flag gets overridden.)

That's odd, since Sage's Boehm GC includes a patch for MacOS X 10.6.

I agree we should open a new ticket for that and perhaps include a p2 spkg in 4.5.rc2(?), 4.5.final or 4.5.1.

The package has to be cleaned up anyway, and perhaps we switch back to ECL 10.4.1.

@rlmill
Copy link
Mannequin

rlmill mannequin commented Jul 13, 2010

comment:24

Replying to @nexttime:

Replying to @jhpalmieri:
I agree we should open a new ticket for that and perhaps include a p2 spkg in 4.5.rc2(?), 4.5.final or 4.5.1.

4.5.1.

@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 13, 2010

comment:25

Replying to @rlmill:

4.5.1.

That's what I meant by changing the milestone rather than the priority. ;-)

@JohnCremona
Copy link
Member

comment:26

Replying to @JohnCremona:

Replying to @jhpalmieri:

I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?

It finished building, now building docs. Then I'll do a ptestlong and report back -- currently I don't know if what was built actually works!

All tests passed == +++

@rlmill
Copy link
Mannequin

rlmill mannequin commented Jul 13, 2010

Merged: sage-4.5.rc1

@rlmill
Copy link
Mannequin

rlmill mannequin commented Jul 13, 2010

comment:27

Merged

http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg

@rlmill rlmill mannequin modified the milestones: sage-4.5.1, sage-4.5 Jul 13, 2010
@rlmill rlmill mannequin removed the s: positive review label Jul 13, 2010
@rlmill rlmill mannequin closed this as completed Jul 13, 2010
@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 13, 2010

comment:28

Replying to @nexttime:

Replying to @jhpalmieri:

(That is, on OS X, it looks like any setting to the --enable-boehm flag gets overridden.)

That's odd, since Sage's Boehm GC includes a patch for MacOS X 10.6.

Just for the record: The patch just avoids a dumb deprecation error(!), and it is already included in ECL's Boehm GC (otherwise ECL wouldn't have built on MacOS X 10.6).

@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 13, 2010

comment:29

For a 4.5 final, we should remove the unnecessary baggage, though (as SPKG.txt instructs, btw):

leif@portland:~/Sage/spkgs/ecl-10.2.1.p1$ du -ch src/msvc/ src/contrib/encodings/
16K	src/msvc/ecl
8.0K	src/msvc/gc
8.0K	src/msvc/gmp/mpz
32K	src/msvc/gmp/mpn/generic
76K	src/msvc/gmp/mpn/amd64i
48K	src/msvc/gmp/mpn/x86i/pentium4/sse2
32K	src/msvc/gmp/mpn/x86i/pentium4/mmx
92K	src/msvc/gmp/mpn/x86i/pentium4
48K	src/msvc/gmp/mpn/x86i/p6/mmx
8.0K	src/msvc/gmp/mpn/x86i/p6/p3mmx
128K	src/msvc/gmp/mpn/x86i/p6
300K	src/msvc/gmp/mpn/x86i
412K	src/msvc/gmp/mpn
12K	src/msvc/gmp/build.vc8/gen-fib
16K	src/msvc/gmp/build.vc8/gen-fac_ui
28K	src/msvc/gmp/build.vc8/lib_gmpxx
256K	src/msvc/gmp/build.vc8/dll_gmp_amd64
16K	src/msvc/gmp/build.vc8/gen-bases
372K	src/msvc/gmp/build.vc8/lib_gmp
240K	src/msvc/gmp/build.vc8/dll_gmp_p4
40K	src/msvc/gmp/build.vc8/lib_gmp_p0
36K	src/msvc/gmp/build.vc8/lib_gmp_p3
240K	src/msvc/gmp/build.vc8/dll_gmp_p0
180K	src/msvc/gmp/build.vc8/dll_mpfr
28K	src/msvc/gmp/build.vc8/gen-psqr
72K	src/msvc/gmp/build.vc8/lib_gmp_gc
40K	src/msvc/gmp/build.vc8/lib_gmp_p4
48K	src/msvc/gmp/build.vc8/lib_gmp_amd64
436K	src/msvc/gmp/build.vc8/dll_gmp_gc
240K	src/msvc/gmp/build.vc8/dll_gmp_p3
208K	src/msvc/gmp/build.vc8/lib_mpfr
4.8M	src/msvc/gmp/build.vc8
5.2M	src/msvc/gmp
8.0K	src/msvc/c
16K	src/msvc/util
12K	src/msvc/doc
5.3M	src/msvc/
544K	src/contrib/encodings/
5.8M	total
leif@portland:~/Sage/spkgs/ecl-10.2.1.p1$ tar cj src/msvc/ src/contrib/encodings/ | wc -c
859300

@sagetrac-drkirkby
Copy link
Mannequin

sagetrac-drkirkby mannequin commented Jul 13, 2010

comment:30

Replying to @nexttime:

For a 4.5 final, we should remove the unnecessary baggage, though (as SPKG.txt instructs, btw):

That's up to Robert, but I tend to feel saving about 840 KB is of relatively low importance, and can be left for 4.5.1. Especially considering there is a new version of ECL available, and we would expect to resolve the ECL/Maxima issues some time in the near future. So the ecl package will probably be updated for 4.5.1.

Dave

@rlmill
Copy link
Mannequin

rlmill mannequin commented Jul 13, 2010

comment:31

Replying to @nexttime:

For a 4.5 final, we should remove the unnecessary baggage

Please open a ticket and make a p2 with only these changes. I think it's too late for 4.5.

@nexttime
Copy link
Mannequin

nexttime mannequin commented Jul 14, 2010

comment:32

Replying to @rlmill:

Replying to @nexttime:

For a 4.5 final, we should remove the unnecessary baggage

Please open a ticket and make a p2 with only these changes. I think it's too late for 4.5.

Ticket up at #9493.

I unfortunately cannot upload an spkg because I don't have an account on sage.math.

(Perhaps I'll ask again someone to upload it from an e-mail attachment, but it's still fairly large.)

@qed777 qed777 mannequin mentioned this issue Jun 8, 2010
Closed
This was referenced Jul 5, 2011
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sagetrac-drkirkby sagetrac-drkirkby

Labels
c: packages: standard p: minor / 4 t: enhancement
Projects
None yet
Milestone
sage-4.5
Development

No branches or pull requests
2 participants
@jhpalmieri @JohnCremona