Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade #10801

williamstein · 2011-02-19T00:57:23Z

This is a frequently requested feature. We should start with some little script that does this, and build on it. I tried all the following, and ran the test suite and it worked fine.

rm SAGE_ROOT/local/lib/*.a
rm SAGE_ROOT/devel/sage/build/lib.*
rm SAGE_ROOT/devel/sage/build/temp.*
strip SAGE_ROOT/local/bin/Singular-* SAGE_ROOT/local/bin/gfan # gfan is a huge win.
jsmath image fonts are in MoinMoin and are HUGE. Just delete everything related to moinmoin...
strip SAGE_ROOT/local/lib/*.so
rm -rf SAGE_ROOT/local/python/site-package/MoinMoin
rm all files in SAGE_ROOT/devel/sage/sage/ that begin "* Generated by Cython" (I didn't do that)

The patches below actually provide a new make target, "micro_release", rather than adding a new option to sage.

Apply attachment: trac_10801-root_repo.patch to the Sage root repository.

Apply attachment: trac_10801-local_bin_repo.patch to the Sage scripts repository.

CC: @jasongrout @kcrisman

Component: packages: standard

Keywords: sd32

Author: William Stein, Keshav Kini

Reviewer: Benjamin Jones, Keshav Kini

Merged: sage-4.7.2.alpha3

Issue created by migration from https://trac.sagemath.org/ticket/10801

The text was updated successfully, but these errors were encountered:

jasongrout · 2011-03-17T16:41:33Z

comment:1

It seems like the LiveCD folks have also worked on stuff like this...

kini · 2011-03-22T00:57:59Z

comment:2

If we mount a sage source installation with mount option strictatime or relatime we can just touch all the files to have an access time in the deep past (with something like cd $SAGE_ROOT ; for x in $(find) ; do touch -ad "1981-02-03 04:05:06" ; done), then run the whole doctest suite, then use stat() to find files which were unused. This should give us a good idea of what is never used. Unfortunately this does not seem possible on sage.math - POSIX atime is not respected in my homedir, probably as it is mounted over NFS...

sagetrac-pi · 2011-03-23T19:09:31Z

comment:4

if you're using linux, take a look at inotify-tools which has nice and clean command line tools, as it will give you a more fine-grained and realtime sense of what files are being used as they are accessed/modified to track down what's being used by which calls/tests/parts of sage.

jhpalmieri · 2011-03-23T23:16:58Z

comment:5

We can probably remove any Mercurial repositories.

kini · 2011-03-25T00:17:01Z

comment:7

Here's some data I collected about what files are used and what files aren't, in sage 4.7.alpha3. Surprisingly many aren't, but come to think of it there's going to be a lot of stuff that users would want which isn't used by any doctest. I guess one should think of this data as a lower bound...

sagetrac-mhampton · 2011-05-28T03:51:38Z

comment:8

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

kini · 2011-06-14T23:19:15Z

comment:9

Not delete, but strip. This removes debug symbols.

nexttime · 2011-08-22T09:09:58Z

comment:10

Slightly related: sage -crap ... (local/bin/sage-crap)

Note that we already have a few *clean targets in the top-level Makefile; there could be more (like strip), and some of the others could perhaps get improved or extended.

I personally wouldn't use -strip for anything else than really stripping debug symbols, i.e., if it is e.g. intended to delete also static libraries (which may be needed if someone installs some package, not necessarily an spkg), I'd choose a different name for such an option (or make target).

williamstein · 2011-08-23T18:02:07Z

comment:11

After a long discussion, Keshav and I came up with:

   make micro_release

since this is a lot like "make release" with other software....

williamstein · 2011-08-23T18:34:54Z

comment:12

Replying to @sagetrac-mhampton:

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

I didn't suggest deleting gfan, but stripping it.

williamstein · 2011-08-23T20:04:58Z

Attachment: trac_10801-root_repo.patch.gz

apply to the repo in $SAGE_ROOT

williamstein · 2011-08-23T20:47:33Z

comment:13

Here's how I'm testing this:

wstein@sage:/tmp/wstein/sage-4.7.2.alpha1-sage.math.washington.edu-x86_64-Linux$
$ du -sh .
2.4G    .
$ du -s .
2468404 .

$ ./sage -hg import https://github.com/sagemath/sage-prod/files/10652123/trac_10801-root_repo.patch.gz
applying https://github.com/sagemath/sage-prod/files/10652123/trac_10801-root_repo.patch.gz

$ cd local/bin/; ../../sage -hg import https://github.com/sagemath/sage-prod/files/10652124/trac_10801-local_bin_repo.patch.gz
applying https://github.com/sagemath/sage-prod/files/10652124/trac_10801-local_bin_repo.patch.gz

$ time make micro_release
real    0m38.227s
user    0m2.370s
sys     0m5.140s

$ du -sh .
1.5G    .
$ du -s .
1547480 .

$ ./sage
# see it works

$ make ptestlong
# observe what happens (I will report back in a follow comment about this).

williamstein · 2011-08-23T21:29:31Z

comment:14

ptestlong passes. W00t.

kini · 2011-08-24T04:56:52Z

comment:15

Some thoughts:

In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.
The *_with_condition() functions should not talk about extensions in their docstrings.
What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

Going a bit farther, considering the target users will never develop or upgrade,

Should we delete mercurial repos?
Should we delete SPKGs?

williamstein · 2011-08-24T05:08:50Z

comment:16

Replying to @kini:

Some thoughts:

In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.

LATER: This could be added, but can we do it later in a future ticket. It would be good if the first version of this patch is quite safe. Hey, it already saves 1 GB as is!

The *_with_condition() functions should not talk about extensions in their docstrings.

OK: I'll fix the patch right now.

What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

NO: Pointless, and scary in that it would break functionality, which is probably a bad idea. Good point about the size. I must have been testing something with some optional fonts installed.

Going a bit farther, considering the target users will never develop or upgrade,

Should we delete mercurial repos?

FUTURE: Yes, but in a future version of this?

Should we delete SPKGs?

FUTURE: It's safer to put a small string in each, to avoid confusion. This happens automatically when you build a binary with "sage -bdist". Anyway, this is something that could be added later.

benjaminfjones · 2011-08-24T19:12:42Z

Reviewer: Benjamin Jones

benjaminfjones · 2011-08-24T19:12:42Z

comment:17

Tested the patches on sage.math and fresh build of Sage-4.7.1:

bjones@sage:/scratch/bjones/sage-4.7.1$ pwd
/scratch/bjones/sage-4.7.1
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.5G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2554916	.

bjones@sage:/scratch/bjones/sage-4.7.1$ time make micro_release
...
real	0m12.064s
user	0m2.560s
sys	0m7.600s
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.0G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2033236	.

So it saved 0.5 GB which is not the 1.0 GB observed by William above. Did you remove stuff by hand in that session that the script doesn't touch? Or maybe because I built the install from source instead of using a binary distribution?

I'm running make ptestlong now, will report back if anything fails.

benjaminfjones · 2011-08-24T19:48:50Z

comment:18

After producing the micro release, make ptestlong had one failure which disappeared when I doctested by hand:

The following tests failed:

        sage -t  -long -force_lib devel/sage/sage/tests/startup.py # 1 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1468.7 seconds
make: *** [ptestlong] Error 128

Subsequently by hand:

bjones@sage:/scratch/bjones/sage-4.7.1$ sage -t  -long -force_lib devel/sage/sage/tests/startup.py
sage -t -long -force_lib "devel/sage/sage/tests/startup.py" 
         [12.1 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 12.1 seconds

williamstein · 2011-08-24T21:34:16Z

comment:19

Ben:

Regarding the size savings, I started with a binary... so we may see a difference.
Regarding the startup time failure, this has nothing to do with my patch. It routinely fails on sage.math. It's a test that it isn't taking more than 2 seconds on sage.math. I put it in back when we got the startup time down to solidly be < 1 second on sage.math, since I didn't want people to screw that up. They did though.

williamstein · 2011-08-24T23:41:47Z

Changed keywords from none to sd32

kini · 2011-08-25T00:35:33Z

Changed reviewer from Benjamin Jones to Benjamin Jones, Keshav Kini

kini · 2011-08-25T00:35:33Z

comment:21

William: OK, sounds good. I'll make a ticket. Sounds like we're good to go here, no? Positive review from me.

benjaminfjones · 2011-08-25T00:39:00Z

comment:22

OK, I checked on OSX 10.6.8 binary install and I get results simular to William's. Looks good here too. Positive review from me as well.

kini · 2011-08-25T02:41:27Z

comment:23

New ticket is #11743.

williamstein · 2011-08-25T05:24:50Z

comment:24

Martin Albrecht pointed out that deleting .a files could make it so users can't compile certain Cython programs, maybe. I'm not sure if we should care though, given the goal of this ticket ('never develop or upgrade').

kini · 2011-08-25T05:25:46Z

comment:25

Hm. If that includes %cython cells, I don't think that counts as "developing".

williamstein · 2011-08-25T05:34:15Z

comment:26

Yes, I'm concerned. I'll simply comment out the line that deletes the .a files, and repost the patch.

williamstein · 2011-08-25T05:36:41Z

replacing the patch by one that doesn't delete .a files.

kini · 2011-08-25T05:53:56Z

comment:27

Attachment: trac_10801-local_bin_repo.patch.gz

Fair enough. Sadly this reduces our space savings by more than 200 MB.

(I'm putting myself as an author on this ticket because I seem to be listed as one the patch...)

kini · 2011-08-25T05:53:56Z

Author: William Stein, Keshav Kini

nexttime · 2011-09-12T18:39:51Z

Merged: sage-4.7.2.alpha3

williamstein added this to the sage-4.7.2 milestone Feb 19, 2011

williamstein added c: packages: standard labels Feb 19, 2011

This comment has been minimized.

Sign in to view

williamstein added the s: needs review label Aug 23, 2011

kini added s: positive review and removed s: needs review labels Aug 25, 2011

This comment has been minimized.

Sign in to view

nexttime mannequin removed the s: positive review label Sep 12, 2011

nexttime mannequin closed this as completed Sep 12, 2011

kini mentioned this issue May 16, 2013

Improve make micro_release #11743

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade #10801

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade #10801

williamstein commented Feb 19, 2011

jasongrout commented Mar 17, 2011

kini commented Mar 22, 2011

This comment has been minimized.

sagetrac-pi mannequin commented Mar 23, 2011

jhpalmieri commented Mar 23, 2011

kini commented Mar 25, 2011

sagetrac-mhampton mannequin commented May 28, 2011

kini commented Jun 14, 2011

nexttime mannequin commented Aug 22, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

kini commented Aug 24, 2011

williamstein commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

williamstein commented Aug 24, 2011

williamstein commented Aug 24, 2011

kini commented Aug 25, 2011

kini commented Aug 25, 2011

benjaminfjones commented Aug 25, 2011

kini commented Aug 25, 2011

williamstein commented Aug 25, 2011

kini commented Aug 25, 2011

williamstein commented Aug 25, 2011

williamstein commented Aug 25, 2011

kini commented Aug 25, 2011

kini commented Aug 25, 2011

This comment has been minimized.

nexttime mannequin commented Sep 12, 2011

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade #10801

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade #10801

Comments

williamstein commented Feb 19, 2011

jasongrout commented Mar 17, 2011

kini commented Mar 22, 2011

This comment has been minimized.

sagetrac-pi mannequin commented Mar 23, 2011

jhpalmieri commented Mar 23, 2011

kini commented Mar 25, 2011

sagetrac-mhampton mannequin commented May 28, 2011

kini commented Jun 14, 2011

nexttime mannequin commented Aug 22, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

williamstein commented Aug 23, 2011

kini commented Aug 24, 2011

williamstein commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

benjaminfjones commented Aug 24, 2011

williamstein commented Aug 24, 2011

williamstein commented Aug 24, 2011

kini commented Aug 25, 2011

kini commented Aug 25, 2011

benjaminfjones commented Aug 25, 2011

kini commented Aug 25, 2011

williamstein commented Aug 25, 2011

kini commented Aug 25, 2011

williamstein commented Aug 25, 2011

williamstein commented Aug 25, 2011

kini commented Aug 25, 2011

kini commented Aug 25, 2011

This comment has been minimized.

nexttime mannequin commented Sep 12, 2011