Make matplotlib support SAGE_SPKG_INSTALL_DOCS

[jasongrout]

This ticket implements #10823 for matplotlib. New spkg at  http://sage.math.washington.edu/home/jason/spkg-docs/matplotlib-1.0.1.p0.spkg (depends on #10588)

CC: @sagetrac-ryan

Component: packages: standard

Keywords: sd32

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/10828

[sagetrac-drkirkby]

comment:2

A few points.

• The SPKG.txt file does not have the number of this ticket in it - only environment variable SAGE_SPKG_INSTALL_DOCS to build and install spkg docs #10823
• On environment variable SAGE_SPKG_INSTALL_DOCS to build and install spkg docs #10823 the patch which updates the Developers Guide shows the documentation being copied not moved, which is better, as it leaves the build directory untouched.
• It would make reviewing much easier, and be a better historical record, if there was a Mercurial patch of the changes attached to the ticket.

It looks like similar issues arise with the other updated .spkg files that you produced which are related to #10823

[jasongrout]

Attachment: 10828.patch.gz

FYI Only---already applied to spkg.

[jasongrout]

FYI Only---already applied to spkg.

[jasongrout]

comment:3

Attachment: 10828-1.patch.gz

[jasongrout]

comment:4

I also found that the last matplotlib spkg from #10588 had a dirty src/ directory, instead a pristine copy of the sources. That was probably all my fault, as I was the reviewer, and the contributor was a new contributor. I'm CCing Ryan to make sure he knows about it, just as an FYI.

The new matplotlib spkg above replaces the src/ directory with a pristine copy of the 1.0.1 matplotlib sources, which removes the build directory, some of the files that are generated in compilation, and trims the final spkg from 19M to 11M.

[jasongrout]

comment:6

New spkg is up, ready for review.

[sagetrac-drkirkby]

Reviewer: David Kirkby

[sagetrac-drkirkby]

comment:7

This generates the HTML and puts it where expected. However, the simple

cp -r doc/* $SAGE_ROOT/local/share/doc/PACKAGE_NAME/

is copying a lot of unnecessary files. There are 29 files with the extension .py, 511 files with the extension .pyc, and possibly some other unnecessary files that are being copied to $SAGE_ROOT/local/share/doc/PACKAGE_NAME/ There are 945 pdf files there too - I'm not quite sure how they got there. The total size of the files copied to $SAGE_ROOT/local/share/doc/PACKAGE_NAME/ is 115 MB

I think we need to find the unnecessary files and remove them. Something like:

find "$SAGE_ROOT/local/share/doc/matplotlib" -name '*.pyc' -exec rm -f {} \;
find "$SAGE_ROOT/local/share/doc/matplotlib" -name '*.py'  -exec rm -f {} \;

would be a start, but there's probably more that can be done.

Should we generate all the PDFs if pdftex is present? (In which case there will be several hundred more), or delete all the PDFs??

I can't see any point in having the .py and .pyc files, but I'm not so sure about the .pdf files.

I'm not going to copy the same comments to all the other packages - lets get one right first, before going on to the others.

I think #10823 should be updated to advise developers to remove unnecessary files if they are present, otherwise we are just going to be copying a lot of unneeded junk, and filling up disks unnecessarily.

Dave

[jasongrout]

comment:8

In matplotlib, it's not "cp -r doc/* $SAGE_ROOT/local/share/doc/PACKAGE_NAME/", but rather, it's copying the results of the build directory.

The documentation for matplotlib includes a lot of example figures (generated in both png and pdf formats) as well as the .py files that generated those examples, so I think those files are legitimate. I'm not sure about the .pyc files---I'll check those.

[jasongrout]

comment:9

This spkg (as well as the other spkgs from #10823) are copying the directories containing the actual built documentation (e.g., build/html), not the entire documentation source tree. As such, only the actual documentation files should be there. In matplotlib's case, lots of pdf and png files are generated as example images (this is a plotting package, after all) and the generating .py files are also included in the docs.

I see some pyc files that probably should not be in the build directory of the docs.
I feel like these should be reported upstream and dealt with on another ticket. I'd really rather use the default upstream doc build system, and report problems upstream.

Please remember that SAGE_SPKG_INSTALL_DOCS is an optional feature, and the people that invoke it are making a decision that the documentation is worth the disk space.

[jasongrout]

comment:10

I've posted a message to the matplotlib list about the .pyc files, as well as a possible redundancy in image files.

[sagetrac-drkirkby]

comment:11

OK, in light of what you have said, I'll put this to positive review.

[jasongrout]

comment:12

An upstream developer has proposed some changes to matplotlib that will cut down on the redundant files. See the mailing list thread here: http://thread.gmane.org/gmane.comp.python.matplotlib.general/26576

[jdemeyer]

Merged: sage-4.7.alpha1

[jdemeyer]

comment:14

matplotlib doesn't build from scratch using SAGE_SPKG_INSTALL_DOCS, see also #10826.

[jdemeyer]

Changed merged from sage-4.7.alpha1 to none

[jasongrout]

comment:16

See #11197 for building docs after Sage is built.

[williamstein]

Changed keywords from none to sd32

[mkoeppe]

Changed reviewer from David Kirkby to none

[mkoeppe]

Changed author from Jason Grout to none

[mkoeppe]

comment:22

outdated, should close

[dimpase]

Reviewer: Dima Pasechnik