Make spkg docs be built and installed after the full build

[jasongrout]

Following on some concerns brought up about dependencies in building spkg docs, as implemented in #10823, here is a ticket for reimplementing #10823 so that docs are built after the full Sage build. In this way, spkg docs are built when the normal Sage docs are built, and we don't have any worries about having python or sphinx or other dependencies installed.

From Dan's message to sage-devel:

How about, in addition to spkg-install and spkg-check, we have spkg-install-docs? We would move the logic that checks the SAGE_SPKG_INSTALL_DOCS variable to the build/install script, which would run the script if the env var is set. This would also allow the build process to determine if a spkg has documentation that can be built, using something roughly like tar jtf foo.spkg | grep spkg-install-docs

As a data point, doing that for all spkgs in 4.7.alpha4 (except the Sage library, which we handle separately) takes about 70 seconds on a mildly-loaded sagenb.kaist.ac.kr, which is an 8-core Xeon machine.

See http://groups.google.com/group/sage-devel/browse_thread/thread/d46ddccff06d8670 and various tickets associated with #10823

Here are a few updated spkgs:

• numpy: Make numpy support SAGE_SPKG_INSTALL_DOCS #10826
• cython: Make cython support SAGE_SPKG_INSTALL_DOCS #10827
• matplotlib: Make matplotlib support SAGE_SPKG_INSTALL_DOCS #10828
• mpmath: Make mpmath support SAGE_SPKG_INSTALL_DOCS #10829
• pari: Make pari support SAGE_SPKG_INSTALL_DOCS #10830
• gap: Update Gap to use #11197 for building spkg docs #11198
• python: Make python support SAGE_SPKG_INSTALL_DOCS #10831
• sagetex: Make SageTeX support SAGE_SPKG_INSTALL_DOCS #10908

---

Apply either

• attachment: trac_11197-spkg-install-docs.patch (builds docs if environment variable SAGE_SPKG_INSTALL_DOCS is "yes"), or

• attachment: trac_11197-spkg-install-docs.v2.patch (adds target "spkg-doc" to Makefile)

CC: @dandrake

Component: build

Keywords: sd32

Issue created by migration from https://trac.sagemath.org/ticket/11197

[jhpalmieri]

comment:6

Here's a patch for the Sage root repo (the file spkg/install). This should perhaps be considered a first draft, because there are a few issues/questions.

• Right now, it records a log for the docbuilding in spkg/logs/numpy-1.5.1.p0-docs.log, for example, instead of just appending to the already existing spkg/logs/numpy-1.5.1.p0.log. It's easy enough to switch it around. Opinions?

• Right now, it stores a placeholder file in spkg/installed if the docs seem to have been built correctly. (Since after building each spkg, we delete the directory where it was built (spkg/build/numpy-...), we can't just rely on 'make' recognizing that it's already been installed, I think.) Is this the right way to do it?

A few things which I think belong on other tickets or which I don't know how to solve:

• With the approach here, running "sage -i ..." or "sage -f ..." won't install the docs. Should we add something to the sage-spkg script which tries to build the docs if SAGE_SPKG_INSTALL_DOCS is "yes", but fails gracefully if it can't? Or only build the docs in sage-spkg if we can tell that the Sage library has already been installed, or some other criterion like this? This may end up with some code duplication, or separating out the doc-building stuff into a separate script (and then gets us entangled in the base repo vs. scripts repo debate...).

• I noticed that the most recent cython spkg (the one in 4.7.alpha4) has files for building the docs, at least some of which weren't added to the Mercurial repository. I haven't checked the other spkgs for this sort of issue, but someone should...

[jhpalmieri]

Author: John Palmieri

[jhpalmieri]

root repo

[jasongrout]

comment:7

Attachment: trac_11197-spkg-install-docs.patch.gz

Replying to @jhpalmieri:

Here's a patch for the Sage root repo (the file spkg/install). This should perhaps be considered a first draft, because there are a few issues/questions. - Right now, it records a log for the docbuilding in spkg/logs/numpy-1.5.1.p0-docs.log, for example, instead of just appending to the already existing spkg/logs/numpy-1.5.1.p0.log. It's easy enough to switch it around. Opinions?

• Right now, it stores a placeholder file in spkg/installed if the docs seem to have been built correctly. (Since after building each spkg, we delete the directory where it was built (spkg/build/numpy-...), we can't just rely on 'make' recognizing that it's already been installed, I think.) Is this the right way to do it? A few things which I think belong on other tickets or which I don't know how to solve:

To me, it seems okay to think about the docs install as a separate thing from the spkg install, so it makes sense to me to have a separate log file and separate placeholder file. I don't know much about this part of the build process, though, so my +1 doesn't count for much.

• With the approach here, running "sage -i ..." or "sage -f ..." won't install the docs. Should we add something to the sage-spkg script which tries to build the docs if SAGE_SPKG_INSTALL_DOCS is "yes", but fails gracefully if it can't? Or only build the docs in sage-spkg if we can tell that the Sage library has already been installed, or some other criterion like this? This may end up with some code duplication, or separating out the doc-building stuff into a separate script (and then gets us entangled in the base repo vs. scripts repo debate...).

Hmm...thinking about this one still...

• I noticed that the most recent cython spkg (the one in 4.7.alpha4) has files for building the docs, at least some of which weren't added to the Mercurial repository. I haven't checked the other spkgs for this sort of issue, but someone should...

Are you talking about the docs directory containing the documentation? That directory is part of the upstream source, so shouldn't be in the spkg repository. However, as of a week or two ago, that docs directory has been merged into the main Cython source tree, so future versions of the spkg will not have that docs directory.

[jhpalmieri]

comment:8

Replying to @jasongrout:

Are you talking about the docs directory containing the documentation?

I mean that if I unpack cython-0.14.1.p2.spkg, cd into the directory and type "hg status", I see

? SPKG.txt.orig
? doc/.hgignore
? doc/Makefile
? doc/README
? doc/TODO
? doc/_static/cython-logo-light.png
? doc/_static/cythonlogo.png
? doc/_static/favicon.ico
? doc/_templates/layout.html
? doc/conf.py
? doc/examples/tutorial/fib1/fib.pyx
? doc/examples/tutorial/fib1/setup.py
? doc/examples/tutorial/great_circle/c1.pyx
? doc/examples/tutorial/great_circle/c2.pyx
? doc/examples/tutorial/great_circle/p1.py
? doc/examples/tutorial/primes/primes.py
? doc/examples/tutorial/primes/primes.pyx
? doc/examples/tutorial/primes/setup.py
? doc/index.rst
? doc/sphinxext/cython_highlighting.py
? doc/sphinxext/ipython_console_highlighting.py
? doc/todo.txt
? doc/welcome.rst

So I guess those need to be added to .hginore, or the whole doc directory needs to added, or if everything in there has since been moved to src, we only need to get rid of SPKG.txt.orig.

[jasongrout]

comment:9

Good point; the doc directory should be added to .hgignore. And SPKG.txt.orig should be deleted. Sorry; that seems pretty sloppy of me.

[jasongrout]

comment:10

I've noted these changes on #10827.

[jhpalmieri]

comment:11

Replying to @jasongrout:

Good point; the doc directory should be added to .hgignore. And SPKG.txt.orig should be deleted. Sorry; that seems pretty sloppy of me.

I've made those kinds of mistakes lots of times myself...

[vbraun]

comment:12

If we are moving to a two-step binary / documentation build process, how about we remove the SAGE_SPKG_INSTALL_DOCS environment variable and, instead, provide a "doc" target in the main Makefile. So you can just "make doc", thats much easier to discover than a cryptic environment variable.

[jasongrout]

comment:13

Replying to @vbraun:

If we are moving to a two-step binary / documentation build process, how about we remove the SAGE_SPKG_INSTALL_DOCS environment variable and, instead, provide a "doc" target in the main Makefile. So you can just "make doc", thats much easier to discover than a cryptic environment variable.

+1

[jasongrout]

comment:14

Maybe we should clearly distinguish between sage's documentation and spkg documentation. I'd expect make doc to just build Sage's documentation. Maybe we could have two targets: make doc (build and install Sage docs) and make spkg-doc (build and install any spkg docs).

[jhpalmieri]

comment:15

Here's a new patch. This does the Makefile approach. I think I like the Makefile approach, but I'm not positive, so I'm marking this as "needs info". Please provide opinions about the two approaches and also about the specific patches.

[jhpalmieri]

root repo; implement "make spkg-doc" from SAGE_ROOT

[williamstein]

comment:16

Attachment: trac_11197-spkg-install-docs.v2.patch.gz

[williamstein]

Changed keywords from none to sd32

[nexttime]

comment:17

Hmmm, one problem I see is that a couple of spkgs would need a (re)configure before e.g. make install-docs etc. would work.

Also, many packages provide [alternate] documentation in various formats (man pages, texinfo files, HTML, dvi / PDF etc.), some of which require tools (i.e., prerequisites either provided by the system or Sage), and some even some small build process just to prepare the sources for other tools. So it isn't always clear what would be installed [in addition]. Re-unpacking and re-configuring an spkg just to install docs is IMHO also a bit odd, just like keeping the build directories until eventually some tools are available after the normal build has finished.

The easiest way would be to simply include some pre-built versions of documentation requiring external tools, and just copy / install them if SAGE_SPKG_INSTALL_DOCS=yes or an spkg's spkg-install-docs is run. This could at least in some cases increase an spkg's size significantly though.

[nexttime]

comment:18

We could also support some kind of SAGE_SPKG_INSTALL_DOCS=no (and =later) in spkg-install, but that still doesn't tell which kind of documentation.

(I personally would for example always install man pages by default, if available.)

No matter if we build / install any documents for some spkg (or whether they are available at all in the spkg itself), we could create HTML pages with links to upstream [documentation] if appropriate.

Same could be done for licenses.

[jhpalmieri]

comment:19

Right now, I'm more in favor of adding a target to the Makefile, rather than the environment variable approach.

Replying to @nexttime:

Hmmm, one problem I see is that a couple of spkgs would need a (re)configure before e.g. make install-docs etc. would work.

Well, this would be taken care of by the doc install script for the particular spkg. The whole point of this ticket is to install the docs separately from the full build, so I'm not sure I understand your concerns. If you want to build the docs during the build, use the approach in #10823, and the spkg-install file can take care of the proper configurations to do things efficiently. (So maybe some spkgs would behave differently depending on whether you ran make spkg-doc compared to make with SAGE_SPKG_INSTALL_DOCS=yes: in the first case, a special script spkg-install-docs would be run, while in the first case, the code would be in spkg-install. Or is that too complicated?)

Also, many packages provide [alternate] documentation in various formats (man pages, texinfo files, HTML, dvi / PDF etc.), some of which require tools (i.e., prerequisites either provided by the system or Sage), and some even some small build process just to prepare the sources for other tools. So it isn't always clear what would be installed [in addition]. Re-unpacking and re-configuring an spkg just to install docs is IMHO also a bit odd, just like keeping the build directories until eventually some tools are available after the normal build has finished.

There could be several possible targets: make spkg-doc which would build whatever the spkg maintainers thought was a sensible default, but also make spkg-doc-html, make spkg-doc-man, etc. Perhaps the script spkg/install-spkg-docs should pass an argument ("html", "pdf", "default", etc.) to each individual spkg-install-docs script.

The easiest way would be to simply include some pre-built versions of documentation requiring external tools, and just copy / install them if SAGE_SPKG_INSTALL_DOCS=yes or an spkg's spkg-install-docs is run. This could at least in some cases increase an spkg's size significantly though.

I don't think the size increase would be acceptable.

Replying to @nexttime:

We could also support some kind of SAGE_SPKG_INSTALL_DOCS=no (and =later) in spkg-install, but that still doesn't tell which kind of documentation.

or SAGE_SPKG_INSTALL_DOCS=html

No matter if we build / install any documents for some spkg (or whether they are available at all in the spkg itself), we could create HTML pages with links to upstream [documentation] if appropriate.

Sure, that sounds like a good idea. Probably for another ticket, once we get this framework set up. At least, setting up these html pages would seem to require another set of ideas and therefore add another level of complications.

[jdemeyer]

Replying to @jasongrout:

Following on some concerns brought up about dependencies in building spkg docs, as implemented in #10823

What are these "concerns"? Can these "concerns" be fixed in a simpler way than adding more complexities to the Sage build procedure?

[jdemeyer]

comment:26

I think it needs to be clarified why this ticket is needed.

[jhpalmieri]

comment:27

I think the situation is:

• we want to be able to install documentation for any of the various packages that provide it, but
• building the documentation may have different prerequisites (e.g., Sphinx) than building the package itself.
  It sounds like the plan in this ticket was to build documentation for the packages at the same time as building the regular Sage documentation.

[mkoeppe]

comment:28

outdated, should close

[mkoeppe]

Changed author from John Palmieri to none