efficient 2-stage parallel builder?

[dimpase]

Due to slowness of Sphinx in buidling of the reference manual of SageMath several years ago we implemented a paralell 2-stage customised Sphinx builder, details may be found in https://trac.sagemath.org/ticket/20080

In a nutshell, it runs Sphinx once to generate intersphinx "object.inv" inventory files and pickle files; these are used in a parallelised 2nd run, where independent builders process subcollections of documents independently, using intersphinx extension to make sure cross-references are maintained; finally, the global index is built.

This gives a many-fold speedup on multi-core CPUs (to know by how much exactly, one would need to create a "standard" Sphinx setup for SageMath, and time it against our builder - but certainly if you have, say, 8 CPUs cores available, the speedup is about 5 to 6-fold, if compared with just using 1 CPU).

---

As Sphinx evolves, our setup gets harder to maintain (it does not help that several people who originally implemented it drifted away from the project), so we would like to streamline and upgrade our setup, using as much as is available from Sphinx upstream.
Perhaps upstream might be interested in having a similar parallel builder based on this idea, then we could just use it, and do not maintain our builder.

[tk0miya]

A while ago, I tried to build the document of sagemath via the original sphinx-build. But I can't find the way to build it. I need to know the structure of the document. It seems https://github.com/sagemath/sage/tree/develop/src/doc contains many sphinx project. @dimpase Is there any document or article for the structure of docs? I'd like to measure how slow the standard build at first.

[dimpase]

By far the biggest document there is reference. The build time of it dominates the rest.
Thus I think it would be the interesting enough to measure on this document alone.

Most of it consists of docstrings in Sagemath Python/Cython library, i.e. in https://github.com/sagemath/sage/tree/develop/src/sage/

[dimpase]

more precisely, the document I mean is here: https://github.com/sagemath/sage/tree/develop/src/doc/en/reference

[tk0miya]

Now I'm trying to create an environment to build the document... it seems I need to install many dependencies to build the docs...

[dimpase]

This is correct - you basically might need a working copy of Sagemath. Do tell me if you have a problem with it; basically,
building Sage is well-supported on stable versions of Linux, such as Debian, Ubuntu, Fedora. It's OK on macOS, too. Windows is very painful though.

One way or another, it might be quicker to get Sagemath on conda.

[tk0miya]

It would be very helpful to me if there are well-prepared docker image (or Dockefile) to build the environment. I don't have experience of conda. So this is very tough for me.

[dimpase]

Please try https://hub.docker.com/r/sagemath/sagemath/

Hopefully @mkoeppe and @saraedum can tell you more about Sagemath on docker, if needed.

[tk0miya]

Thank you for the info. I'll try it this weekend.

[dimpase]

I wonder if you had a chance to try this out. Our doc builder keeps getting us troubles.

[tk0miya]

Sorry for the late response. I just tried the docker image now. But I still don't find the way to build the doc of sagemath yet. It's very helpful if somebody let me know the way to build the document of sagemath on my local.

[dimpase]

@mkoeppe is there a way to quickly set up a docbuilding environment for Sage in docker?