pip-installable packages sagemath-doc-src, sagemath-doc-inventory, sagemath-doc-html, sagemath-doc-pdf

[mkoeppe]

Similar to #29411 + #29950, which made sagelib a pip-installable distribution (which is then installed by the sage-the-distribution script package sagelib), we change the build/installation scheme of the sage documentation.

We create pip-installable packages sagemath-doc-html and sagemath-doc-pdf. By providing wheels on pypi, users can get access to an offline copy of the documentation without having to build it themselves.

We create a package sagemath-doc-inventory, which is in charge of building and installing the inventory files.
It is a build dependency of sagemath-doc-html and sagemath-doc-pdf.

All of these packages depend on sagemath-doc-src. It has a build-system dependency on sagelib. An sdist of it consists of SAGE_SRC/doc. A wheel of sagemath-doc-src (and thus an installation) contains a copy of the sources (including the auto-generated rst files) in share/doc/sage/src/ and the doctrees in share/doc/sage/doctrees.
The incremental build features of the docbuild will only be active when the package sagemath-doc-src is installed directly using setup.py install, setup.py develop or pip install --editable. In these cases, the doctrees (including environment pickes) are kept in the source directory.
pip wheel will always make a from-scratch build of the documentation, eliminating incremental build errors.

The script sage-grepdoc will be provided by sagemath-doc-html.

Follow-up tickets:

• When installed using pip, the packages provide nbextensions that provide offline help in a Jupyter notebook. This is useful in particular with local notebooks connecting to remote sage kernels (see Meta-ticket: Use system Jupyter notebook / JupyterLab #30306 (3)).
  • https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Distributing%20Jupyter%20Extensions%20as%20Python%20Packages.html
• Or alternatively as a JupyterLab extension
  • https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html

See also:

• Split sage_setup.docbuild out to a separate distribution sage_docbuild #30010: Split sage_setup.docbuild out to a separate package
• build/make/Makefile.in: Rename make targets SPKG-clean to SPKG-uninstall #29097: build/make/Makefile.in: Rename make targets SPKG-clean to SPKG-uninstall
• add intersphinx mapping for SciPy #29231: Add intersphinx mapping for SciPy -- with discussion about docbuild and inventory files

Depends on #33852

CC: @isuruf @kiwifb @antonio-rojas @timokau @nbruin @dimpase @jhpalmieri @mwageringel @zlscherr

Component: build

Keywords: sd111

Issue created by migration from https://trac.sagemath.org/ticket/29868

[mkoeppe]

Changed keywords from none to sd111

[mkoeppe]

Dependencies: #31356

[mkoeppe]

comment:14

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

comment:15

Input and help from docbuild experts is very welcome.

[mkoeppe]

comment:16

Some questions:

• Who generates files such as src/doc/en/reference/monoids/sage/monoids/free_monoid.rst, and should they be part of what sagemath-doc-src installs (and thus ships with the wheel)?

• Is there a way to only generate the doctrees but not build HTML or PDF?

• Do the HTML and PDF builds need to be able to run the Sage library, or can they build these formats if given only the doctrees?

[kiwifb]

comment:17

Replying to @mkoeppe:

Some questions:

• Is there a way to only generate the doctrees but not build HTML or PDF?

I do believe there is a sphinx option for doing just that.

• Do the HTML and PDF builds need to be able to run the Sage library, or can they build these formats if given only the doctrees?

As far as I know they need to be able to run the sage library. In particular all graphs/plots are generated during the documentation build.

[kiwifb]

comment:18

Replying to @kiwifb:

Replying to @mkoeppe:

Some questions:

• Is there a way to only generate the doctrees but not build HTML or PDF?

I do believe there is a sphinx option for doing just that.

I cannot find such an option anymore. If it existed. There is an option to say where the doctrees should be stored and found, but nothing on producing it in isolation.

[jhpalmieri]

comment:19

Well, ./sage --docbuild all inventory builds the inventory files and the doctree files, but not html or pdf.

[jhpalmieri]

comment:20

Replying to @mkoeppe:

Some questions:

• Who generates files such as src/doc/en/reference/monoids/sage/monoids/free_monoid.rst, and should they be part of what sagemath-doc-src installs (and thus ships with the wheel)?

The autodoc procedure does this, with sage/monoids/free_monoid.py as input. (Edit: or maybe just src/doc/en/reference/monoids/index.rst as input?) I'm not sure precisely how this is done, but the file sage_docbuild/ext/sage_autodoc.py is certainly relevant. (This is Sage's fork of the Sphinx autodoc module.)

[jhpalmieri]

comment:21

No, I'm wrong about that. It's done in sage_docbuild/__init__.py, in the write_auto_rest_file method.

[mkoeppe]

Changed dependencies from #31356 to #31356, #32713

[mkoeppe]

comment:25

According to #33064 comment:19, there is no need to install doctrees in SAGE_LOCAL, not even for introspection (sage.misc.sphinxify)

[mkoeppe]

Changed dependencies from #31356, #32713 to #33852

[kwankyu]

• Who generates files such as src/doc/en/reference/monoids/sage/monoids/free_monoid.rst,

The autodoc procedure does this, with sage/monoids/free_monoid.py as input. (Edit: or maybe just src/doc/en/reference/monoids/index.rst as input?) I'm not sure precisely how this is done, but the file sage_docbuild/ext/sage_autodoc.py is certainly relevant. (This is Sage's fork of the Sphinx autodoc module.)

No. Reference doc builder (part of Sage, in src/sage_docbuild) is responsible for that, as tangentially explained in #36692 (comment). Then autodoc (extension to Sphinx) is responsible to render the generated rst files. We are shipping sage-specific version of autodoc, but that should be replaced eventually with official autodoc provided by Sphinx

and should they be part of what sagemath-doc-src installs (and thus ships with the wheel)?

Yes, if it intends to include all rst files from which htmls (or pdfs) are generated by Sphinx.

[kwankyu]

comment:21
No, I'm wrong about that. It's done in sage_docbuild/__init__.py, in the write_auto_rest_file method.

Now the builders are in src/sage_docbuild/builders.py.