Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkgs/sagemath-doc-{html,pdf} #36730

Draft
wants to merge 51 commits into
base: develop
Choose a base branch
from
Draft

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Nov 17, 2023

We create pip-installable packages that provide the Sage documentation. This is a step toward #29868, which has the full design.

We want to install the HTML documentation in share/doc/sage/html. So we need a build system that supports such "non-package data".

For synergy with #34630, we use meson-python - with a trivial pkgs/sagemath-doc-html/meson.build file that just runs make with the existing Makefile.
The actual build happens using ninja.

In the current implementation:

  • if SAGE_EDITABLE is set to yes (default), this uses pip install -e, which does the incremental docbuild (in pkgs/sagemath-doc-*/build) and creates symlinks in $SAGE_LOCAL/share/doc/sage/ that point there.
  • if SAGE_WHEELS is set to yes, this builds a wheel from scratch, which contains the HTML documentation as non-package data files. If SAGE_EDITABLE is not set, then it installs the wheel using --prefix=$SAGE_LOCAL so that the HTML documentation appears in the familiar location.

We do the same with PDF in sagemath-doc-pdf.

GH Actions workflows:

  • On release tags, the workflow dist.yml
    The workflow (run on release tags) builds the sdists and uploads them to PyPI, along with the other pip-installable packages.
  • On release tags, doc-build.yml builds the live doc (HTML + PDF) by going through the wheels and uploads to PyPI if the credentials are present.

To test:

  • make sagemath_doc_html and make sagemath_doc_pdf should continue to work
  • make SAGE_WHEELS=yes sagemath_doc_html and make SAGE_WHEELS=yes sagemath_doc_pdf should both work and create (architecture-independent) wheels in prefix/var/lib/sage/wheels/
  • make sagemath_doc_html-sdist and make sagemath_doc_pdf-sdist should both create .tar.gz files in upstream/
  • these sdists declare the Sage library as a build dependency and should be buildable when disabling build isolation in a virtual environment with an installed Sage library, for example by unpacking the sdist and then using ./sage -sh -c 'python3 -m build --no-isolation --skip-dependency-check --wheel .'
  • This should still work if you move away SAGE_ROOT/src/doc; and building should not touch prefix/share/doc at all. The documentation source comes from the sdist, and building the wheel should only write to a temporary directory, from which it creates the wheel.

In follow-up PRs:

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe mkoeppe self-assigned this Nov 17, 2023
@mkoeppe mkoeppe marked this pull request as draft November 17, 2023 04:28
@mkoeppe mkoeppe changed the title pkgs/sagemath-doc-html pkgs/sagemath-doc-{html,pdf} Nov 18, 2023
@mkoeppe mkoeppe marked this pull request as ready for review November 18, 2023 21:58
@mkoeppe mkoeppe requested a review from kwankyu November 19, 2023 01:15
Copy link
Contributor

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments concerning the meson file.

pkgs/sagemath-doc-html/meson.build Outdated Show resolved Hide resolved
pkgs/sagemath-doc-html/meson.build Show resolved Hide resolved
pkgs/sagemath-doc-html/meson.build Show resolved Hide resolved
pkgs/sagemath-doc-html/meson.build Outdated Show resolved Hide resolved
pkgs/sagemath-doc-html/meson.build Show resolved Hide resolved
@kwankyu
Copy link
Collaborator

kwankyu commented Nov 22, 2023

 export SAGE_WHEELS=yes
./bootstrap && ./configure && make -j4 

successfully finished, though

[sagemath_doc_html-10.2.rc3] [10976/10976] /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-doc-html/.mesonpy-zdxa0nog/html/tr/a_tour_of_sage/_static/styles/furo.css.map
[sagemath_doc_html-10.2.rc3] Successfully built sagemath_doc_html-10.2rc3-py3-none-any.whl
[sagemath_doc_html-10.2.rc3] WARNING: Skipping sagemath_doc_html as it is not installed.
[sagemath_doc_html-10.2.rc3] Using pip 23.3.1 from /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/pip (python 3.11)
[sagemath_doc_html-10.2.rc3] Looking in links: /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/wheels
[sagemath_doc_html-10.2.rc3] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/wheels/sagemath_doc_html-10.2rc3-py3-none-any.whl
[sagemath_doc_html-10.2.rc3] Installing collected packages: sagemath-doc-html
[sagemath_doc_html-10.2.rc3] Successfully installed sagemath-doc-html-10.2rc3

This is rc4. Why is it sagemath_doc_html-10.2.rc3?

Then I ran make sagemath_doc_pdf. I got

[sagemath_doc_pdf-10.2.rc3] The Meson build system
[sagemath_doc_pdf-10.2.rc3] Version: 1.2.1
[sagemath_doc_pdf-10.2.rc3] Source dir: /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-doc-pdf
[sagemath_doc_pdf-10.2.rc3] Build dir: /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-doc-pdf/.mesonpy-aid6czcw
[sagemath_doc_pdf-10.2.rc3] Build type: native build
[sagemath_doc_pdf-10.2.rc3] 
[sagemath_doc_pdf-10.2.rc3] ../meson.build:12:58: ERROR: Expecting rbracket got rparen.
[sagemath_doc_pdf-10.2.rc3]   command: [meson.current_source_dir() / 'doc' / 'make.sh'),
[sagemath_doc_pdf-10.2.rc3]            ^______________________________________________^
[sagemath_doc_pdf-10.2.rc3] 
[sagemath_doc_pdf-10.2.rc3] A full log can be found at /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-doc-pdf/.mesonpy-aid6czcw/meson-logs/meson-log.txt
[sagemath_doc_pdf-10.2.rc3] 
[sagemath_doc_pdf-10.2.rc3] ERROR Backend subprocess exited when trying to invoke build_wheel
[sagemath_doc_pdf-10.2.rc3] ********************************************************************************
[sagemath_doc_pdf-10.2.rc3] Failure building wheel
[sagemath_doc_pdf-10.2.rc3] ********************************************************************************

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 22, 2023

This is rc4. Why is it sagemath_doc_html-10.2.rc3?

Version numbers of packages are hardcoded in pkgs/*/VERSION.txt and are update by a commit done by the release manager

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

Interesting. I have wanted some of this for the documentation in sage-on-gentoo for some time. This branch lets me creates sdist for just the doc instead of having to checkout the whole tree or tarball. I guess I can figure out how to properly build from the sdist with just the cd doc && make doc-html command but I tried with python -m build to see.

fbissey@tarazed ~/sandbox/sagemath_doc_html-10.2rc3 $ python -m build -n -x -w
* Building wheel...
+ meson setup /home/fbissey/sandbox/sagemath_doc_html-10.2rc3 /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf/meson-python-native-file.ini
The Meson build system
Version: 1.3.0
Source dir: /home/fbissey/sandbox/sagemath_doc_html-10.2rc3
Build dir: /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf
Build type: native build
Project name: sagemath-doc-html
Project version: 10.2.rc3
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 1

sagemath-doc-html 10.2.rc3

  User defined options
    Native files: /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf/meson-python-native-file.ini
    buildtype   : release
    b_ndebug    : if-release
    b_vscrt     : md

Found ninja-1.11.1 at /usr/bin/ninja
+ /usr/bin/ninja                                                                                                                    
[1/1] Generating HTML documentation with a custom command (wrapped by meson to set env)
FAILED: index.html 
env SAGE_DOC=/home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc/make.sh -C /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc doc-html
/home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc/make.sh: line 12: exec: -C: invalid option
exec: usage: exec [-cl] [-a name] [command [argument ...]] [redirection ...]
ninja: build stopped: subcommand failed.

ERROR Backend subprocess exited when trying to invoke build_wheel

@kwankyu
Copy link
Collaborator

kwankyu commented Nov 23, 2023

This also failed

$ pip install upstream/sagemath_doc_pdf-10.2rc3.tar.gz 
WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
Processing ./upstream/sagemath_doc_pdf-10.2rc3.tar.gz
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [89 lines of output]
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
      Collecting meson-python
        Downloading meson_python-0.15.0-py3-none-any.whl (25 kB)
      Collecting sage-docbuild~=10.2rc4
        Downloading sage-docbuild-10.2rc4.tar.gz (71 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.3/71.3 kB 2.8 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting sphinx<8,>=5.2
        Using cached sphinx-7.2.6-py3-none-any.whl (3.2 MB)
      Collecting sphinx-copybutton
        Downloading sphinx_copybutton-0.5.2-py3-none-any.whl (13 kB)
      Collecting furo
        Downloading furo-2023.9.10-py3-none-any.whl (324 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 324.4/324.4 kB 6.8 MB/s eta 0:00:00
      Collecting jupyter-sphinx
        Downloading jupyter_sphinx-0.4.0-py3-none-any.whl (21 kB)
      Collecting sagemath-standard~=10.2rc4
        Downloading sagemath-standard-10.2rc4.tar.gz (19.4 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.4/19.4 MB 8.1 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: still running...
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'error'
        error: subprocess-exited-with-error
      
        × Getting requirements to build wheel did not run successfully.
        │ exit code: 1
        ╰─> [20 lines of output]
            Error in sitecustomize; set PYTHONVERBOSE for traceback:
            AssertionError:
            Traceback (most recent call last):
              File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
                main()
              File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
                json_out['return_val'] = hook(**hook_input['kwargs'])
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
                return hook(config_settings)
                       ^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
                return self._get_build_requires(config_settings, requirements=['wheel'])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
                self.run_setup()
              File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
                exec(code, locals())
              File "<string>", line 42, in <module>
            ModuleNotFoundError: No module named 'sage_setup'
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × Getting requirements to build wheel did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
      WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
      WARNING: There was an error checking the latest version of pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
WARNING: There was an error checking the latest version of pip.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 23, 2023

env SAGE_DOC=/home/fbissey/sandbox/sagemath_doc_html-10.2rc3/.mesonpy-qfdw7fkf /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc/make.sh -C /home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc doc-html
/home/fbissey/sandbox/sagemath_doc_html-10.2rc3/doc/make.sh: line 12: exec: -C: invalid option
exec: usage: exec [-cl] [-a name] [command [argument ...]] [redirection ...]
ninja: build stopped: subcommand failed.

Thanks for testing! This one should be fixed by 12264c0

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

Makes perfect sense (no pun intended), I should have thought it was the case.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 23, 2023

This also failed

$ pip install upstream/sagemath_doc_pdf-10.2rc3.tar.gz 
WARNING: Skipping /usr/local/lib/python3.11/site-packages/numpy-1.26.1-py3.11.egg-info due to invalid metadata entry 'name'

@kwankyu In this run, it is trying to install sagemath-standard from scratch, and that is failing somehow. Hard to see why without more output. Try if pip install -v -v -v "sagemath-standard~=10.2rc4" works

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

I am just looking at sage-doc-html right now, but I am guessing it will be the same with pdf, the sdist includes pyproject.toml.m4 which we do not want. It would be nice to remove it but that's definitely not a top priority.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 23, 2023

Should be fixed in 62b706c (not tested)

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

Should be fixed in 62b706c (not tested)

Appears to have done the job, thanks. Building sage-doc-html, single threaded... started 20mn ago.

@kwankyu
Copy link
Collaborator

kwankyu commented Nov 23, 2023

pip install -v -v -v "sagemath-standard~=10.2rc4"

Failed with same error

  WARNING: Skipping /usr/local/lib/python3.11/site-packages/packaging-23.2.dist-info due to invalid metadata entry 'name'
  WARNING: There was an error checking the latest version of pip.
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
      main()
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
      self.run_setup()
    File "/usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 42, in <module>
  ModuleNotFoundError: No module named 'sage_setup'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

Full log:
log.txt

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

@kwankyu is sage_setup installed under /usr/local/lib/python3.11/site-package or otherwise in the python path?

@kwankyu
Copy link
Collaborator

kwankyu commented Nov 23, 2023

/usr/local/lib/python3.11/site-package

sage_setup is not there.

In /usr/local/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages, there is no sage_setup either.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Nov 23, 2023

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

It should work for me in sage-on-gentoo though. I will try.

@kiwifb
Copy link
Member

kiwifb commented Nov 23, 2023

Should be fixed in 62b706c (not tested)

Appears to have done the job, thanks. Building sage-doc-html, single threaded... started 20mn ago.

I have now successfully build a sage-doc-html wheel with python -m build from the sdist.

fbissey@tarazed ~/sandbox/sagemath_doc_html-10.2rc3 $ ll dist/
total 436M
drwxr-xr-x 2 fbissey fbissey 4.0K Nov 23 16:51 .
drwxr-xr-x 4 fbissey fbissey 4.0K Nov 23 16:51 ..
-rw-r--r-- 1 fbissey fbissey 436M Nov 23 16:51 sagemath_doc_html-10.2rc3-py3-none-any.whl

Matthias Koeppe added 29 commits October 21, 2024 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants