Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Add PDF doc building #35169

Closed
wants to merge 0 commits into from
Closed

CI: Add PDF doc building #35169

wants to merge 0 commits into from

Conversation

mkoeppe
Copy link
Contributor

@mkoeppe mkoeppe commented Feb 22, 2023

📚 Description

We add another workflow that is run on every PR: building PDF docs.

📝 Checklist

  • I have made sure that the title is self-explanatory and the description concisely explains the PR.
  • I have linked an issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe mkoeppe requested review from vbraun and tobiasdiez February 22, 2023 00:11
@mkoeppe mkoeppe added this to the sage-10.0 milestone Feb 22, 2023
@mkoeppe mkoeppe self-assigned this Feb 22, 2023
@codecov-commenter
Copy link

codecov-commenter commented Feb 22, 2023

Codecov Report

Patch coverage has no change and project coverage change: -0.01 ⚠️

Comparison is base (c00e6c2) 88.62% compared to head (7f3a924) 88.61%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #35169      +/-   ##
===========================================
- Coverage    88.62%   88.61%   -0.01%     
===========================================
  Files         2148     2148              
  Lines       398855   398855              
===========================================
- Hits        353480   353459      -21     
- Misses       45375    45396      +21     

see 27 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 22, 2023

This run failed with

2023-02-22T04:23:53.6488895Z [sagemath_doc_pdf-none] Rule 'pdflatex': File changes, etc:
2023-02-22T04:23:53.6489356Z [sagemath_doc_pdf-none]    Changed files, or newly in use since previous run(s):
2023-02-22T04:23:53.6495865Z [sagemath_doc_pdf-none]       '/root/.texlive2019/texmf-var/web2c/pdftex/pdflatex.fmt'
2023-02-22T04:23:53.6496435Z [sagemath_doc_pdf-none] Latexmk: Maximum runs of pdflatex reached without getting stable files
2023-02-22T04:23:53.6496951Z [sagemath_doc_pdf-none] Latexmk: Summary of warnings from last run of *latex:
2023-02-22T04:23:53.6497438Z [sagemath_doc_pdf-none]   =====Latex reported missing or unavailable character(s).
2023-02-22T04:23:53.6497879Z [sagemath_doc_pdf-none] =====See log file for details.
2023-02-22T04:23:53.6504195Z [sagemath_doc_pdf-none]   Latex failed to resolve 973 citation(s)
2023-02-22T04:23:53.6504612Z [sagemath_doc_pdf-none] ----------------------
2023-02-22T04:23:53.6505039Z [sagemath_doc_pdf-none] This message may duplicate earlier message.
2023-02-22T04:23:53.6505529Z [sagemath_doc_pdf-none] Latexmk: Failure in processing file 'combinat.tex':
2023-02-22T04:23:53.6505990Z [sagemath_doc_pdf-none]    'pdflatex' needed too many passes
2023-02-22T04:23:53.6506365Z [sagemath_doc_pdf-none] ----------------------
2023-02-22T04:23:53.6512886Z [sagemath_doc_pdf-none] Latexmk: Use the -f option to force complete processing,
2023-02-22T04:23:53.6513455Z [sagemath_doc_pdf-none]  unless error was exceeding maximum runs, or warnings treated as errors.
2023-02-22T04:23:53.6513917Z [sagemath_doc_pdf-none] Latexmk: Examining 'combinat.log'
2023-02-22T04:23:53.6514319Z [sagemath_doc_pdf-none] === TeX engine is 'pdfTeX'
2023-02-22T04:23:53.6514731Z [sagemath_doc_pdf-none] Latexmk: applying rule 'pdflatex'...
2023-02-22T04:23:53.6515422Z [sagemath_doc_pdf-none] Latexmk: All targets (combinat.pdf) are up-to-date
2023-02-22T04:23:53.6522277Z [sagemath_doc_pdf-none] make[5]: *** [Makefile:29: combinat.pdf] Error 12
2023-02-22T04:23:53.6522738Z [sagemath_doc_pdf-none] Error building the documentation.
2023-02-22T04:23:53.6523161Z [sagemath_doc_pdf-none] Traceback (most recent call last):
2023-02-22T04:23:53.6523610Z [sagemath_doc_pdf-none]   File "<frozen runpy>", line 198, in _run_module_as_main
2023-02-22T04:23:53.6524075Z [sagemath_doc_pdf-none]   File "<frozen runpy>", line 88, in _run_code
2023-02-22T04:23:53.6531441Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/src/sage_docbuild/__main__.py", line 498, in <module>
2023-02-22T04:23:53.6531913Z [sagemath_doc_pdf-none]     sys.exit(main())
2023-02-22T04:23:53.6532383Z [sagemath_doc_pdf-none]              ^^^^^^
2023-02-22T04:23:53.6532846Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/src/sage_docbuild/__main__.py", line 495, in main
2023-02-22T04:23:53.6533883Z [sagemath_doc_pdf-none]     builder()
2023-02-22T04:23:53.6541906Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/src/sage_docbuild/builders.py", line 300, in pdf
2023-02-22T04:23:53.6542455Z [sagemath_doc_pdf-none]     raise RuntimeError(error_message % (command, tex_dir))
2023-02-22T04:23:53.6543066Z [sagemath_doc_pdf-none] RuntimeError: failed to run $MAKE all-pdf in /sage/local/share/doc/sage/latex/en/reference/combinat
2023-02-22T04:23:53.6543494Z [sagemath_doc_pdf-none] 
2023-02-22T04:23:53.6543932Z [sagemath_doc_pdf-none]     Note: incremental documentation builds sometimes cause spurious
2023-02-22T04:23:53.6544465Z [sagemath_doc_pdf-none]     error messages. To be certain that these are real errors, run
2023-02-22T04:23:53.6547924Z [sagemath_doc_pdf-none]     "make doc-clean doc-uninstall" first and try again.
2023-02-22T04:23:58.4423136Z [sagemath_doc_pdf-none] [plot3d   ] building [latex]: all documents
2023-02-22T04:23:59.1009353Z [sagemath_doc_pdf-none] [plot3d   ] processing plot3d.tex... index sage/plot/plot3d/introduction sage/plot/plot3d/plot3d sage/plot/plot3d/parametric_plot3d sage/plot/plot3d/revolution_plot3d sage/plot
/plot3d/plot_field3d sage/plot/plot3d/implicit_plot3d sage/plot/plot3d/list_plot3d sage/plot/plot3d/base sage/plot/plot3d/shapes sage/plot/plot3d/shapes2 sage/plot/plot3d/platonic sage/plot/plot3d/parametric_surface sage/plot/plo
t3d/implicit_surface sage/plot/plot3d/texture sage/plot/plot3d/index_face_set sage/plot/plot3d/transform sage/plot/plot3d/tri_plot sage/plot/plot3d/tachyon threejs
2023-02-22T04:23:59.1016994Z [sagemath_doc_pdf-none] [plot3d   ] resolving references...
2023-02-22T04:23:59.4794189Z [sagemath_doc_pdf-none] [plot3d   ] done
2023-02-22T04:24:01.2175171Z [sagemath_doc_pdf-none] make[4]: *** [Makefile:69: doc-pdf--reference-combinat] Error 1

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 22, 2023

It builds pdflatex.fmt too many times; probably related to the weird message "resetting $HOME value".

2023-02-22T04:18:06.7017772Z [sagemath_doc_pdf-none] kpathsea: Running mktexfmt pdflatex.fmt
2023-02-22T04:18:06.8275609Z [sagemath_doc_pdf-none] mktexfmt [WARNING]: resetting $HOME value (was /github/home) to root's actual home (/root).
2023-02-22T04:18:07.0131178Z [sagemath_doc_pdf-none] mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
2023-02-22T04:18:07.0131769Z [sagemath_doc_pdf-none] mktexfmt:   /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
2023-02-22T04:18:07.0132289Z [sagemath_doc_pdf-none] mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
2023-02-22T04:18:07.0132790Z [sagemath_doc_pdf-none] mktexfmt:   /root/.texlive2019/texmf-config/web2c/fmtutil.cnf
2023-02-22T04:18:07.0142799Z [sagemath_doc_pdf-none] mktexfmt [INFO]: writing formats under /root/.texlive2019/texmf-var/web2c
2023-02-22T04:18:07.0160273Z [sagemath_doc_pdf-none] mktexfmt [INFO]: --- remaking pdflatex with pdftex
2023-02-22T04:18:07.0429130Z [sagemath_doc_pdf-none] mktexfmt: running `pdftex -ini   -jobname=pdflatex -progname=pdflatex -translate-file=cp227.tcx *pdflatex.ini' ...

Copy link
Contributor

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add the pdf build as a step after the html build? Otherwise both jobs compile sage and build things like the index twice.

@tobiasdiez tobiasdiez changed the title .github/workflows/doc-build-pdf.yml: New CI: Add PDF doc building Feb 22, 2023
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 22, 2023

That would of course also be a solution, but I didn't want to make the time to HTML deployment longer.

@tobiasdiez
Copy link
Contributor

How long would the pdf docbuild alone take? The html docbuild is already taken quite some time, adding a bit more shouldn't matter that much...

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 22, 2023

About 30 minutes wall time, according to the log:

2023-02-22T09:38:39.3257968Z make --no-print-directory sagemath_doc_pdf-SAGE_DOCS-no-deps
2023-02-22T09:38:40.0063286Z [sagemath_doc_pdf-none] mkdir -p en/reference/repl
[...]
2023-02-22T10:08:03.4844378Z [sagemath_doc_pdf-none] 
2023-02-22T10:08:03.4845122Z [sagemath_doc_pdf-none] PDF documents have been created in subdirectories of
2023-02-22T10:08:03.4899605Z [sagemath_doc_pdf-none] 
2023-02-22T10:08:03.4900016Z [sagemath_doc_pdf-none]   /sage/local/share/doc/sage/pdf/en/reference
2023-02-22T10:08:03.4900347Z [sagemath_doc_pdf-none] 

@tobiasdiez
Copy link
Contributor

Then I would prefer to add it directly in the docbuild workflow.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 24, 2023

Half an hour is pretty long when people are making edits to the documentation, I'd say

@tobiasdiez
Copy link
Contributor

The current workflow already takes 1.5h, so it's not like that you get realtime feedback. The main build also takes 2h or more, so both workflows would be finished at about the same time.

@dimpase
Copy link
Member

dimpase commented Feb 26, 2023

I don't think this should be triggered automatically. It's a slow workflow, and only rarely pdf building gets broken (assuming html building works).

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 26, 2023

The workflow is not slower than "build".

And since we were upgraded to GH Teams, we have many more parallel runners.

@dimpase
Copy link
Member

dimpase commented Feb 26, 2023

The workflow is not slower than "build".

really? it's an extra 30min to 1h of wall clock to making html docs - as it would pull in a good chunk of TeXLive, and as it is something to be run (for quite a while) after html docs have been built.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 26, 2023

Yes, really. Except currently it's building incrementally from an ancient image and so it recompiles more than the other workflows (Build&Test, Build documentation).

@dimpase
Copy link
Member

dimpase commented Feb 26, 2023

OK - let's see how it goes. If it proves to be a CI bottleneck we should turn it off by default, then.

@dimpase
Copy link
Member

dimpase commented Feb 28, 2023

Can the results from the PDF build be uploaded to netify, just like it's done for html?

@tobiasdiez
Copy link
Contributor

I still think it should be just an added step in the (html) docbuild workflow. Then you can also easily upload it to netlify.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 1, 2023

I don't think there's value in making the pdfs available. All that matters is that it builds.

@dimpase
Copy link
Member

dimpase commented Mar 1, 2023

you still want to know whether an iffy character you used in a docstring and broke pdf building, looks correct after a workaround

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 23, 2023

Let's get this in please.

@dimpase
Copy link
Member

dimpase commented Mar 23, 2023

OK, pdf upload may be done in another PR

Copy link
Member

@dimpase dimpase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added build is failing (after the merge at least).

Moreover, it will never be possible to upload the pdf to netlify from a second workflow (since the publish workflow can only be triggered by one workflow). So this is a serious design flaw and should be fixed. Finally, the uploaded artefact is 500mb big. Are these only the doc pdfs? I cannot check since the download fails for me at a couple of mb. But this is another point why it would be important to upload the pdfs to netlify since no one will download 500mb to just check a symbol in the compiled pdf.

.github/workflows/doc-build-pdf.yml Outdated Show resolved Hide resolved
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 24, 2023

The only purpose of this workflow is to flag an error before it hits Volker's queue.
Nobody is going to check the built PDFs either way.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 24, 2023

"serious design flaw" - no

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

It does fail, with a strange tex error.

2023-03-24T22:42:55.0016004Z [sagemath_doc_pdf-none] pdflatex: fatal: Could not undump 6688 4-byte item(s) from /root/.texlive2019/texmf-var/web2c/pdftex/pdflatex.fmt.
2023-03-24T22:42:55.0126267Z [sagemath_doc_pdf-none] Latexmk: fls file doesn't appear to have been made.
2023-03-24T22:42:55.0126804Z [sagemath_doc_pdf-none] Collected error summary (may duplicate other messages):
2023-03-24T22:42:55.0127250Z [sagemath_doc_pdf-none]   pdflatex: Command for 'pdflatex' gave return code 1
2023-03-24T22:42:55.0127667Z [sagemath_doc_pdf-none]       Refer to 'installation.log' for details
2023-03-24T22:42:55.0128029Z [sagemath_doc_pdf-none] ----------------------
2023-03-24T22:42:55.0128396Z [sagemath_doc_pdf-none] This message may duplicate earlier message.
2023-03-24T22:42:55.0128818Z [sagemath_doc_pdf-none] Latexmk: Failure in processing file 'installation.tex':
2023-03-24T22:42:55.0129294Z [sagemath_doc_pdf-none]    *LaTeX didn't generate the expected log file 'installation.log'
2023-03-24T22:42:55.0129882Z [sagemath_doc_pdf-none] ----------------------
2023-03-24T22:42:55.0130273Z [sagemath_doc_pdf-none] Latexmk: Use the -f option to force complete processing,
2023-03-24T22:42:55.0130749Z [sagemath_doc_pdf-none]  unless error was exceeding maximum runs, or warnings treated as errors.
2023-03-24T22:42:55.0131201Z [sagemath_doc_pdf-none] Latexmk: Errors, so I did not complete making targets
2023-03-24T22:42:55.0131694Z [sagemath_doc_pdf-none] mktexfmt [INFO]: /root/.texlive2019/texmf-var/web2c/pdftex/pdflatex.fmt installed.
2023-03-24T22:42:55.0132153Z [sagemath_doc_pdf-none] make[5]: *** [Makefile:29: installation.pdf] Error 12
2023-03-24T22:42:55.0132538Z [sagemath_doc_pdf-none] Error building the documentation.
2023-03-24T22:42:55.0132902Z [sagemath_doc_pdf-none] Traceback (most recent call last):
2023-03-24T22:42:55.0133302Z [sagemath_doc_pdf-none]   File "<frozen runpy>", line 198, in _run_module_as_main
2023-03-24T22:42:55.0133785Z [sagemath_doc_pdf-none]   File "<frozen runpy>", line 88, in _run_code
2023-03-24T22:42:55.0134272Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/pkgs/sage-docbuild/sage_docbuild/__main__.py", line 498, in <module>
2023-03-24T22:42:55.0134668Z [sagemath_doc_pdf-none]     sys.exit(main())
2023-03-24T22:42:55.0135045Z [sagemath_doc_pdf-none]              ^^^^^^
2023-03-24T22:42:55.0135478Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/pkgs/sage-docbuild/sage_docbuild/__main__.py", line 495, in main
2023-03-24T22:42:55.0135853Z [sagemath_doc_pdf-none]     builder()
2023-03-24T22:42:55.0136281Z [sagemath_doc_pdf-none]   File "/__w/sage/sage/pkgs/sage-docbuild/sage_docbuild/builders.py", line 300, in pdf
2023-03-24T22:42:55.0136747Z [sagemath_doc_pdf-none]     raise RuntimeError(error_message % (command, tex_dir))
2023-03-24T22:42:55.0137258Z [sagemath_doc_pdf-none] RuntimeError: failed to run $MAKE all-pdf in /sage/local/share/doc/sage/latex/en/installation
2023-03-24T22:42:55.0137628Z [sagemath_doc_pdf-none] 

Copy link
Member

@dimpase dimpase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedora 31? Please, no. It's dead and buried - and perhaps its 4-years old texlive is to blame here.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 25, 2023

I chose this outdated one because it is close to one of @vbraun's buildbot configurations.
https://github.com/sagemath/buildbot-sage/blob/master/sage_bot/workers.py#L126

@dimpase
Copy link
Member

dimpase commented Mar 25, 2023

Blast from the past... Fedora 30 has been dead for almost 3 years.

@github-actions
Copy link

Documentation preview for this PR is ready! 🎉
Built with commit: 7f3a924

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Mar 26, 2023

This was merged (at commit 5330147) into 10.0.beta6.
Closing ref: 883b993

@mkoeppe mkoeppe closed this Mar 26, 2023
@mkoeppe mkoeppe deleted the ci-doc-pdf branch March 26, 2023 20:49
vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 1, 2023
    
<!-- Please provide a concise, informative and self-explanatory title.
-->
<!-- Don't put issue numbers in the title. Put it in the Description
below. -->
<!-- For example, instead of "Fixes sagemath#12345", use "Add a new method to
multiply two integers" -->

### 📚 Description

Follow-up on sagemath#35169.

The container `sage-docker-fedora-31-maximal-with-targets` used for the
PDF docbuild turned out to be not reliable.
Here we replace it by `ubuntu-focal-standard-with-targets` and install
texlive in it.

We also copy over the incremental build from doc-build.yml and the
method to get CI fixes from blocker tickets from sagemath#36338.

This workflow is currently disabled in sagemath/sage.
Example run:
https://github.com/mkoeppe/sage/actions/runs/6318741659/job/17158468016

<!-- Describe your changes here in detail. -->
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. It should be `[x]` not `[x
]`. -->

- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36338
- Depends on sagemath#36348

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#35373
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik
vbraun pushed a commit that referenced this pull request Oct 8, 2023
    
<!-- Please provide a concise, informative and self-explanatory title.
-->
<!-- Don't put issue numbers in the title. Put it in the Description
below. -->
<!-- For example, instead of "Fixes #12345", use "Add a new method to
multiply two integers" -->

### 📚 Description

Follow-up on #35169.

The container `sage-docker-fedora-31-maximal-with-targets` used for the
PDF docbuild turned out to be not reliable.
Here we replace it by `ubuntu-focal-standard-with-targets` and install
texlive in it.

We also copy over the incremental build from doc-build.yml and the
method to get CI fixes from blocker tickets from #36338.

This workflow is currently disabled in sagemath/sage.
Example run:
https://github.com/mkoeppe/sage/actions/runs/6318741659/job/17158468016

<!-- Describe your changes here in detail. -->
<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. It should be `[x]` not `[x
]`. -->

- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->
- Depends on #36338
- Depends on #36348

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #35373
Reported by: Matthias Köppe
Reviewer(s): Dima Pasechnik
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants