CI build, doc-build: Run containers explicitly, separate jobs for pyright, build, modularized tests, long tests

[mkoeppe]

Running the containers explicitly, instead of using the declarative container: feature of GH Actions gives us more control:

• we can create more space on the host if necessary; we just scraped by an out of space condition in CI docbuild: Do not rebuild sagelib from scratch #36473 / Revert to non-incremental docbuild #36469 (comment)
• we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from CI: Merge open blocker PRs in all CI workflows + other improvements #36338 (comment), ci: improve things around auto-apply of fixes #36349
• it enables caching between the various workflows (as first discussed in CI: Merge workflows into build.yml #36446).

We split out static type checking with pyright into its separate workflow:

• pyright.yml: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in ci: improve things around auto-apply of fixes #36349.

The workflow build.yml first launches a job:

• test-new: It builds incrementally (using a tox-generated Dockerfile and https://github.com/marketplace/actions/build-and-push-docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change.

After this is completed, more jobs are launched:

• test-mod: It again builds incrementally and tests a modularized distribution. Later (with more from Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095), more jobs will be added to this matrix job for other distributions.
• test-long: It again builds incrementally and runs the long test.

The workflows doc-build.yml and doc-build-pdf.yml again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.)

Splitting the workflow jobs implements @kwankyu's suggestion from:

• CI Build & Test: First build incrementally and test changed files only #35652 (comment) (Fixes build.yml: Report failures (e.g. from incremental test) earlier #35814)

The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache-backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion.

There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• Depends on CI Build & Test: Show doctest failures, warnings as annotations in the 'Files changed' tab #36938 (merged here)
• Depends on pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #36951 (merged here)
• Depends on CI: Update Linux platforms / runners, remove ubuntu-trusty, opensuse-{15.3,15.4} #37351 (merged here)
• Depends on CI Build&Test: Fix "test modularized distributions" after #37022 #37750 (merged here)
• Depends on pip 24, setuptools 69.5.1, hatchling 1.22.5, hatch_fancy_pypi_readme 24.1.0, platformdirs 4.2.0, packaging 24.0, trove_classifiers 2024.4.10, wheel 0.43.0 #37277 (merged here)

[kwankyu]

src/sage/schemes/toric/variety.py sneaked in?

[mkoeppe]

via #36443

[tobiasdiez]

I don't get the point of this PR. It seems to mostly complicate the setup and essentially reimplements the container feature of github. For the out-of-space issue it would be nice to first know what's the route cause to be able to evaluate if steps to fixing it go into the correct direction. For potential speedups by caching the docker image, I'm not sure if this is really needed. Yes, from a conceptual viewpoint you only want to build sagelib once, and then reuse it - but well, it mostly takes 2-10 mins of incremental compilation anyway. Personally, I think we have more pressing issues with the ci setup and better not introduce more moving parts (atm at least).

[mkoeppe]

For the out-of-space issue it would be nice to first know what's the root cause to be able to evaluate if steps to fixing it go into the correct direction.

It's really not very complicated. GitHub runs the container from its very full runner image. We know that it does not have enough space for our large containers.

[mkoeppe]

It seems to [...] essentially reimplements the container feature of github.

That's right. The declarative container feature is convenient, but we have run into the two restrictions described in the PR description. So we just use docker directly; problem solved.

[mkoeppe]

Yes, from a conceptual viewpoint you only want to build sagelib once, and then reuse it - but well, it mostly takes 2-10 mins of incremental compilation anyway.

Actually, the compilation can take much longer when low level parts of sagelib are touched, and when certain packages are updated.

[tobiasdiez]

For the out-of-space issue it would be nice to first know what's the root cause to be able to evaluate if steps to fixing it go into the correct direction.

It's really not very complicated. GitHub runs the container from its very full runner image. We know that it does not have enough space for our large containers.

Github provides 14gb of space (after its runner image), that should be sufficient (and was sufficient) to run a 7gb image in it and install sage. Also our image got bigger by a few houndred mbs.

[mkoeppe]

If you want to understand more details, I'd suggest that you experiment with the space parameters in https://github.com/sagemath/sage/blob/develop/.github/workflows/docker.yml#L148

[tobiasdiez]

It's already quite easy to run commands outside the container. Here is an example for exactly your first use case: https://github.com/osquery/osquery/blob/a916e80a895f8abb9b749161d774d528d0466b31/.github/workflows/hosted_runners.yml#L257-L263 I don't think we need more flexibility.