GH Actions: Add test of the modularized distributions

[mkoeppe]

... as part of the workflow build.yml. This is to catch modularization regressions on tickets.

To this end, we extend build/pkgs/sagemath-{objects,categories,environment,repl}/spkg-install scripts so that they build and store wheels in SAGE_SPKG_WHEELS (but do not install them!), and then run tox on the wheels (unless disabled by setting SAGE_CHECK=no).

A new top-level make target pypi-wheels runs these scripts for these and other small distribution packages, ensuring that wheels are present even if configure --enable-editable is in use.

Another top-level make target wheels does the same but also includes sage_conf and sagemath_standard. This is useful for venv installs and helps with the problem noted in #32913.

We also extend the script src/bin/sage-update-version so that it automatically updates the install-requires.txt files for our distribution packages.

CC: @dimpase @saraedum @kwankyu @jhpalmieri

Component: refactoring

Author: Matthias Koeppe

Branch/Commit: 64b212c

Reviewer: Tobias Diez, Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/33817

[mkoeppe]

Dependencies: #33803

[mkoeppe]

Branch: u/mkoeppe/gh_actions__add_test_of_the_modularized_distributions

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9c8f668	build/pkgs/sagemath_objects/spkg-install: Handle errors
dbfe66c	build/pkgs/sagemath_objects/dependencies: Add sage_setup
af78202	Makefile (pypi-wheels, wheels): New
9c634c2	.github/workflows/build.yml: Build/test modularized distributions

[sagetrac-git]

Commit: 9c634c2

[sagetrac-git]

Changed commit from 9c634c2 to ce003ce

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ce003ce

build/pkgs/sagemath_objects/spkg-install: Only use Sage wheels for the build

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Changed commit from ce003ce to 83cfb74

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

83cfb74

.github/workflows/build.yml: Test modularized distributions earlier, continue testing on failures

[tobiasdiez]

comment:8

How much longer does the build workflow now take?

[mkoeppe]

comment:9

Running right now at https://github.com/sagemath/sagetrac-mirror/runs/6336808131?check_suite_focus=true

[sagetrac-git]

Changed commit from 83cfb74 to b03779d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

b03779d

Makefile: Move sage_docbuild from PYPI_WHEEL_PACKAGES to WHEEL_PACKAGES

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

171b7c6	build/pkgs/sagemath_objects/spkg-install: When ensurepip is not available, fall back to using build --no-isolation
3f4c1b8	.github/workflows/build.yml: Run modularized dists earlier

[sagetrac-git]

Changed commit from b03779d to 3f4c1b8

[sagetrac-git]

Changed commit from 3f4c1b8 to 193f067

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

193f067

build/pkgs/sagemath_objects/spkg-install: Run tox with --discover

[mkoeppe]

Changed dependencies from #33803 to #33803, #32716

[sagetrac-git]

Changed commit from 193f067 to b87f813

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

064f453	build/pkgs/sagemath_objects/spkg-install: Use tox -e sagepython
5ce30b8	pkgs/sage-setup, sagemath-{objects,categories,standard}: Add tox environment sagepython
9db6b2f	src/doc/en/developer/packaging_sage_library.rst: Use sagepython instead of hardcoded py39
b87f813	Merge #32716

[sagetrac-git]

Changed commit from b87f813 to d1a1402

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d1a1402

build/pkgs/sagemath_objects/spkg-install: Fixup

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6a6ea4c

pkgs/*/tox.ini: Use {envpython}

[sagetrac-git]

Changed commit from d1a1402 to 6a6ea4c

[sagetrac-git]

Changed commit from 6a6ea4c to bf0d35a

[sagetrac-git]

Changed commit from 52e8c41 to 6948887

[mkoeppe]

comment:56

New test running at https://github.com/mkoeppe/sage/actions/runs/2736095948

[sagetrac-git]

Changed commit from 6948887 to a5c1c0c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a5c1c0c

pkgs/sagemath-{categories,repl,objects}: Use sage -t --initial

[kwankyu]

Changed branch from u/mkoeppe/gh_actions__add_test_of_the_modularized_distributions to u/klee/gh_actions__add_test_of_the_modularized_distributions

[kwankyu]

comment:59

I removed some white spaces.

Other than that, the code looks good to me as far as I can read (not very far as usual).

The new workflow seems work well in github.

But on my machine (mac), I get

$ ./configure --editable; make SAGE_CHECK=yes pypi-wheels
...
...
[sagemath_categories-9.7.beta6] WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
[sagemath_categories-9.7.beta6] [40038] /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories$ /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/sagepython-norequirements/bin/python -m pip install --no-deps -U .tox/.tmp/package/1/sagemath_categories-9.7b6-cp39-cp39-macosx_12_0_x86_64.whl >.tox/sagepython-norequirements/log/sagepython-norequirements-2.log
[sagemath_categories-9.7.beta6] ERROR: invocation failed (exit code 1), logfile: /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/sagepython-norequirements/log/sagepython-norequirements-2.log
[sagemath_categories-9.7.beta6] ================================== log start ===================================
[sagemath_categories-9.7.beta6] Looking in links: file:///Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels
[sagemath_categories-9.7.beta6] ERROR: sagemath_categories-9.7b6-cp39-cp39-macosx_12_0_x86_64.whl is not a supported wheel on this platform.
[sagemath_categories-9.7.beta6] 
[sagemath_categories-9.7.beta6] =================================== log end ====================================
[sagemath_categories-9.7.beta6] ___________________________________ summary ____________________________________
[sagemath_categories-9.7.beta6] ERROR:   sagepython-norequirements: InvocationError for command /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/sagepython-norequirements/bin/python -m pip install --no-deps -U .tox/.tmp/package/1/sagemath_categories-9.7b6-cp39-cp39-macosx_12_0_x86_64.whl (exited with code 1)
make[3]: *** [sagemath_categories-SAGE_VENV-no-deps] Error 1
make[2]: *** [/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.9/var/lib/sage/installed/sagemath_categories-9.7.beta6] Error 2

real	2m58.107s
user	2m41.881s
sys	0m14.927s
***************************************************************
Error building Sage.

Is this expected?

---

New commits:

097a96e

Removed some white spaces

[kwankyu]

Changed commit from a5c1c0c to 097a96e

[kwankyu]

comment:60

I still get the same even with make distclean; ./configure; make pypi-wheels.

[mkoeppe]

comment:61

Is this python3.9 from Homebrew? And is this tox from Homebrew?

[kwankyu]

comment:62

Homebrew python3.9 according to ./configure,

Checking whether SageMath should install SPKG python3...
checking whether any of bzip2 liblzma libffi is installed as or will be installed as SPKG... no
checking for python3 >= 3.8.0, < 3.11.0 with modules sqlite3, ctypes, math, hashlib, crypt, socket, zlib, distutils.core, ssl, ensurepip... 
checking ... whether /usr/local/bin/python3 is good... yes
checking for python3 >= 3.8.0, < 3.11.0 with modules sqlite3, ctypes, math, hashlib, crypt, socket, zlib, distutils.core, ssl, ensurepip... /usr/local/bin/python3
configure: will use system package and not install SPKG python3
checking whether /usr/local/bin/python3 is configured to build multiarch extensions... no
checking whether "-march=native" works with the C/C++ compilers configured for building extensions for /usr/local/bin/python3... yes

[kwankyu]

comment:63

If I run tox from SAGE_ROOT, it is the tox of Homebrew.

[kwankyu]

comment:64

It is Hombrew tox according to ./configure,

Checking whether SageMath should install SPKG tox...
checking for tox >= 3.21.4... /usr/local/bin/tox
configure: will use system package and not install SPKG tox

[mkoeppe]

comment:65

Thanks. I can reproduce this error here. I'll investigate

[mkoeppe]

comment:66

OK I solved the same problem before for sagemath-standard. Fix coming up

[mkoeppe]

Changed branch from u/klee/gh_actions__add_test_of_the_modularized_distributions to u/mkoeppe/gh_actions__add_test_of_the_modularized_distributions

[sagetrac-git]

Changed commit from 097a96e to 64b212c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

64b212c

pkgs/sagemath-*/tox.ini: Make sagepython-norequirements actually use sage python

[jhpalmieri]

comment:69

Replying to @kwankyu:

But on my machine (mac), I get

$ ./configure --editable; make SAGE_CHECK=yes pypi-wheels

Is this expected?

Did you mean/do --enable-editable? For what it's worth, I tend to use && rather than ; between commands so if the first one fails, it doesn't continue to the second one.

[kwankyu]

Changed reviewer from Tobias Diez, ... to Tobias Diez, Kwankyu Lee, ...

[kwankyu]

comment:71

Thanks. This ./configure --enable-editable && make pypi-wheels now works well.

Then I am positive with this ticket.

[mkoeppe]

comment:72

Thank you!

[kwankyu]

Changed reviewer from Tobias Diez, Kwankyu Lee, ... to Tobias Diez, Kwankyu Lee

[vbraun]

Changed branch from u/mkoeppe/gh_actions__add_test_of_the_modularized_distributions to 64b212c