cibuildwheel workflow for sagemath-objects, sagemath-categories #33800

mkoeppe · 2022-05-04T21:22:16Z

We rename the GH Actions workflow sdist.yml to dist.yml and add wheel-building for the distributions sagemath-objects and sagemath-categories for macOS (Intel) and Linux (x86_64, i686) using cibuildwheel.

The necessary libraries are built using the Sage distribution.

Example run: https://github.com/mkoeppe/sage/actions/runs/2522295930

Follow-ups:

After Refactor distributions sagemath-{objects,categories} through sagemath-{environment,repl} #33812 (Refactor distributions sagemath-{objects,categories}), make the built wheel for sagemath-objects available for the wheel building of sagemath-categories
Possibly use --prefix=$(pwd)/wheelhouse/local-ARCH --with-sage-venv so that SAGE_LOCAL gets copied in/out of the container

Depends on #29549
Depends on #33983
Depends on #34017

CC: @dimpase @kliem @tobiasdiez @saraedum

Component: distribution

Author: Matthias Koeppe

Branch/Commit: 93d5d11

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33800

The text was updated successfully, but these errors were encountered:

mkoeppe · 2022-05-05T01:49:45Z

Branch: u/mkoeppe/cibuildwheel_workflow_for_sagemath_objects__sagemath_categories_etc_

sagetrac-git · 2022-05-05T03:17:49Z

Commit: 64b46a1

sagetrac-git · 2022-05-05T03:17:49Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`3133c8e`	`.github/workflows/sdist.yml: Only build wheels for sagemath-objects, -categories`
`3d5850a`	`sage.env.cython_aliases: Handle ecl via default_required_modules`
`e77efb3`	`pkgs/sagemath-objects/MANIFEST.in: Add missing sage.misc.package_dir`
`434d539`	`pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.sagedoc`
`64b46a1`	`.github/workflows/dist.yml: Rename from sdist.yml`

sagetrac-git · 2022-05-05T04:27:28Z

Changed commit from 64b46a1 to 025200b

sagetrac-git · 2022-05-05T04:27:28Z

Branch pushed to git repo; I updated commit sha1. New commits:

`474104b`	`.github/workflows/dist.yml: Install gmp before building the wheel`
`025200b`	`.github/workflows/dist.yml: Install gmp, mpfr, mpc before building the wheel`

mkoeppe · 2022-05-05T04:32:53Z

comment:4

The method of installing the required system packages (gmp, mpfr, mpc) with whatever is provided by the target distribution does not work because the manylinux distributions are deliberately ancient.
Getting error: #error "GMPY2 requires MPC 1.0.3 or later." (https://github.com/mkoeppe/sage/runs/6300752059).

Also the macOS arm64 cross-build does not work: fatal error: 'gmp.h' file not found (https://github.com/mkoeppe/sage/runs/6300752117)

mkoeppe · 2022-05-05T04:38:24Z

Dependencies: #33803

mkoeppe · 2022-05-05T04:43:54Z

Work Issues: Rebase on #33803

mkoeppe · 2022-05-05T06:48:51Z

comment:7

We could add a section [tool.cibuildwheel] to pkg/*/pyproject.toml.m4 and add conda installation + conda package installation there as before-all.

sagetrac-git · 2022-06-04T04:42:14Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`4d54d21`	`.github/workflows/sdist.yml: Build wheels too`
`431492d`	`.github/workflows/sdist.yml: Set CIBW_PROJECT_REQUIRES_PYTHON`
`1e14375`	`.github/workflows/sdist.yml: Only build wheels for sagemath-objects, -categories`
`152aca2`	`.github/workflows/dist.yml: Rename from sdist.yml`
`f0bd1cf`	`.github/workflows/dist.yml: Install gmp before building the wheel`
`6f91ddc`	`.github/workflows/dist.yml: Install gmp, mpfr, mpc before building the wheel`
`79549d9`	`.github/workflows/dist.yml: Build prereqs of wheels using Sage distro`

sagetrac-git · 2022-06-04T04:42:14Z

Changed commit from 025200b to 79549d9

sagetrac-git · 2022-06-04T06:00:59Z

Branch pushed to git repo; I updated commit sha1. New commits:

`08acac4`	`build/pkgs/sage_setup/dependencies: Add jinja2`
`42e991a`	`.github/workflows/dist.yml: One package at a time, build out of pkgs/`

sagetrac-git · 2022-06-04T06:00:59Z

Changed commit from 79549d9 to 42e991a

mkoeppe · 2022-06-04T20:45:20Z

Changed dependencies from #33803 to none

mkoeppe · 2022-06-04T20:45:20Z

Changed work issues from Rebase on #33803 to none

dimpase · 2022-06-05T13:02:08Z

comment:11

check out gmpy2 wheel builder

https://github.com/aleaxit/gmpy/tree/master/.github%2Fworkflows

mkoeppe · 2022-06-05T17:37:21Z

comment:12

They are using straightforward cibuildwheel

dimpase · 2022-06-05T22:20:23Z

comment:13

but they need to have gmp installed first.

mkoeppe · 2022-06-05T22:25:31Z

comment:14

Sure, they install the libraries from source with a custom script.
In Sage, we can just use our spkg scripts for the same purpose.

mkoeppe · 2022-06-18T19:25:59Z

Dependencies: #29549

mkoeppe · 2022-06-18T22:14:41Z

Changed dependencies from #29549 to #29549, #33983

sagetrac-git · 2022-06-18T22:20:42Z

Changed commit from 42e991a to c741878

sagetrac-git · 2022-06-18T22:20:42Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`3fd1cb5`	`.github/workflows/dist.yml: Use pkgconfig, not pkg-config on centos`
`f9407af`	`.github/workflows/dist.yml: Set ACLOCAL_PATH`
`fcbf814`	`.github/workflows/dist.yml: Use cibuildwheel==2.7.0`
`77affa8`	`tox.ini: Add manylinux_2_28`
`dc352d9`	`.github/workflows/dist.yml: Use --enable-build-as-root`
`ac0e301`	`do not use prefix`
`7999353`	`update PATH`
`07a018b`	`Use CIBW_BUILD_FRONTEND=build`
`e653ff1`	`Build wheels from sdists`
`c741878`	`Unpack sdist for cibuildwheel`

sagetrac-git · 2022-06-18T22:44:43Z

Changed commit from c741878 to ae12df2

sagetrac-git · 2022-06-18T22:44:43Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ae12df2`	`.github/workflows/dist.yml: Disable musllinux for now`

mkoeppe · 2022-06-18T23:02:14Z

Author: Matthias Koeppe

sagetrac-git · 2022-06-18T23:09:57Z

Branch pushed to git repo; I updated commit sha1. New commits:

`65bc0f1`	`.github/workflows/dist.yml: Add comment`

sagetrac-git · 2022-06-18T23:09:57Z

Changed commit from ae12df2 to 65bc0f1

sagetrac-git · 2022-06-19T01:00:57Z

Changed commit from 65bc0f1 to 036cf70

sagetrac-git · 2022-06-19T01:00:57Z

Branch pushed to git repo; I updated commit sha1. New commits:

`036cf70`	`.github/workflows/dist.yml: Build Linux x86_64 and i686 in separate jobs`

mkoeppe · 2022-06-19T03:28:51Z

Changed dependencies from #29549, #33983 to #29549, #33983, #34017

sagetrac-git · 2022-06-19T03:34:33Z

Branch pushed to git repo; I updated commit sha1. New commits:

`58d62f4`	`build/bin/write-dockerfile.sh: ADD src/VERSION.txt`
`926bfcd`	`Merge #34017`

sagetrac-git · 2022-06-19T03:34:33Z

Changed commit from 036cf70 to 926bfcd

sagetrac-git · 2022-06-28T18:25:45Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e52fd00`	`build/bin/write-dockerfile.sh: Copy config.rpath to container`
`93d5d11`	`Merge #29549`

sagetrac-git · 2022-06-28T18:25:45Z

Changed commit from 926bfcd to 93d5d11

mkoeppe · 2022-06-28T18:26:45Z

comment:27

Merged latest #29549, ready for review

dimpase · 2022-06-29T07:20:24Z

comment:28

is this being tested on GH Actions?

mkoeppe · 2022-06-29T15:53:12Z

comment:29

Yes, see link in the ticket description

dimpase · 2022-06-29T16:35:51Z

comment:30

lgtm

dimpase · 2022-06-29T16:35:51Z

Reviewer: Dima Pasechnik

mkoeppe · 2022-06-29T20:48:30Z

comment:31

Thanks!

vbraun · 2022-07-09T22:33:36Z

Changed branch from u/mkoeppe/cibuildwheel_workflow_for_sagemath_objects__sagemath_categories_etc_ to 93d5d11

mkoeppe added this to the sage-9.7 milestone May 4, 2022

mkoeppe added c: distribution labels May 4, 2022

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~cibuildwheel workflow for sagemath-objects, sagemath-categories etc.~~ cibuildwheel workflow for sagemath-objects, sagemath-categories Jun 18, 2022

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Jun 19, 2022

dimpase added s: positive review and removed s: needs review labels Jun 29, 2022

vbraun removed the s: positive review label Jul 9, 2022

vbraun closed this as completed in 042e07e Jul 9, 2022

cibuildwheel workflow for sagemath-objects, sagemath-categories #33800

cibuildwheel workflow for sagemath-objects, sagemath-categories #33800

Comments

mkoeppe commented May 4, 2022

mkoeppe commented May 5, 2022

sagetrac-git mannequin commented May 5, 2022

sagetrac-git mannequin commented May 5, 2022

sagetrac-git mannequin commented May 5, 2022

sagetrac-git mannequin commented May 5, 2022

mkoeppe commented May 5, 2022

mkoeppe commented May 5, 2022

mkoeppe commented May 5, 2022

mkoeppe commented May 5, 2022

sagetrac-git mannequin commented Jun 4, 2022

sagetrac-git mannequin commented Jun 4, 2022

sagetrac-git mannequin commented Jun 4, 2022

sagetrac-git mannequin commented Jun 4, 2022

mkoeppe commented Jun 4, 2022

mkoeppe commented Jun 4, 2022

dimpase commented Jun 5, 2022

mkoeppe commented Jun 5, 2022

dimpase commented Jun 5, 2022

mkoeppe commented Jun 5, 2022

mkoeppe commented Jun 18, 2022

mkoeppe commented Jun 18, 2022

sagetrac-git mannequin commented Jun 18, 2022

sagetrac-git mannequin commented Jun 18, 2022

sagetrac-git mannequin commented Jun 18, 2022

sagetrac-git mannequin commented Jun 18, 2022

mkoeppe commented Jun 18, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jun 18, 2022

sagetrac-git mannequin commented Jun 18, 2022

sagetrac-git mannequin commented Jun 19, 2022

sagetrac-git mannequin commented Jun 19, 2022

This comment has been minimized.

mkoeppe commented Jun 19, 2022

sagetrac-git mannequin commented Jun 19, 2022

sagetrac-git mannequin commented Jun 19, 2022

sagetrac-git mannequin commented Jun 28, 2022

sagetrac-git mannequin commented Jun 28, 2022

mkoeppe commented Jun 28, 2022

dimpase commented Jun 29, 2022

mkoeppe commented Jun 29, 2022

dimpase commented Jun 29, 2022

dimpase commented Jun 29, 2022

mkoeppe commented Jun 29, 2022

vbraun commented Jul 9, 2022