relocatable wheel version of package sage_conf #31396

mkoeppe · 2021-02-14T22:06:47Z

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in pypi/support#985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on
https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI.
Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin

Install the wheel in the user scheme:

$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

Run the installed sage.

$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

also build a sagemath-standard wheel
build wheels for manylinux using tox -e docker-manylinux....
- wheel is much too big: 2.3GB
- auditwheel complains: RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:; so we should replace our current abuse of package data (install_data) by platlib (install_lib). pypa/build on Linux - extension / binary modules in purelib pypa/packaging-problems#542 (comment): subclass build_ext instead of build_py
disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible -- for example by storing a tar file sage_root.tar (with symlinks) instead of including the sage root as a directory - https://docs.python.org/3/library/tarfile.html#tarfile.open
modularized relocatable wheel version of sage_conf #31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-02-15T03:32:14Z

Branch: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf

sagetrac-git · 2021-02-15T06:36:08Z

Commit: 9573b7c

sagetrac-git · 2021-02-15T06:36:08Z

Branch pushed to git repo; I updated commit sha1. New commits:

`952fb9a`	`Fix up package_data`
`9573b7c`	`Closer to working`

mkoeppe · 2021-02-15T06:40:47Z

Author: Matthias Koeppe

mkoeppe · 2021-02-15T07:56:12Z

comment:4

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
	/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.
Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

sagetrac-git · 2021-02-16T02:56:32Z

Changed commit from 9573b7c to f785d3c

sagetrac-git · 2021-02-16T02:56:32Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f785d3c`	`src/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15`

mkoeppe · 2021-02-16T02:58:06Z

comment:6

Using -isysroot helps but giac configuration needs more work - #31403

mkoeppe · 2021-02-16T03:26:34Z

comment:7

Replying to @mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

mkoeppe · 2021-02-19T00:01:30Z

Changed dependencies from #29039 to #29039, #31409

sagetrac-git · 2021-02-19T00:02:07Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6da7ba2`	`Add variable TARGETS`
`b2d4ab8`	`build/pkgs/{r,rpy2}: Downgrade to optional`
`8732076`	`src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2`
`dbdf084`	`src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2`
`737b21c`	`src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2`
`4403924`	`src/sage/interfaces/r.py: Mark all tests # optional - rpy2`
`e56a2a9`	`Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf`

sagetrac-git · 2021-02-19T00:02:07Z

Changed commit from f785d3c to e56a2a9

sagetrac-git · 2021-02-19T02:19:30Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a228974`	`src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building`

sagetrac-git · 2021-02-19T02:19:30Z

Changed commit from e56a2a9 to a228974

sagetrac-git · 2021-02-19T03:35:35Z

Changed commit from a228974 to c260942

sagetrac-git · 2021-02-19T03:35:35Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0cac965`	`build/pkgs/ecl: Update to 21.2.1, remove patches included in new version`
`6a62484`	`delete all merged upstream patches: MR 210, MR 231`
`62fac00`	`Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c260942`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url`

sagetrac-git · 2021-02-19T06:10:23Z

Changed commit from c260942 to 4f17a04

sagetrac-git · 2021-02-19T06:10:23Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`dc5e225`	`build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS`
`df17ad4`	`tox.ini: Add configuration factors for specific homebrew python3.x`
`914ea58`	`build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST`
`986739b`	`m4/sage_check_python_for_venv.m4: Fix reason`
`0b3e70d`	`build/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it`
`372adcb`	`.github/workflows/tox.yml: Update xcode versions`
`fc8b676`	`SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting`
`5ae8044`	`Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c5c73a9`	`src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode`
`4f17a04`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary`

sagetrac-git · 2021-02-19T06:40:05Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8514352`	`src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps`

sagetrac-git · 2021-02-19T06:40:05Z

Changed commit from 4f17a04 to 8514352

sagetrac-git · 2021-03-15T06:02:00Z

Changed commit from 8514352 to 1baf390

sagetrac-git · 2021-03-15T06:02:00Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`c3e4093`	`Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES`
`4916415`	`Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default`
`a35428e`	`Merge #30383`
`e098133`	`src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS`
`5dbd712`	`src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built`
`839c534`	`src/pkgs/sage_conf-pypi/setup.py: Fix typo`
`100ff51`	`src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src`
`afa2ffd`	`Add configure option --disable-sagelib`
`a4f75b8`	`src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README`
`1baf390`	`Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf`

sagetrac-git · 2022-07-11T03:48:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e4d36b5`	`pkgs/sage-conf_relocatable/setup.py: Ignore more files when copying sage_root`

saraedum · 2022-07-14T20:22:17Z

comment:179

There's an isolated

+        #

Is a comment missing there?

saraedum · 2022-07-14T20:25:57Z

comment:180

I don't understand the STICKY business. Why is there a hardcoded /var/tmp there?

saraedum · 2022-07-14T20:26:22Z

comment:181

Is the plan to eventually make these wheels available on PyPI?

saraedum · 2022-07-14T20:27:24Z

Reviewer: Julian Rüth

saraedum · 2022-07-14T20:27:24Z

comment:182

Note to self. I don't really understand what pkgs/sage-conf_relocatable/setup.py does because of the above. The rest looks fine.

mkoeppe · 2022-07-14T20:38:16Z

comment:183

/var/tmp is a sticky directory, meaning every user can create a new subdirectory, which then securely belongs to them.

The README of https://pypi.org/project/sage-conf/ (section "sage_conf wheels") explains more

mkoeppe · 2022-07-14T20:46:04Z

comment:184

Replying to @saraedum:

Is the plan to eventually make these wheels available on PyPI?

Ticket description explains why they can't go to PyPI.

sagetrac-git · 2022-07-27T01:53:39Z

Changed commit from e4d36b5 to 2eeb402

sagetrac-git · 2022-07-27T01:53:39Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ecc4a90`	`Merge tag '9.7.beta6' into t/33817/gh_actions__add_test_of_the_modularized_distributions`
`e82312a`	`pkgs/sagemath-{environment,repl}/tox.ini: Update from sagemath-objects`
`8358976`	`src/bin/sage-update-version: Update install-requires.txt for the distribution packages in pkgs/`
`b79b5c6`	`build/pkgs/sagemath_objects/dependencies: Restore comment lost in merge`
`7bb6dee`	`build/pkgs/python_build: Rename from build/pkgs/build`
`52e8c41`	`build/bin/sage-build-env[-config]: Do not override SAGE_EDITABLE if it is set already`
`64b212c`	`pkgs/sagemath-*/tox.ini: Make sagepython-norequirements actually use sage python`
`2eeb402`	`Merge #33817`

saraedum · 2022-12-14T22:35:30Z

comment:189

Sorry for not having a look at this ticket for a while. There are merge conflicts now btw.

sagetrac-git · 2022-12-15T01:16:24Z

Changed commit from 2eeb402 to 1b2d4be

sagetrac-git · 2022-12-15T01:16:24Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1b2d4be`	`Merge tag '9.8.beta5' into t/31396/relocatable_wheel_version_of_package_sage_conf`

mkoeppe · 2023-09-29T22:08:35Z

Replaced by #36366.

mkoeppe added this to the sage-9.3 milestone Feb 14, 2021

mkoeppe added c: relocation labels Feb 14, 2021

This comment has been minimized.

Sign in to view

saraedum added s: needs info and removed s: needs review labels Jul 14, 2022

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs info labels Jul 27, 2022

mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022

saraedum added s: needs work and removed s: needs review labels Dec 14, 2022

mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023

mkoeppe removed this from the sage-10.0 milestone Apr 30, 2023

mkoeppe added a commit to mkoeppe/sage that referenced this issue Sep 29, 2023

pkgs/sage-conf_relocatable: New (squashed from sagemath#31396)

4d98583

mkoeppe closed this as not planned Won't fix, can't repro, duplicate, stale Sep 29, 2023

github-actions bot removed the s: needs work label Sep 29, 2023

mkoeppe added a commit to mkoeppe/sage that referenced this issue Oct 22, 2024

pkgs/sage-conf_relocatable: New (squashed from sagemath#31396)

85dc489

relocatable wheel version of package sage_conf #31396

relocatable wheel version of package sage_conf #31396

Comments

mkoeppe commented Feb 14, 2021

mkoeppe commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

This comment has been minimized.

mkoeppe commented Feb 15, 2021

mkoeppe commented Feb 15, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 16, 2021

sagetrac-git mannequin commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Mar 15, 2021

sagetrac-git mannequin commented Mar 15, 2021

sagetrac-git mannequin commented Jul 11, 2022

saraedum commented Jul 14, 2022

saraedum commented Jul 14, 2022

saraedum commented Jul 14, 2022

saraedum commented Jul 14, 2022

saraedum commented Jul 14, 2022

mkoeppe commented Jul 14, 2022

mkoeppe commented Jul 14, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jul 27, 2022

sagetrac-git mannequin commented Jul 27, 2022

saraedum commented Dec 14, 2022

sagetrac-git mannequin commented Dec 15, 2022

sagetrac-git mannequin commented Dec 15, 2022

mkoeppe commented Sep 29, 2023