Install packages in temporary root before copying to $SAGE_LOCAL

[embray]

Currently we build packages, in most cases, by setting --prefix=$SAGE_LOCAL (in ./configure, or the equivalent in non-autoconf packages), and then installing directly to that prefix.

This is fine to keep as is. But with automake packages we can also set DESTDIR (https://www.gnu.org/software/automake/manual/html_node/DESTDIR.html#DESTDIR) to an alternative (typically temporary) directory into which to install. This has a couple advantages:

• Although it's not as usual for make install to fail after a successful make, it can happen. This prevents messy partial installs in the case of a failed make install run.

• It gives us the opportunity to make an exact list of the files that were installed. This is a prerequisite to improving package uninstallation/reinstallation in sage-dist (see Enhanced package uninstallation for Sage spkgs #22510).

This is also standard operating procedure in most other packaging systems, so it would be good for Sage to adopt too. The general approach is the same as that used for converting an arbitrary Makefile to support this convention--everywhere a file is installed, prefix the installation path with $(DESTDIR) (or in this case a variable we'll call $SAGE_DESTDIR).

For packages that already support the DESTDIR convention, then, we pass DESTDIR=$SAGE_DESTDIR to make install. It is also already possible for Python packages (e.g. pip install --root).

Since the introduction of the build helper scripts in #23160, it makes the most sense to make these changes directly in the helper scripts, and then update more packages to use those helper scripts where possible.

Dependent tickets:

• Install packages in temporary root before copying to $SAGE_LOCAL (simplified) #24106 - implement the basic functionality of supporting staged installs through sage-spkg
• Meta-ticket: Update other packages to using the sage-dist-helpers functions + DESTDIR where applicable #24024 - update more spkg-build/install scripts to use the sdh_ helper functions, since that is the fastest way to add DESTDIR support for many packages.
• Add support for staged installations in packages that use sdh_pip_install #24646 - add support for --root=${SAGE_DESTDIR} to Python spkgs
• Support for spkg-postinst scripts #24645 - add support for post-install scripts; this can be useful for cases where, for example, local files are generated after the package is installed, or to handle cases where one file is shared by multiple packages and needs to be updated when a package that uses it is installed (see e.g. #22510 comment:5)
• #????? - convert all spkgs to support staged-install; for many packages this can be covered by using the helper functions, but some packages require additional steps; in particular, packages that have files that are manually copied into $SAGE_LOCAL, rather than through a standard installer

Depends on #23059
Depends on #23096
Depends on #23160
Depends on #23781

CC: @mkoeppe

Component: build

Author: Erik Bray

Branch/Commit: u/embray/build/destdir @ c1d5453

Issue created by migration from https://trac.sagemath.org/ticket/22509

[embray]

Author: Erik Bray

[embray]

Dependencies: #23096

[embray]

Commit: a25a42a

[embray]

Branch: u/embray/build/destdir

[embray]

comment:3

Started work on this, and converted I think all standard packages to work with a staged installation (install first to a temporary directory, then copy from the temp directory into $SAGE_LOCAL).

As I wrote in the description, most packages are easy--either they use automake, or at least a Makefile that's compatible with the DESTDIR convention, or they are Python packages in which case the --root option does the same thing. A handful of packages needed hand-tweaking.

This is currently implemented in such a way that if a package doesn't yet support staged install, it will still install just fine. The only major downside is that the installed files are not recorded yet, so they wouldn't be able to take advantage of improved uninstall.

That said, I'm going to continue converting the remaining packages. My only question is whether to continue work in this ticket, or separate the general changes (i.e. to sage-spkg) from the updates to the packages themselves.

---

Last 10 new commits:

3fa088a	Update pip install to use --root
30aad2f	Update freetype and pkgconf to use DESTDIR
b1aabaa	Updated various packages with non-standard installers or that are otherwise
223608f	Update cephes to work with staged install.
27878c3	Update openblas to use staged install
ecd214c	Update lcalc to support staged install
3d64db8	Update zn_poly, which doesn't really have a proper installer, to use staged install
609393a	Roll back accidental change from dc46ea2dfa97be79d7ef8ddbe9130d4c717acc3d that broke build of Python 2 on Cygwin.
03424e1	Merge remote-tracking branch 'upstream/u/embray/cygwin/ticket-23059' into u/embray/build/destdir
a25a42a	Updated the python2/3 pacakges to use staged install.

[embray]

Changed dependencies from #23096 to #23059 #23096

[embray]

comment:5

By way of explanation, the idea here is that there is a variable exported by sage-spkg called $SAGE_INST_TEMP (the name is open to bikeshedding), so this is expected to be passed to each spkg-install to use an alternative install root. In principle this could be defined in each spkg-install individually, but it's less trouble to standardize in one place, and in any case sage-spkg still needs to know this path to copy files from it to $SAGE_LOCAL.

Of course, if we take this approach, the developer docs will also need to be updated with some guidance on how to implement staged installation for new packages.

[sagetrac-git]

Changed commit from a25a42a to 8a621af

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3a32187	Roll back accidental change from dc46ea2dfa97be79d7ef8ddbe9130d4c717acc3d that broke build of Python 2 on Cygwin.
8a621af	Updated the python2/3 pacakges to use staged install.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1391089

Remove the '-v' from the 'cp' command.

[sagetrac-git]

Changed commit from 8a621af to 1391089

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e294123

Add staged install support for atlas

[sagetrac-git]

Changed commit from 1391089 to e294123

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

00f6de8

This file should be deleted from the temp install dir during staged install on OSX

[sagetrac-git]

Changed commit from e294123 to 00f6de8

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

16bc700	Update freetype and pkgconf to use DESTDIR
eb660c7	Updated various packages with non-standard installers or that are otherwise
8de16d2	Update cephes to work with staged install.
73c45e6	Update openblas to use staged install
e1ad6d5	Update lcalc to support staged install
f07e056	Update zn_poly, which doesn't really have a proper installer, to use staged install
9997a37	Updated the python2/3 pacakges to use staged install.
f9cfa6f	Remove the '-v' from the 'cp' command.
e51f578	Add staged install support for atlas
48e9ffb	This file should be deleted from the temp install dir during staged install on OSX

[sagetrac-git]

Changed commit from 00f6de8 to 48e9ffb

[jdemeyer]

comment:13

This already fails on the first package... patch:

Making install in src
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
 /bin/mkdir -p '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/bin'
  /usr/bin/install -c patch '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/bin'
make[6]: Nothing to be done for `install-data-am'.
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/src'
Making install in tests
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[6]: Nothing to be done for `install-exec-am'.
make[6]: Nothing to be done for `install-data-am'.
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src/tests'
make[5]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[6]: Entering directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[6]: Nothing to be done for `install-exec-am'.
 /bin/mkdir -p '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/share/man/man1'
 /usr/bin/install -c -m 644 'patch.man' '/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/inst/opt/sage/sage-git/local/share/man/man1/patch.1'
make[6]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[5]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[4]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
make[3]: Leaving directory `/opt/sage/sage-git/local/var/tmp/sage/build/patch-2.7.5/src'
Cannot find the patch program we just installed

[embray]

comment:110

Tickets still needing working or clarification should be moved to the next release milestone at the soonest (please feel free to revert if you think the ticket is close to being resolved).

[embray]

comment:111

Ticket retargeted after milestone closed

[mkoeppe]

comment:112

I suggest to close this ticket because its goals have been achieved and the remaining tasks seem to be covered by #24024.