Replace "$SAGE_LOCAL/bin" by more specific variables to make Sage easier to package, use in venvs

[tobihan]

Various scripts refer to "$SAGE_LOCAL/bin" in a way that conflates several different uses. We introduce new variables to give packagers more flexibility without having to patch, and to enable new installation schemes, including the use of Sage in users' venvs.

1a. Introduce $SAGE_BIN as the place where the binaries of all the SPKGs get installed to. Default is $SAGE_LOCAL/bin/.

1b. (Wishlist item:) Make it configurable through ./configure --bindir, but this would need major changes to spkg-configure scripts. (Supporting --bindir would be another step toward more autotools compliance, see Meta-ticket #21566.)

2a. Scripts such as src/bin/sage, src/bin/sage-eval are installed by the sagelib package using the scripts facility of distutils/setuptools. If sagelib is installed into a venv, these scripts are installed into the bin directory of the venv. For the default installation in sage-the-distribution --with-system-python, this venv is identical with SAGE_LOCAL. But if other venvs are created by the user, it will be unrelated to SAGE_LOCAL; in fact, SAGE_LOCAL may be undefined (in distribution packaging and when installing sagelib through pip). As proposed in #29013, we use the variable SAGE_VENV for this, so these scripts would be referred to as $SAGE_VENV/bin/sage-eval etc.

2b. Referring to the python3 used for sage. Likewise, this comes from the venv, not $SAGE_LOCAL/bin. So it should be referred to as $SAGE_VENV/bin/python3.

2c. The subset of the scripts in src/bin, currently installed by the sagelib package, that are "libexec"-like (not user-facing). Distribution packagers would like to hide these scripts. (In the old description of this ticket, it was proposed to use $SAGE_SCRIPTS_DIR for this, but a more specific name is needed: We will use $SAGE_VENV_SCRIPTS_DIR.)

In this ticket, we introduce SAGE_VENV for 2abc, ignoring the "libexec" aspect (using $SAGE_VENV/bin directly instead of going through SAGE_VENV_SCRIPTS_DIR). Items 1 and the libexec aspect of 2c can be done on follow-up tickets.

We also make sage-env-config optional from the viewpoint of the other scripts. This is preparation for #29850 / #29852.

See also:

• https://salsa.debian.org/science-team/sagemath/-/blob/master/debian/patches/u1-scripts-dir.patch
• src/bin/sage-env: Make sure $SAGE_VENV/bin is at the beginning of the PATH #30013 src/bin/sage-env: Make sure $SAGE_VENV/bin is at the beginning of the PATH
• src/bin/sage-env: Make SAGE_ROOT and SAGE_LOCAL optional #29951 src/bin/sage-env: Make SAGE_ROOT and SAGE_LOCAL optional
• Add src/requirements.txt for installation of sagelib in a venv #30578 Add src/requirements.txt for installation of sagelib in a venv

Depends on #29951

CC: @mkoeppe @jhpalmieri @orlitzky @tobiasdiez @isuruf @antonio-rojas @mwageringel

Component: porting

Author: Matthias Koeppe

Branch/Commit: 38eebc3

Reviewer: Tobias Diez, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/22731

[tobihan]

Branch: u/thansen/allow_installation_of_sage_scripts_to_an_arbotrary_location_by_using_sage_scripts_dir_

[tobihan]

Author: Tobias Hanen

[tobihan]

Commit: 6ae9bf1

[tobihan]

New commits:

6ae9bf1

Use SAGE_SCRIPTS_DIR in more places and define a fallback in env.py.

[tobihan]

Changed author from Tobias Hanen to Tobias Hansen

[jdemeyer]

comment:4

The name SAGE_SCRIPTS_DIR refers to the scripts of Sage itself. It is only meant to find sage-env and friends, nothing more.

Now, if you want a invent a new variable with the purpose of defining the bindir (to replace SAGE_LOCAL/bin), that's fine. Just don't call it SAGE_SCRIPTS_DIR.

[saraedum]

Changed branch from u/thansen/allow_installation_of_sage_scripts_to_an_arbotrary_location_by_using_sage_scripts_dir_ to u/saraedum/allow_installation_of_sage_scripts_to_an_arbotrary_location_by_using_sage_scripts_dir_

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

502f6ab

Rename SAGE_SCRIPTS_DIR to SAGE_BIN in places where we are not looking for sage-env's friends

[sagetrac-git]

Changed commit from 6ae9bf1 to 502f6ab

[saraedum]

comment:7

I have not had a chance to try this out locally (as my Sage is rebuilding.) Is this more or less what you had in mind jdemeyer?

[saraedum]

Changed author from Tobias Hansen to Tobias Hansen, Julian Rüth

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

07649aa	Replace more occurences of SAGE_LOCAL/bin with SAGE_BIN
7b6d51f	Fixed typos related to SAGE_BIN

[sagetrac-git]

Changed commit from 502f6ab to 7b6d51f

[jdemeyer]

comment:9

It makes no sense to add both $SAGE_LOCAL/bin and $SAGE_BIN to the PATH. The whole point should be to replace $SAGE_LOCAL/bin by $SAGE_BIN.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

b91dbcb

Drop SAGE_LOCAL/bin from PATH

[mkoeppe]

comment:111

Thanks!

[vbraun]

Changed branch from u/mkoeppe/22731 to f3b7a9c

[vbraun]

comment:113

$ git clean -f -d -x .
$ ./bootstrap -d
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
src/doc/bootstrap:60: installing src/doc/en/installation/arch*.txt
src/doc/bootstrap:60: installing src/doc/en/installation/debian*.txt
src/doc/bootstrap:60: installing src/doc/en/installation/fedora*.txt
src/doc/bootstrap:60: installing src/doc/en/installation/cygwin*.txt
src/doc/bootstrap:60: installing src/doc/en/installation/homebrew*.txt
src/doc/bootstrap:70: installing src/doc/en/reference/spkg/*.rst
src/doc/bootstrap:102: installing src/doc/en/reference/repl/options.txt
Error: You must set either the SAGE_LOCAL or SAGE_SCRIPTS_DIR environment variable to run this
Error setting environment variables by sourcing '/Users/vbraun/Sage/src/bin/sage-env';
possibly contact sage-devel (see http://groups.google.com/group/sage-devel).

[vbraun]

Changed branch from f3b7a9c to u/mkoeppe/22731

[kiwifb]

comment:114

I noticed it as well in sage-on-gentoo. I was trying to minimize the patch to bin/sage but that got me. This is ultimately caused by the last line of src/doc/bootstrap

./sage -advanced > "$OUTPUT"

Why is this line needed, I am wondering?

[mkoeppe]

comment:115

Replying to @kiwifb:

This is ultimately caused by the last line of src/doc/bootstrap

./sage -advanced > "$OUTPUT"

Why is this line needed, I am wondering?

It generates part of the documentation.

[dimpase]

comment:116

Is at the error point the value of SAGE_SCRIPTS_DIR already known?

[kiwifb]

comment:117

I see how it is used now. And no, SAGE_SCRIPTS_DIR or anything like that is not known, we are bootstrapping the documentation, which you should be able to do before running configure.

Some dummy values could be set in the bootstrap script on the assumption you do that on a clean tree.

[mkoeppe]

comment:118

The problem is solved in #30013, so I need to move some of the changes to this ticket.

[kiwifb]

comment:119

Replying to @mkoeppe:

The problem is solved in #30013, so I need to move some of the changes to this ticket.

Yes, it looks like it would help.

[mkoeppe]

comment:120

I will work on this after the next beta comes out.

[dimpase]

comment:121

Anyhow, just in case, here one could do

--- a/bootstrap
+++ b/bootstrap
@@ -113,6 +113,8 @@ SAGE_SPKG_CONFIGURE_$(echo ${pkgname} | tr '[a-z]' '[A-Z]')"
     # ONLY stderr, and to re-output the results back to stderr leaving
     # stdout alone. Basically we swap the two descriptors using a
     # third, filter, and then swap them back.
+    touch src/bin/sage-env-config
+    SAGE_SCRIPTS_DIR=./src/bin
     BOOTSTRAP_QUIET="${BOOTSTRAP_QUIET}" \
     SAGE_ROOT="$SAGE_ROOT" \
     src/doc/bootstrap && \

and it all goes through.

[sagetrac-git]

Changed commit from f3b7a9c to 1fe77a8

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1fe77a8

src/bin/sage-env: Do not set SAGE_SCRIPTS_DIR

[mkoeppe]

comment:123

Cherry-picked a commit from the original branch of #30888, where it was observed that
SAGE_SCRIPTS_DIR is currently unused because the feature introduced in #25486 ("Discover SAGE_SCRIPTS_DIR to make $SAGE_LOCAL/bin/sage work from any directory, in an environment without SAGE_* variables") was killed by #30128.
(#30888 will re-introduce this feature using a different mechanism.)

[sagetrac-git]

Changed commit from 1fe77a8 to 38eebc3

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

38eebc3

Merge tag '9.3.beta2' into t/22731/22731

[mkoeppe]

comment:125

Merged the latest beta. Ready for another round of review.

[dimpase]

comment:126

this works, after "git clean" as above, too.

[mkoeppe]

comment:127

Thanks!

[vbraun]

Changed branch from u/mkoeppe/22731 to 38eebc3