Trac #33851: Script package _develop; improve installation instructio…

…ns in the manual

(from #29536)

This is like `_recommended` (from #29557), but for development tools.

Define a script package representing "useful stuff for Sage developers
to have on a Docker image":

- include the stuff added in #33740 to `environment-dev.yml`
- include what is needed on VS Code devcontainers - #33671

We also use this package to define packages that would appear as
"packages that developers should install" in
https://doc.sagemath.org/html/en/installation/source.html#linux-
recommended-installation;
the new version from this ticket can be seen in
https://c96c7c5813bd70d7b2d8d51135d57cc6acba18ad--sagemath-
tobias.netlify.app/installation/source.html#system-specific-requirements

We also simplify the `bootstrap-conda` script and make it a bit faster.

URL: https://trac.sagemath.org/33851
Reported by: mkoeppe
Ticket author(s): Matthias Koeppe
Reviewer(s): Samuel Lelièvre

README.md

@@ -207,8 +207,8 @@ in the Installation Guide.
     [void.txt](build/pkgs/_prereq/distros/void.txt), or visit
     https://doc.sagemath.org/html/en/reference/spkg/_prereq.html#spkg-prereq
 
-4.  [Git] If you plan to work with ticket branches that make changes
-    to packages, install the bootstrapping prerequisites. See the
+4.  [Git] If you plan to do Sage development or otherwise work with ticket branches
+    and not only releases, install the bootstrapping prerequisites. See the
     files in the folder
     [build/pkgs/_bootstrap/distros](build/pkgs/_bootstrap/distros), or
     visit

bootstrap-conda

@@ -7,14 +7,17 @@
 export PATH="$(pwd)/build/bin:$PATH"
 
 STRIP_COMMENTS="sed s/#.*//;"
-RECOMMENDED_SPKG_PATTERN="@(_recommended$(for a in $(head -n 1 build/pkgs/_recommended/dependencies); do echo -n "|"$a; done))"
+
+shopt -s extglob
+
+DEVELOP_SPKG_PATTERN="@(_develop$(for a in $(head -n 1 build/pkgs/_develop/dependencies); do echo -n "|"$a; done))"
 
 BOOTSTRAP_PACKAGES=$(echo $(${STRIP_COMMENTS} build/pkgs/_bootstrap/distros/conda.txt))
 SYSTEM_PACKAGES=
 OPTIONAL_SYSTEM_PACKAGES=
 SAGELIB_SYSTEM_PACKAGES=
 SAGELIB_OPTIONAL_SYSTEM_PACKAGES=
-RECOMMENDED_SYSTEM_PACKAGES=
+DEVELOP_SYSTEM_PACKAGES=
 for PKG_BASE in $(sage-package list --has-file distros/conda.txt); do
     PKG_SCRIPTS=build/pkgs/$PKG_BASE
     SYSTEM_PACKAGES_FILE=$PKG_SCRIPTS/distros/conda.txt
@@ -26,8 +29,8 @@ for PKG_BASE in $(sage-package list --has-file distros/conda.txt); do
                 *:standard)
                     SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
                     ;;
-                $RECOMMENDED_SPKG_PATTERN:*)
-                    RECOMMENDED_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                $DEVELOP_SPKG_PATTERN:*)
+                    DEVELOP_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
                     ;;
                 *)
                     OPTIONAL_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
@@ -46,36 +49,49 @@ for PKG_BASE in $(sage-package list --has-file distros/conda.txt); do
     fi
 done
 echo >&2 $0:$LINENO: generate conda environment files
-echo "name: sage-build" >  environment.yml
-echo "channels:"        >> environment.yml
-echo "  - conda-forge"  >> environment.yml
-echo "  - nodefaults"   >> environment.yml
-echo "dependencies:"    >> environment.yml
-for pkg in $SYSTEM_PACKAGES; do
-    echo "  - $pkg" >> environment.yml
-done
-echo "  # Packages needed for ./bootstrap" >> environment.yml
-for pkg in $BOOTSTRAP_PACKAGES; do
-    echo "  - $pkg" >> environment.yml
-done
-sed 's/name: sage-build/name: sage/' environment.yml > src/environment.yml
-for pkg in $SAGELIB_SYSTEM_PACKAGES; do
-    echo "  - $pkg" >> src/environment.yml
-done
-sed 's/name: sage/name: sage-dev/' src/environment.yml > src/environment-dev.yml
-echo "  # Additional dev tools" >> src/environment-dev.yml
-echo "  - openssh" >> src/environment-dev.yml
-echo "  - pycodestyle" >> src/environment-dev.yml
-echo "  - pytest" >> src/environment-dev.yml
-echo "  - esbonio" >> src/environment-dev.yml
+(
+    echo "name: sage-build"
+    echo "channels:"
+    echo "  - conda-forge"
+    echo "  - nodefaults"
+    echo "dependencies:"
+    for pkg in $SYSTEM_PACKAGES; do
+        echo "  - $pkg"
+    done
+    echo "  # Packages needed for ./bootstrap"
+    for pkg in $BOOTSTRAP_PACKAGES; do
+        echo "  - $pkg"
+    done
+) > environment.yml
 
-cp environment.yml environment-optional.yml
-    echo "  # optional packages" >> environment-optional.yml
-for pkg in $OPTIONAL_SYSTEM_PACKAGES; do
-    echo "  - $pkg" >> environment-optional.yml
+(
+    sed 's/name: sage-build/name: sage/' environment.yml
+    echo "  # Additional packages providing all dependencies for the Sage library"
+    for pkg in $SAGELIB_SYSTEM_PACKAGES; do
+        echo "  - $pkg"
     done
-cp src/environment.yml src/environment-optional.yml
-    echo "  # optional packages" >> src/environment-optional.yml
-for pkg in $OPTIONAL_SYSTEM_PACKAGES $SAGELIB_OPTIONAL_SYSTEM_PACKAGES; do
-    echo "  - $pkg" >> src/environment-optional.yml
-done
+) > src/environment.yml
+
+(
+    sed 's/name: sage/name: sage-dev/' src/environment.yml
+    echo "  # Additional dev tools"
+    for pkg in $DEVELOP_SYSTEM_PACKAGES; do
+        echo "  - $pkg"
+    done
+) > src/environment-dev.yml
+
+(
+    cat environment.yml
+    echo "  # optional packages"
+    for pkg in $OPTIONAL_SYSTEM_PACKAGES; do
+        echo "  - $pkg"
+    done
+) > environment-optional.yml
+
+(
+    cat src/environment.yml
+    echo "  # optional packages"
+    for pkg in $OPTIONAL_SYSTEM_PACKAGES $SAGELIB_OPTIONAL_SYSTEM_PACKAGES; do
+        echo "  - $pkg"
+    done
+) > src/environment-optional.yml

build/pkgs/_develop/SPKG.rst

@@ -0,0 +1,7 @@
+\_develop: Represents system packages recommended for development
+=================================================================
+
+Description
+-----------
+
+Script package representing a list of system packages recommended for developers.

build/pkgs/_develop/dependencies

@@ -0,0 +1 @@
+_bootstrap git pytest pytest_xdist

build/pkgs/_develop/distros/alpine.txt

@@ -0,0 +1,2 @@
+gnupg-gpgconf
+openssh-client

build/pkgs/_develop/distros/arch.txt

@@ -0,0 +1,2 @@
+gnupg
+openssh

build/pkgs/_develop/distros/conda.txt

@@ -0,0 +1,3 @@
+openssh
+pycodestyle
+esbonio

build/pkgs/_develop/distros/cygwin.txt

@@ -0,0 +1 @@
+gnupg2

build/pkgs/_develop/distros/debian.txt

@@ -0,0 +1,4 @@
+# Needed for devcontainer support in VS code
+gpgconf
+openssh-client
+openssh

build/pkgs/_develop/distros/fedora.txt

@@ -0,0 +1,2 @@
+gnupg2
+openssh

build/pkgs/_develop/distros/freebsd.txt

@@ -0,0 +1,2 @@
+security/gnupg
+security/openssh-portable

build/pkgs/_develop/distros/gentoo.txt

@@ -0,0 +1,2 @@
+app-crypt/gnupg
+net-misc/openssh

build/pkgs/_develop/distros/homebrew.txt

@@ -0,0 +1 @@
+gnupg

build/pkgs/_develop/distros/macports.txt

@@ -0,0 +1 @@
+gnupg2

build/pkgs/_develop/distros/nix.txt

@@ -0,0 +1,2 @@
+gnupg
+openssh

build/pkgs/_develop/distros/opensuse.txt

@@ -0,0 +1,2 @@
+gpg2
+openssh

build/pkgs/_develop/distros/repology.txt

@@ -0,0 +1,2 @@
+gnupg
+openssh

build/pkgs/_develop/distros/slackware.txt

@@ -0,0 +1,2 @@
+gnupg2
+openssh

build/pkgs/_develop/distros/void.txt

@@ -0,0 +1,2 @@
+gnupg2
+openssh

build/pkgs/_develop/spkg-configure.m4

@@ -0,0 +1,3 @@
+SAGE_SPKG_CONFIGURE([_develop], [
+  sage_spkg_install__develop=yes
+])

build/pkgs/_develop/spkg-install

@@ -0,0 +1,2 @@
+#! /usr/bin/env bash
+# Nothing to do

build/pkgs/_develop/type

@@ -0,0 +1 @@
+optional

src/doc/bootstrap

@@ -26,13 +26,15 @@ mkdir -p "$OUTPUT_DIR"
 shopt -s extglob
 
 RECOMMENDED_SPKG_PATTERN="@(_recommended$(for a in $(head -n 1 build/pkgs/_recommended/dependencies); do echo -n "|"$a; done))"
+DEVELOP_SPKG_PATTERN="@(_develop$(for a in $(head -n 1 build/pkgs/_develop/dependencies); do echo -n "|"$a; done))"
 
-for SYSTEM in arch debian fedora cygwin homebrew; do
+for SYSTEM in arch debian fedora cygwin homebrew opensuse; do
     SYSTEM_PACKAGES=
     OPTIONAL_SYSTEM_PACKAGES=
     SAGELIB_SYSTEM_PACKAGES=
     SAGELIB_OPTIONAL_SYSTEM_PACKAGES=
     RECOMMENDED_SYSTEM_PACKAGES=
+    DEVELOP_SYSTEM_PACKAGES=
     for PKG_BASE in $(sage-package list --has-file distros/$SYSTEM.txt); do
                 PKG_SCRIPTS=build/pkgs/$PKG_BASE
                 SYSTEM_PACKAGES_FILE=$PKG_SCRIPTS/distros/$SYSTEM.txt
@@ -44,6 +46,9 @@ for SYSTEM in arch debian fedora cygwin homebrew; do
                            *:standard)
                                SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
                                ;;
+                           $DEVELOP_SPKG_PATTERN:*)
+                               DEVELOP_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                               ;;
                            $RECOMMENDED_SPKG_PATTERN:*)
                                RECOMMENDED_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
                                ;;
@@ -52,8 +57,11 @@ for SYSTEM in arch debian fedora cygwin homebrew; do
                                ;;
                        esac
                     else
-                        case "$PKG_TYPE" in
-                            standard)
+                        case "$PKG_BASE:$PKG_TYPE" in
+                           $DEVELOP_SPKG_PATTERN:*)
+                               DEVELOP_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                               ;;
+                            *:standard)
                                 SAGELIB_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
                                 ;;
                             *)
@@ -70,6 +78,7 @@ for SYSTEM in arch debian fedora cygwin homebrew; do
     echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM.txt
     echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $OPTIONAL_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-optional.txt
     echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $RECOMMENDED_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-recommended.txt
+    echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $DEVELOP_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-develop.txt
 done
 
 OUTPUT_DIR="src/doc/en/reference/spkg"

src/doc/en/installation/source.rst

@@ -126,7 +126,7 @@ processor. On Linux, this means you need a recent version of
 Python for venv
 ^^^^^^^^^^^^^^^
 
-By default, Sage will try to use system's `python3` to set up a virtual
+By default, Sage will try to use system's ``python3`` to set up a virtual
 environment, a.k.a. `venv <https://docs.python.org/3.10/library/venv.html>`_
 rather than building a Python 3 installation from scratch.
 Use the ``configure`` option ``--without-system-python3`` in case you want Python 3
@@ -168,6 +168,22 @@ On Redhat-derived systems not all perl components are installed by
 default and you might have to install the ``perl-ExtUtils-MakeMaker``
 package.
 
+On Linux systems (e.g., Ubuntu, Redhat, etc), ``ar`` and ``ranlib`` are in the
+`binutils <https://www.gnu.org/software/binutils/>`_ package.
+The other programs are usually located in packages with their respective names.
+Assuming you have sufficient privileges, you can install the ``binutils`` and
+other necessary/standard components. The lists provided below are longer than
+the minimal prerequisites, which are basically ``binutils``, ``gcc``/``clang``, ``make``,
+``tar``, but there is no real need to build compilers and other standard tools
+and libraries on a modern Linux system, in order to be able to build Sage.
+If you do not have the privileges to do this, ask your system administrator to
+do this, or build the components from source code.
+The method of installing additional software varies from distribution to
+distribution, but on a `Debian <https://www.debian.org/>`_ based system (e.g.
+`Ubuntu <https://www.ubuntu.com/>`_ or `Mint <https://www.linuxmint.com/>`_),
+you would use
+:wikipedia:`apt-get <Advanced_Packaging_Tool>`.
+
 Installing prerequisites
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -186,55 +202,83 @@ either ``perl`` is not installed, or it is installed but not in your
 
 .. _sec-installation-from-sources-linux-recommended-installation:
 
-Linux recommended installation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-On Linux systems (e.g., Ubuntu, Redhat, etc), ``ar`` and ``ranlib`` are in the
-`binutils <https://www.gnu.org/software/binutils/>`_ package.
-The other programs are usually located in packages with their respective names.
-Assuming you have sufficient privileges, you can install the ``binutils`` and
-other necessary/standard components. The lists provided below are longer than
-the minimal prerequisites, which are basically ``binutils``, ``gcc``/``clang``, ``make``,
-``tar``, but there is no real need to build compilers and other standard tools
-and libraries on a modern Linux system, in order to be able to build Sage.
-If you do not have the privileges to do this, ask your system administrator to
-do this, or build the components from source code.
-The method of installing additional software varies from distribution to
-distribution, but on a `Debian <https://www.debian.org/>`_ based system (e.g.
-`Ubuntu <https://www.ubuntu.com/>`_ or `Mint <https://www.linuxmint.com/>`_),
-you would use
-:wikipedia:`apt-get <Advanced_Packaging_Tool>`.
+Debian/Ubuntu prerequisite installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 On Debian ("buster" or newer) or Ubuntu ("bionic" or newer):
 
 .. literalinclude:: debian.txt
 
-On Fedora / Redhat / CentOS:
+If you wish to do Sage development, additionally install the following:
 
-.. literalinclude:: fedora.txt
+.. literalinclude:: debian-develop.txt
 
-On Arch Linux:
+For all users, we recommend the following:
 
-.. literalinclude:: arch.txt
+.. literalinclude:: debian-recommended.txt
 
 In addition to these, if you don't want Sage to build optional packages that might
 be available from your OS, cf. the growing list of such packages on :trac:`27330`,
-install on Debian ("buster" or newer) or Ubuntu ("bionic" or newer):
+install:
 
 .. literalinclude:: debian-optional.txt
 
-On Fedora / Redhat / CentOS:
+Fedora/Redhat/CentOS prerequisite installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. literalinclude:: fedora.txt
+
+If you wish to do Sage development, additionally install the following:
+
+.. literalinclude:: fedora-develop.txt
+
+For all users, we recommend the following:
+
+.. literalinclude:: fedora-recommended.txt
+
+In addition to these, if you don't want Sage to build optional packages that might
+be available from your OS, cf. the growing list of such packages on :trac:`27330`,
+install:
 
 .. literalinclude:: fedora-optional.txt
 
-On Arch Linux:
+Arch Linux prerequisite installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. literalinclude:: arch.txt
+
+If you wish to do Sage development, additionally install the following:
+
+.. literalinclude:: arch-develop.txt
+
+For all users, we recommend the following:
+
+.. literalinclude:: arch-recommended.txt
+
+In addition to these, if you don't want Sage to build optional packages that might
+be available from your OS, cf. the growing list of such packages on :trac:`27330`,
+install:
 
 .. literalinclude:: arch-optional.txt
 
-On other Linux systems, you might use
-:wikipedia:`rpm <RPM_Package_Manager>`,
-:wikipedia:`yum <Yellowdog_Updater,_Modified>`,
-or other package managers.
+OpenSUSE prerequisite installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. literalinclude:: opensuse.txt
+
+If you wish to do Sage development, additionally install the following:
+
+.. literalinclude:: opensuse-develop.txt
+
+For all users, we recommend the following:
+
+.. literalinclude:: opensuse-recommended.txt
+
+In addition to these, if you don't want Sage to build optional packages that might
+be available from your OS, cf. the growing list of such packages on :trac:`27330`,
+install:
+
+.. literalinclude:: opensuse-optional.txt
 
 .. _section_macprereqs:
 
@@ -300,6 +344,14 @@ Sage, run ::
 command like this to your shell profile if you want the settings to
 persist between shell sessions.
 
+If you wish to do Sage development, additionally install the following:
+
+.. literalinclude:: homebrew-develop.txt
+
+For all users, we recommend the following:
+
+.. literalinclude:: homebrew-recommended.txt
+
 Some additional optional packages are taken care of by:
 
 .. literalinclude:: homebrew-optional.txt