Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL #29013

mkoeppe · 2020-01-14T22:26:27Z

#27824 made $SAGE_LOCAL a venv if a suitable system python3 is found.

In this ticket, we add support for installing builds of Sage with several python versions, sharing the non-Python packages in SAGE_LOCAL:

keeping install records for Python packages (recognized by having install-requires.txt or requirements.txt) in $SAGE_VENV/var/lib/sage/{installed,scripts,wheels}, separate from those of non-Python packages ($SAGE_LOCAL/var/lib/sage/{installed,scripts})
likewise for $SAGE_LOCAL/var/tmp/sage/build
here $SAGE_VENV defaults to $SAGE_LOCAL, but can be overridden to an arbitrary directory that will be used as the wheel-building venv, for example $SAGE_LOCAL/var/lib/sage/venv/$PYTHON_TAG.

Configuration:

This is activated by running, for example,
./configure --with-python=/usr/bin/python3.8 --with-sage-venv="$SAGE_LOCAL/var/lib/sage/venv/python3.8".

As of this ticket, we are able to establish the venv in an arbitrary configured directory instead of SAGE_LOCAL. This is already useful for #31396 to create a wheel-building venv that will not be packaged as part of SAGE_LOCAL.

In follow-up tickets, we gain the full functionality that facilitates testing with several Python versions without having to rebuild the Sage distribution. This depends on:

Repackage pynac as a pip-installable package #30534 Repackage pynac as a pip-installable package

Follow-ups:

We also want to support ./configure --with-sage-venv=no which would suppress making any venv (./configure --with-sage-venv=none #30896)
Support the tricky case: When system python3 is not in use and SAGE_VENV != SAGE_LOCAL, then both a real $SAGE_LOCAL/bin/python3 needs to be built and a venv in SAGE_VENV created... or should python3 be installed in $SAGE_VENV?)
Optional: A file build/pkgs/SPKG/trees could override the install tree determination - this would allow us to for example install a package such as jupyter_core both into something like SAGE_NOTEBOOK_VENV and SAGE_VENV...

CC: @dimpase @jdemeyer @embray @vbraun @jhpalmieri

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: d68e861

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29013

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-01-15T01:19:58Z

Author: Matthias Koeppe

mkoeppe · 2020-01-15T01:52:39Z

Branch: public/29013-use-py3-venv

mkoeppe · 2020-01-15T01:55:49Z

comment:3

A simple change. Compiles from a fresh checkout and runs without problems on macOS.

New commits:

`1363425`	`In a python3 build, install all Python packages into a venv`
`a394268`	`sage-pip-install: Use PYTHON=sage-python23`

mkoeppe · 2020-01-15T01:55:49Z

Commit: a394268

mkoeppe · 2020-01-15T04:47:29Z

comment:4

In the branch #27824, the venv is created by slightly more complicated code in build/make/Makefile. This complexity is not necessary for the present ticket.

dimpase · 2020-01-15T10:21:32Z

comment:5

Does this ticket make sense separately, or only as part of #27824 ?

mkoeppe · 2020-01-15T14:08:42Z

comment:6

#27824 is the use case that I have in mind.
This ticket is a less intrusive change and should go through the betas first.

mkoeppe · 2020-01-15T16:48:54Z

Dependencies: #29002

jhpalmieri · 2020-01-15T20:02:22Z

comment:8

I tried building with this branch on OS X, and I got doctest failures:

sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/matrix/matrix2.pyx  # 2 doctests failed
sage -t --long src/sage/finance/time_series.pyx  # 5 doctests failed
sage -t --long src/sage/misc/trace.py  # 4 doctests failed
sage -t --long src/sage/env.py  # 1 doctest failed
sage -t --long src/sage/repl/configuration.py  # 1 doctest failed
sage -t --long src/sage/schemes/affine/affine_homset.py  # 1 doctest failed

For example

File "src/sage/plot/plot.py", line 521, in sage.plot.plot
Failed example:
    os.system("sage -c \"if 'matplotlib' in sys.modules: sys.exit(1)\"") # long time
Expected:
    0
Got:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.1.beta0/local/bin/sage-eval", line 4, in <module>
        from sage.all import *
    ModuleNotFoundError: No module named 'sage'
    256

and

**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1127, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse()  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1131, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse(algorithm="numpy")  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
1 item had failures:
   2 of  25 in sage.matrix.matrix2.Matrix.pseudoinverse
    [2472 tests, 2 failures, 9.06 s]

mkoeppe · 2020-01-15T21:25:23Z

comment:9

Thanks for testing!

mkoeppe · 2020-01-15T22:54:09Z

Changed dependencies from #29002 to #29002, #29022

mkoeppe · 2020-01-15T22:54:09Z

comment:10

Replying to @jhpalmieri:

sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed

Not sure what's causing the numerical failures.

sage -t --long src/sage/env.py # 1 doctest failed

For this one, I have created #29022.

sagetrac-git · 2020-01-15T22:56:29Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d4fbe40`	`_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL`

sagetrac-git · 2020-01-15T22:56:29Z

Changed commit from a394268 to d4fbe40

sagetrac-git · 2020-01-15T23:37:36Z

Changed commit from d4fbe40 to d1779cc

sagetrac-git · 2020-01-15T23:37:36Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bba08ec`	`Create module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL`
`d1779cc`	`Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv`

mkoeppe · 2020-01-15T23:39:51Z

comment:13

Replying to @mkoeppe:

sage -t --long src/sage/env.py # 1 doctest failed

For this one, I have created #29022.

Fixed.

mkoeppe · 2020-01-16T02:21:36Z

comment:14

The numerical problems seems to happen because numpy does not find BLAS.

mkoeppe · 2020-01-16T02:37:35Z

comment:15

Actually, it does not find openblas or atlas, but it does find the "Accelerate framework":

accelerate_info:
  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

sagetrac-git · 2020-01-16T03:04:22Z

Changed commit from d1779cc to 93bae6f

sagetrac-git · 2020-01-16T03:04:22Z

Branch pushed to git repo; I updated commit sha1. New commits:

`85147f9`	`build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg`
`93bae6f`	`Merge branch 't/29025/numpy__site_cfg_needs_a__default__section' into t/29013/public/29013-use-py3-venv`

mkoeppe · 2020-01-16T03:05:05Z

comment:17

Merged #29025, fixes the numerical doctests.

mkoeppe · 2020-01-16T03:05:05Z

Changed dependencies from #29002, #29022 to #29002, #29022, #29025

dimpase · 2020-01-16T09:29:55Z

comment:18

I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)

sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py
**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - cryptominisat
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        from pycryptosat import Solver
    ModuleNotFoundError: No module named 'pycryptosat'

I wonder whether it uses a convoluted way to create python extensions, which got broken.

sagetrac-git · 2021-04-05T18:26:47Z

Changed commit from 92f1a1b to 9dcded5

sagetrac-git · 2021-04-05T18:26:47Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1942363`	`build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3`
`9dcded5`	`src/bin/sage: Only do the check for an incomplete installation when run from the source tree`

jhpalmieri · 2021-04-05T23:58:56Z

comment:96

With ./configure --with-python=/usr/bin/python3 --with-sage-venv="`pwd`/local/var/lib/sage/venv/python3.8", I'm getting some doctest failures:

sage -t --long --warn-long 101.2 --random-seed=0 src/sage/misc/package.py  # 5 doctests failed

(because it doesn't find some Python packages). Also

sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/find.py  # 2 doctests failed
sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/clean.py  # 1 doctest failed

Details:

File "src/sage_setup/find.py", line 365, in sage_setup.find.installed_files_by_module
Failed example:
    f1
Expected:
    'sage/structure/__init__.py'
Got:
    '/Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 367, in sage_setup.find.installed_files_by_module
Failed example:
    f2
Expected:
    'sage/structure/....pyc'
Got:
    'sage/structure/__init__.py'
**********************************************************************
File "src/sage_setup/clean.py", line 96, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: /Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
...
[followed by many many lines of "stale" files]

mkoeppe · 2021-04-06T00:04:08Z

comment:97

The ones in find and clean are I think #31314.

For the failure in package I know what to do

sagetrac-git · 2021-04-06T03:11:42Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6e0c19e`	`sage.misc.package.installed_packages: After SAGE_SPKG_INST, also go through SAGE_VENV_SPKG_INST`

sagetrac-git · 2021-04-06T03:11:42Z

Changed commit from 9dcded5 to 6e0c19e

jhpalmieri · 2021-04-06T21:13:46Z

comment:100

The sage_setup doctest failures have vanished, but I am seeing different problems with misc/package.py:

**********************************************************************
File "src/sage/misc/package.py", line 554, in sage.misc.package.package_manifest
Failed example:
    sagetex_manifest = package_manifest('sagetex')  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[1]>", line 1, in <module>
        sagetex_manifest = package_manifest('sagetex')  # optional - build
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/misc/package.py", line 570, in package_manifest
        with open(stamp_file) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/installed/sagetex-3.5'
**********************************************************************
File "src/sage/misc/package.py", line 555, in sage.misc.package.package_manifest
Failed example:
    sagetex_manifest['package_name'] == 'sagetex'  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[2]>", line 1, in <module>
        sagetex_manifest['package_name'] == 'sagetex'  # optional - build
    NameError: name 'sagetex_manifest' is not defined
**********************************************************************
File "src/sage/misc/package.py", line 557, in sage.misc.package.package_manifest
Failed example:
    'files' in sagetex_manifest  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[3]>", line 1, in <module>
        'files' in sagetex_manifest  # optional - build
    NameError: name 'sagetex_manifest' is not defined
**********************************************************************

I guess the issue is that manifests are now written to two different places, depending on whether it's a Python package or not.

mkoeppe · 2021-04-06T21:17:30Z

comment:101

Right, I forgot about this function. It will need a slightly generalized interface

sagetrac-git · 2021-04-07T00:19:42Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c5dd7ba`	`Merge tag '9.3.rc2' into t/29013/public/29013-use-py3-venv`
`f0fb082`	`sage.misc.package.installed_packages: Refactor through new generator _spkg_inst_dirs`
`d68e861`	`sage.misc.package.package_manifest: Go through all _spkg_inst_dirs()`

sagetrac-git · 2021-04-07T00:19:42Z

Changed commit from 6e0c19e to d68e861

jhpalmieri · 2021-04-07T21:10:22Z

comment:104

That fixes the package.py doctest for me, thank you. I'm not sure I fully understand how venvs are supposed to work. If I do ./configure --with-sage-venv=... && make and then separately do ./configure && make, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right? Now if I do ./configure && make pysingular, then I can do import PySingular, but I switch to the other venv, I won't be able to. However, 'pysingular' in installed_packages() returns True regardless of whether I'm using the venv, since it searches both installation directories.

(a) Am I understanding the situation correctly? and

(b) is it worth trying to fix?

mkoeppe · 2021-04-07T22:21:44Z

comment:105

Replying to @jhpalmieri:

I'm not sure I fully understand how venvs are supposed to work. If I do ./configure --with-sage-venv=... && make and then separately do ./configure && make, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right?

Yes, that's right. Separate for Python packages, sharing the non-Python packages, which are only in SAGE_LOCAL.

Now if I do ./configure && make pysingular, then I can do import PySingular, but I switch to the other venv, I won't be able to. However, 'pysingular' in installed_packages() returns True regardless of whether I'm using the venv, since it searches both installation directories.

OK, thanks for catching this. Yes, the sage.misc.package code does not handle this situation correctly. I would simply suggest to not mix builds that use with --with-sage-venv=... with builds that don't. You can switch between several venvs in this way.

jhpalmieri · 2021-04-07T23:59:23Z

Reviewer: John Palmieri

jhpalmieri · 2021-04-07T23:59:23Z

comment:106

Great, let's merge it, then.

mkoeppe · 2021-04-08T00:01:02Z

comment:107

Thank you!

vbraun · 2021-05-27T20:30:21Z

Changed branch from public/29013-use-py3-venv to d68e861

mkoeppe added this to the sage-9.1 milestone Jan 14, 2020

mkoeppe added c: build labels Jan 14, 2020

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Jan 15, 2020

mkoeppe changed the title ~~Support installation of Python packages into separate venvs depending on the python version~~ Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL Apr 4, 2021

mkoeppe added s: needs work and removed s: needs review labels Apr 6, 2021

mkoeppe added s: needs review and removed s: needs work labels Apr 6, 2021

jhpalmieri added s: needs work and removed s: needs review labels Apr 6, 2021

mkoeppe added s: needs review and removed s: needs work labels Apr 7, 2021

jhpalmieri added s: positive review and removed s: needs review labels Apr 7, 2021

vbraun removed the s: positive review label May 27, 2021

vbraun closed this as completed in 7ee6218 May 27, 2021

tobihan mentioned this issue Dec 5, 2020

Replace "$SAGE_LOCAL/bin" by more specific variables to make Sage easier to package, use in venvs #22731

Closed

Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL #29013

Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL #29013

Comments

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 15, 2020

This comment has been minimized.

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

dimpase commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

jhpalmieri commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 16, 2020

mkoeppe commented Jan 16, 2020

sagetrac-git mannequin commented Jan 16, 2020

sagetrac-git mannequin commented Jan 16, 2020

mkoeppe commented Jan 16, 2020

mkoeppe commented Jan 16, 2020

dimpase commented Jan 16, 2020

sagetrac-git mannequin commented Apr 5, 2021

sagetrac-git mannequin commented Apr 5, 2021

jhpalmieri commented Apr 5, 2021

mkoeppe commented Apr 6, 2021

sagetrac-git mannequin commented Apr 6, 2021

sagetrac-git mannequin commented Apr 6, 2021

jhpalmieri commented Apr 6, 2021

mkoeppe commented Apr 6, 2021

sagetrac-git mannequin commented Apr 7, 2021

sagetrac-git mannequin commented Apr 7, 2021

jhpalmieri commented Apr 7, 2021

mkoeppe commented Apr 7, 2021

jhpalmieri commented Apr 7, 2021

jhpalmieri commented Apr 7, 2021

mkoeppe commented Apr 8, 2021

vbraun commented May 27, 2021