Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile

[mkoeppe]

A lot of the stuff (not necessarily all) in $SAGE_SRC/ext belongs directly in the sage Python package--in particular any files that are needed for the package itself to function properly.

A model for that is sage.gaprc, a file required for the libgap module to work, which is installed in sage.libs.gap as "package data" as of #22626.

In this ticket, we install all of ext as package data of sage.

• In a follow up ticket, if we identify any files in $SAGE_SRC/ext that aren't explicitly required from the package, we will move them to a more appropriate place.

Depends on #29233

CC: @jdemeyer @embray @kiwifb @slel @saraedum @isuruf @dimpase @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch: fef5a5f

Reviewer: François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/21785

[kiwifb]

comment:2

That would actually help me. I see extra potential to this for my jupyter kernel install woes as well (even if I know Jeroen doesn't see it that way).

[slel]

comment:3

There was a reference to this ticket in a "sage-packaging" discussion:

• https://groups.google.com/d/msg/sage-packaging/McjYh9wYlmk/PgBK1yryCwAJ

[embray]

comment:4

I would add, a lot of the stuff (not necessarily all) in $SAGE_SRC/ext might better belong directly in the sage Python package--in particular any files that are needed for the package itself to function properly. For example you can see in #22626 that I put sage.gaprc, a file required for the libgap module to work, in sage.libs.gap as "package data".

There are several files in $SAGE_SRC/ext that there's no reason to ever allow to be out of sync with the Python package, and that could be given the same treatment.

For files in $SAGE_SRC/ext in general, that aren't explicitly required from the package (are there any such files) it might be better to move out of src/ entirely?

[embray]

comment:5

Ticket retargeted after milestone closed

[mkoeppe]

comment:7

setuptools experts: To move this ticket forward, should it go through #22655 (Support package_data-like of non-Python resource files in Python packages)?

[embray]

comment:8

Something like it, yes, though I think maybe it can be simplified.

[mkoeppe]

comment:9

Erik, should discussion then take place on #22655 or here?

[mkoeppe]

comment:10

Description adapted from embray's comment 4.

[mkoeppe]

Dependencies: #29062

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Branch: u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile

[mkoeppe]

comment:13

Branch is on top of #29062.

Installation of the package data does not work yet... what am I missing?

---

New commits:

568afdc	build/pkgs/python3/spkg-install: Install valgrind-python.supp directly in SAGE_LOCAL, not first in SAGE_SRC
ceb35bb	src/bin/sage-valgrind: Find python.supp/python3.supp in SAGE_LOCAL/lib/valgrind or system directories
06a0226	Move src/ext to src/sage/ext_data and install it as 'package_data'

[mkoeppe]

Commit: 06a0226

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1edc654

src/setup.py [package_data]: List all ext_data subdirectories

[sagetrac-git]

Changed commit from 06a0226 to 1edc654

[mkoeppe]

comment:16

@embray Also this ticket would need review.

[sagetrac-git]

Changed commit from 1edc654 to 749928c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

95c8316	Move src/ext to src/sage/ext_data and install it as 'package_data'
749928c	src/setup.py [package_data]: List all ext_data subdirectories

[mkoeppe]

comment:18

Rebased on top of current #29062 (and 9.1.beta2)

[mkoeppe]

comment:49

Thanks for reviewing!

[vbraun]

comment:50

I'm still seeing this on the buildbot:

**********************************************************************
File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests
Failed example:
    for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time
        path = os.path.relpath(path)
        dirs.sort(); files.sort()
        for F in files:
            _, ext = os.path.splitext(F)
            if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'):
                filename = os.path.join(path, F)
                FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True))
                FDS._test_enough_doctests(verbose=False)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module>
        FDS._test_enough_doctests(verbose=False)
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 809, in _test_enough_doctests
        tests, _ = self.create_doctests({})
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests
        load(filename, namespace) # errors raised here will be caught in DocTestTask
      File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load
        exec(code, globals)
      File "./postprocess.py", line 19, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: '-p'
**********************************************************************
1 item had failures:
   1 of   9 in sage.doctest.sources.FileDocTestSource._test_enough_doctests
    [367 tests, 1 failure, 26.05 s]
    1 memlimit not run
----------------------------------------------------------------------
sage -t --long src/sage/doctest/sources.py  # 1 doctest failed
----------------------------------------------------------------------

And this one on my machine:

[dochtml] [installat] The HTML pages are in local/share/doc/sage/html/en/installation.
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/release/Sage/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[dochtml]     "__main__", fname, loader, pkg_name)
[dochtml]   File "/home/release/Sage/local/lib/python2.7/runpy.py", line 72, in _run_code
[dochtml]     exec code in run_globals
[dochtml]   File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 1720, in main
[dochtml]     builder()
[dochtml]   File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 336, in _wrapper
[dochtml]     build_many(build_other_doc, L)
[dochtml]   File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 280, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] OSError: /home/release/Sage/src/doc/en/installation/source.rst:228: WARNING: Include file u'/home/release/Sage/src/doc/en/installation/debian.txt' not found or reading it failed

I'm guessing this ticket has issues with incremental builds, can you give this a try?

[vbraun]

comment:51

I made #29233 for the documentation build issue

[mkoeppe]

Dependencies: #29233

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0dc97db	Clean src/doc/en/installation/*.txt in bootstrap-clean, not misc-clean
51e7f51	Merge branch 't/29233/bootstrap____make_distclean_broken' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile

[sagetrac-git]

Changed commit from 50a5d25 to 51e7f51

[mkoeppe]

comment:54

I don't know how to reproduce this error:

FileNotFoundError: [Errno 2] No such file or directory: '-p'

[kiwifb]

comment:55

Ic ouldn't reproduce it either. I suspect this is an incremental build issue.

[sagetrac-git]

Changed commit from 51e7f51 to fef5a5f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

fef5a5f

FileDocTestSource._test_enough_doctest doctest: Do not load .py files that are not a sage module

[mkoeppe]

comment:58

Found it, fixed. Needs review

[kiwifb]

comment:59

OK let's try it again.

[vbraun]

Changed branch from u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile to fef5a5f

[kiwifb]

comment:61

Could you check that .nodoctest files are properly installed in vanilla sage? Because vanilla sage runs doctests on original sage sources (while in sage-on-gentoo it is run on the install because the sources are not available at runtime) it may have been missed.

[kiwifb]

Changed commit from fef5a5f to none

[paulmasson]

comment:62

This ticket messes up anyone with data in the old src/ext directory. Would have been nice to have some advance notice on sage-devel, you know?

[mkoeppe]

comment:63

Let me know if you need help with resolving a merge conflict.