Cache module import locations

[vbraun]

This ticket implements a cached module importer for Python. It compiles a list of all available files during build (when sage -b is run), hooks into the Python import framework and finds modules by the predetermined file list.

Building the Sage library (sage -b) automatically creates the cache and the sitecustomize.py that is used to load the importer into the Python session. The cache is a standard Python dictionary in sage/misc/modules_cache.py.

[vbraun@volker-laptop-two hg]$ strace -e file -f sage -c quit |& grep ENOENT | wc -l
26327
[vbraun@volker-laptop-two hg]$ sage -b
...
[vbraun@volker-laptop-two hg]$ strace -e file -f sage -c quit |& grep ENOENT | wc -l
3785

If you encounter any problems while testing this patch, you have to

rm $SAGE_LOCAL/lib/python/site-packages/sitecustomize.py*

to not load the importer during Python startup.

Apply trac_11729_cached_importer.patch to the sage library and trac_11729_import_cache_during_build.patch to the sage_scripts ($AGE_LOCAL/bin) repository.

CC: @williamstein @robertwb @ohanar @simon-king-jena

Component: misc

Keywords: sd32

Author: Volker Braun

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/11729

[williamstein]

Changed keywords from none to sd32

[williamstein]

comment:6

NOTE: If you try to test this, watch out because of these lines:

	113	cat <<EOF > "$SAGE_LOCAL/lib/python/site-packages/sitecustomize.py" 
 	114	from sage.misc.importer import sage_importer 
 	115	sage_importer.register() 
 	116	EOF 

They can make it so your Sage install is totally broken. To fix, just remove the contents of sitecustomize.py. I spent 15 minutes confused by this.

[vbraun]

comment:7

You have to delete both $SAGE_LOCAL/lib/python/site-packages/sitecustomize.py and $SAGE_LOCAL/lib/python/site-packages/sitecustomize.pyc, though the latter might not exist. The final version of the patch will test that sage starts up successfully and otherwise delete the sitecustomize.py automatically.

Also, if you run into problems (presumably some ImportError), please post what is causing it.

[vbraun]

Updated patch

[vbraun]

Attachment: trac_11729_import_cache_during_build.patch.gz

Attachment: trac_11729_cached_importer.patch.gz

Updated patch

[vbraun]

comment:8

I've added another workaround for PIL.Image, now make ptest passes! I would say that the patch is now feature-complete, though it lacks documentation and doctests. Though writing doctests is tricky without breaking the module importing. Somebody should test it on OSX :-)

[mkoeppe]

comment:15

outdated, should close

[dimpase]

Reviewer: Dima Pasechnik