Remove init.py files for subpackages designated to be namespace packages #33011

mkoeppe · 2021-12-11T04:27:50Z

This ticket changes the following packages to namespace packages (by removing __init__.py):

sage.matrix (this allows the new distribution sage-meataxe introduced in Modularization of sagelib: Break out a separate package sagemath-meataxe #30151 to add the module sage.matrix.matrix_gfpn_dense) to the package)
sage.libs (for .meataxe - Modularization of sagelib: Break out a separate package sagemath-meataxe #30151, .sirocco, .coexter3, .fes)
sage.interfaces (for .primecount)
sage.graphs (for .bliss, .mcqd)
sage.graphs.graph_decompositions (for .tdlib -> sage-tdlib, Modularization of sagelib: Create package sage-tdlib #29864.)
sage.numerical, sage.numerical.backends. (This is preparation for later work, in sage_setup: Modify clean_stale_files to support out-of-tree namespace packages #30152.)

A complication lies in the behavior of the Python import machinery:
setup.py puts the source path in front of setup.py because sage_setup uses sage.env and is_package_or_sage_namespace_package_dir (#33033).
But when an old version of sage that is an ordinary package is installed already, the source will not shadow it.

To avoid this complication, in this ticket we do not yet remove src/sage/__init__.py. That is done in #34187.

CC: @tobiasdiez @kwankyu

Component: refactoring

Author: Matthias Koeppe

Branch/Commit: 39aa2f1

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/33011

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-12-11T04:44:45Z

Branch: u/mkoeppe/remove___init___py_files_for_packages_designated_to_be_namespace_packages

mkoeppe · 2021-12-11T04:45:02Z

Last 10 new commits:

`2a4a323`	`Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap`
`9ef0567`	`Merge #29039`
`e0399a1`	`pkgs/sagemath-objects: Install a fixed set of scripts, not all of src/bin`
`7519048`	`src/bin/sage[-env]: Put SAGE_ROOT/src/bin in front of path only if run out of this directory`
`00024e6`	`Merge #32933`
`bfc953b`	`src/sage/doctest/reporting.py: Indicate --environment if not default`
`36d6b09`	`pkgs/sagemath-categories/setup.cfg.m4: Add scripts`
`8648741`	`Merge #29865`
`d91e23b`	`Merge #28925`
`137f5ee`	`Remove __init__.py files`

mkoeppe · 2021-12-11T04:45:02Z

Commit: 137f5ee

sagetrac-git · 2021-12-17T07:07:21Z

Changed commit from 137f5ee to 699ca47

sagetrac-git · 2021-12-17T07:07:21Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`2e995dd`	`src/sage/{arith,ext,libs,rings,sets}/all__sagemath_objects.py`
`f244b57`	`pkgs/sagemath-objects/MANIFEST.in: Exclude generated files`
`2f6d473`	`pkgs/sagemath-categories/MANIFEST.in.m4: Exclude generated files`
`1a3afe6`	`Merge #28925`
`7f9c9b0`	`Remove __init__.py files`
`cf7371f`	`src/sage_setup/find.py: Update import of is_package_or_namespace_package_dir`
`5808f3d`	`src/sage/misc/package_dir.py: New file for is_package_or_sage_namespace_package_dir`
`f5addb3`	`Merge #33033`
`464b0a5`	`Merge #28925`
`699ca47`	`Remove __init__.py files, add some all.py files`

sagetrac-git · 2021-12-17T16:39:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2062384`	`src/sage/__init__.py: Restore`

sagetrac-git · 2021-12-17T16:39:54Z

Changed commit from 699ca47 to 2062384

mkoeppe · 2021-12-17T16:43:03Z

Author: Matthias Koeppe

sagetrac-git · 2022-01-23T06:26:15Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3939af7`	`Merge tag '9.5.rc3' into t/33033/prepare_sage_doctest_for_namespace_packages`
`178d111`	`Merge #33033`
`efbc045`	`Merge #28925`

sagetrac-git · 2022-01-23T06:26:15Z

Changed commit from 2062384 to efbc045

sagetrac-git · 2022-02-14T05:46:21Z

Changed commit from efbc045 to be39670

sagetrac-git · 2022-02-14T05:46:21Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5c873e0`	`Merge tag '9.6.beta1' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_`
`37b3299`	`pkgs/sagemath-{objects,categories}/setup.cfg.m4: Update python_requires`
`5546c63`	`pkgs/sagemath-objects/MANIFEST.in: Add sage.misc.package_dir`
`be39670`	`Merge #28925`

mkoeppe · 2022-02-14T06:05:30Z

Work Issues: broken with "configure --enable-editable" (gitpod)

sagetrac-git · 2022-02-14T06:55:41Z

Changed commit from be39670 to 87277b3

sagetrac-git · 2022-02-14T06:55:41Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f961a7d`	`src/setup.py: Support namespace packages in Cython 0.x`
`87277b3`	`Merge #28925`

mkoeppe · 2022-02-14T07:27:05Z

Changed work issues from broken with "configure --enable-editable" (gitpod) to none

sagetrac-git · 2022-07-01T01:57:34Z

Branch pushed to git repo; I updated commit sha1. New commits:

`81237e1`	`Merge tag '9.7.beta2' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_`
`dd2c525`	`Merge tag '9.7.beta3' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_`
`2cc9002`	`pkgs/sagemath-objects/setup.py: Remove unused import`
`cd4d8e9`	`pkgs/sagemath-standard/setup.py: Monkey-patching cython is now done by the sage_build_cython command`
`af9ac78`	`src/sage_setup/find.py (find_extra_files): Document new arg 'distributions'`
`13e66a3`	`src/sage_setup/find.py (find_extra_files): Fix indentation`
`9ab3200`	`src/sage_setup/clean.py (clean_install_dir): Document new arg 'distributions'`
`6a76779`	`Merge #28925`

sagetrac-git · 2022-07-01T01:57:34Z

Changed commit from 56e9123 to 6a76779

mkoeppe · 2022-07-04T03:01:09Z

comment:65

Replying to @kwankyu:

Replying to @mkoeppe:
Replying to @kwankyu:
When I rebuild sage by sage -b, I observe much bigger log, including
...
[sagelib-9.7.beta1] package init file 'sage/misc/__init__.py' not found (or not a regular file)
[sagelib-9.7.beta1] package init file 'sage/numerical/__init__.py' not found (or not a regular file)
[...]
These messages are normal output when namespace packages are in use.
They are not very informative and look like error messages.

setuptools 63.1.0 will eliminate them - https://setuptools.pypa.io/en/latest/history.html#v63-1-0

kwankyu · 2022-07-04T04:18:58Z

comment:66

Replying to @mkoeppe:

Replying to @kwankyu:
Replying to @mkoeppe:
Replying to @kwankyu:
When I rebuild sage by sage -b, I observe much bigger log, including
...
[sagelib-9.7.beta1] package init file 'sage/misc/__init__.py' not found (or not a regular file)
[sagelib-9.7.beta1] package init file 'sage/numerical/__init__.py' not found (or not a regular file)
[...]
These messages are normal output when namespace packages are in use.
They are not very informative and look like error messages.
setuptools 63.1.0 will eliminate them - https://setuptools.pypa.io/en/latest/history.html#v63-1-0

Nice!

Those changing mode of ... lines are not seen any more already.

sagetrac-git · 2022-07-11T00:26:14Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bd185b5`	`Merge tag '9.7.beta5' into t/33011/remove___init___py_files_for_packages_designated_to_be_namespace_packages`

sagetrac-git · 2022-07-11T00:26:14Z

Changed commit from 6a76779 to bd185b5

mkoeppe · 2022-07-11T00:26:35Z

Changed dependencies from #28925 to none

kwankyu · 2022-07-12T04:07:20Z

comment:69

I get

sage -t --warn-long 67.8 --random-seed=272636909256794490188818216866457263419 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 406, in sage_setup.find.installed_files_by_module
Failed example:
    files_by_module['sage.graphs.graph_decompositions']
Expected:
    set()
Got:
    {'sage/graphs/graph_decompositions/__init__.py',
     'sage/graphs/graph_decompositions/__pycache__/__init__.cpython-39.pyc'}
**********************************************************************
1 item had failures:
   1 of  10 in sage_setup.find.installed_files_by_module
    [49 tests, 1 failure, 1.77 s]
----------------------------------------------------------------------
sage -t --warn-long 67.8 --random-seed=272636909256794490188818216866457263419 src/sage_setup/find.py  # 1 doctest failed
----------------------------------------------------------------------

There is no __init__.py in src/sage/graphs/graph_decompositions, but there is one in local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/graphs/graph_decompositions. Even if I delete it, it gets created when I rebuild sage by sage -b.

Do I need to make distclean first?

mkoeppe · 2022-07-12T04:13:01Z

comment:70

Hm, I guess the __init__.py file survives in pkgs/sagemath-standard/build/lib.*

mkoeppe · 2022-07-12T04:15:32Z

comment:71

make distclean (or just rm -rf pkgs/sagemath-standard/build) will solve it. Not sure if there's a good way to make it work with incremental builds

kwankyu · 2022-07-12T04:32:16Z

comment:72

Replying to @mkoeppe:

Hm, I guess the __init__.py file survives in pkgs/sagemath-standard/build/lib.*

Yes. it's there.

kwankyu · 2022-07-12T04:37:52Z

comment:73

Replying to @mkoeppe:

make distclean (or just rm -rf pkgs/sagemath-standard/build) will solve it. Not sure if there's a good way to make it work with incremental builds

I deleted the directory pkgs/sagemath-standard/build and did make -j8. This will save time, but I don't know if this is a valid way to build sage.

kwankyu · 2022-07-12T04:38:56Z

comment:74

Replying to @kwankyu:

Replying to @mkoeppe:
(or just rm -rf pkgs/sagemath-standard/build)

Ah, you mentioned this.

kwankyu · 2022-07-13T00:48:14Z

comment:75

Replying to @kwankyu:

I get

sage -t --warn-long 67.8 --random-seed=272636909256794490188818216866457263419 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 406, in sage_setup.find.installed_files_by_module
Failed example:
    files_by_module['sage.graphs.graph_decompositions']
Expected:
    set()
Got:
    {'sage/graphs/graph_decompositions/__init__.py',
     'sage/graphs/graph_decompositions/__pycache__/__init__.cpython-39.pyc'}

How about adding a comment like

     files_by_module['sage.graphs.graph_decompositions']  # make distclean if failed

to guide people what to do to resolve the failure?

sagetrac-git · 2022-07-13T15:36:02Z

Changed commit from bd185b5 to 39aa2f1

sagetrac-git · 2022-07-13T15:36:02Z

Branch pushed to git repo; I updated commit sha1. New commits:

`39aa2f1`	`src/sage_setup/command/sage_install.py (sage_clean): Clean __init__.py when installed package was ordinary, source package is namespace`

mkoeppe · 2022-07-13T15:37:18Z

comment:77

I think I found a proper solution for this

kwankyu · 2022-07-15T08:43:49Z

comment:78

Replying to @mkoeppe:

I think I found a proper solution for this

Yes. It looks good.

I have some questions. The file sage/__init__.py is not deleted yet here. Then can a subpackage of sage be a namespace package? I guess the answer is yes, but no two distribution packages can install into the namespace package. Am I right?

Is the complication of removing sage/__init__.py still valid?

kwankyu · 2022-07-15T10:11:14Z

comment:79

Replying to @kwankyu:

Is the complication of removing sage/__init__.py still valid?

I am asking this even though I do not understand the complication (I tried to).

mkoeppe · 2022-07-15T14:26:14Z

comment:80

Replying to @kwankyu:

The file sage/__init__.py is not deleted yet here. Then can a subpackage of sage be a namespace package? I guess the answer is yes, but no two distribution packages can install into the namespace package. Am I right?

That's right.

Is the complication of removing sage/__init__.py still valid?

Yes, I've opened #34187 for it

kwankyu · 2022-07-15T14:58:09Z

comment:81

I think sage is now ready to get this in.

kwankyu · 2022-07-15T14:58:09Z

Reviewer: Kwankyu Lee

mkoeppe · 2022-07-15T15:34:48Z

comment:82

Thank you!

vbraun · 2022-07-21T21:12:13Z

Changed branch from u/mkoeppe/remove___init___py_files_for_packages_designated_to_be_namespace_packages to 39aa2f1

mkoeppe added this to the sage-9.6 milestone Dec 11, 2021

mkoeppe added t: refactoring labels Dec 11, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Remove __init__.py files for packages designated to be namespace packages~~ Remove __init__.py files for subpackages designated to be namespace packages Dec 17, 2021

mkoeppe added the s: needs review label Dec 17, 2021

mkoeppe added s: needs work and removed s: needs review labels Feb 14, 2022

mkoeppe added s: needs review and removed s: needs work labels Feb 14, 2022

This comment has been minimized.

Sign in to view

kwankyu added s: positive review and removed s: needs review labels Jul 15, 2022

vbraun removed the s: positive review label Jul 21, 2022

vbraun closed this as completed in ea11a51 Jul 21, 2022

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#33011

88b93db

Remove __init__.py files for subpackages designated to be namespace packages #33011

Remove __init__.py files for subpackages designated to be namespace packages #33011

Comments

mkoeppe commented Dec 11, 2021

mkoeppe commented Dec 11, 2021

mkoeppe commented Dec 11, 2021

mkoeppe commented Dec 11, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented Dec 17, 2021

sagetrac-git mannequin commented Dec 17, 2021

sagetrac-git mannequin commented Dec 17, 2021

sagetrac-git mannequin commented Dec 17, 2021

This comment has been minimized.

mkoeppe commented Dec 17, 2021

sagetrac-git mannequin commented Jan 23, 2022

sagetrac-git mannequin commented Jan 23, 2022

sagetrac-git mannequin commented Feb 14, 2022

sagetrac-git mannequin commented Feb 14, 2022

mkoeppe commented Feb 14, 2022

sagetrac-git mannequin commented Feb 14, 2022

sagetrac-git mannequin commented Feb 14, 2022

mkoeppe commented Feb 14, 2022

sagetrac-git mannequin commented Jul 1, 2022

sagetrac-git mannequin commented Jul 1, 2022

mkoeppe commented Jul 4, 2022

kwankyu commented Jul 4, 2022

sagetrac-git mannequin commented Jul 11, 2022

sagetrac-git mannequin commented Jul 11, 2022

mkoeppe commented Jul 11, 2022

kwankyu commented Jul 12, 2022

mkoeppe commented Jul 12, 2022

mkoeppe commented Jul 12, 2022

kwankyu commented Jul 12, 2022

kwankyu commented Jul 12, 2022

kwankyu commented Jul 12, 2022

kwankyu commented Jul 13, 2022

sagetrac-git mannequin commented Jul 13, 2022

sagetrac-git mannequin commented Jul 13, 2022

mkoeppe commented Jul 13, 2022

kwankyu commented Jul 15, 2022

kwankyu commented Jul 15, 2022

mkoeppe commented Jul 15, 2022

This comment has been minimized.

kwankyu commented Jul 15, 2022

kwankyu commented Jul 15, 2022

mkoeppe commented Jul 15, 2022

vbraun commented Jul 21, 2022

Remove init.py files for subpackages designated to be namespace packages #33011

Remove init.py files for subpackages designated to be namespace packages #33011