Add build/pkgs/SPKG/install-requires.txt for all Python packages, remove some unneeded packages

[mkoeppe]

These files:

• serve as a marker that an SPKG is a Python package (for Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL #29013)
• may include comments that explain why we reject certain versions
• provide version constraints (lower and upper bounds)
• as such, are preparation for Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages) #29023: Meta-ticket: In a python 3 build, optionally use system Python packages
• can be used to generate other metadata such as src/Pipfile, src/setup.cfg [install_requires] (sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041) in the bootstrap phase, to avoid duplicating this information in various places

CC: @tobiasdiez @isuruf @kiwifb @antonio-rojas @dimpase

Component: build

Author: Matthias Koeppe

Branch/Commit: 6ec00dd

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30719

[mkoeppe]

Branch: u/mkoeppe/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages

[mkoeppe]

comment:2

Tobias, before I add more of these, do you think this format is useful for generating other files?

---

New commits:

a754e8c

Add some build/pkgs/*/install-requires.txt

[mkoeppe]

Commit: a754e8c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

11ce883

Add more build/pkgs/*/install-requires.txt

[sagetrac-git]

Changed commit from a754e8c to 11ce883

[tobiasdiez]

comment:4

So the idea would be to concatenate all these install-requires.txt files to obtain the install_requires section of src/setup.cfg? That should work. Except that cython is a more a compile time dependency ([build-system] requires = ["cython"] in pyproject.toml), right?

I have to admit though that I don't see the benefit of having all these text files over having a single src/setup.cfg (or pyproject for that matter). It feels like it's the same information, just in a different location.
In fact, it feels a bit like you are reinventing the wheel. pyproject etc also have support for different kinds of dependencies, e.g. optional, compile time (cython), test (pytest) and build dependencies (tox).

Maybe that's a bit too naïve, but I would approach this as follows: sagelib (everything in /src) is a python library and as such only declares minimum dependencies and their requirements, trying not to pin versions (except if it's really necessary). I think, the most modern way would be to do this via a pyproject.toml file.
Then sage-the-application (the actual thing used by the enduser) is a python application and thus includes sagelib as a library and needs to satisfy its dependencies (this is the point where you have to specify in some way the exact version you want to use).
sage-the library should be completely independent of sage-the-application (i.e. it should work to have them in different git repositories).

[mkoeppe]

comment:5

Replying to @tobiasdiez:

So the idea would be to concatenate all these install-requires.txt files to obtain the install_requires section of src/setup.cfg?

Yes.

That should work. Except that cython is a more a compile time dependency ([build-system] requires = ["cython"] in pyproject.toml), right?

Right; but sagelib also exposes cython at runtime for interactive use.

I have to admit though that I don't see the benefit of having all these text files over having a single src/setup.cfg (or pyproject for that matter). It feels like it's the same information, just in a different location.

With modularization, I expect to see dozens of setup.cfg files for the various subset distributions. Basically, I don't want to duplicate version ranges between these.

In fact, it feels a bit like you are reinventing the wheel. pyproject etc also have support for different kinds of dependencies, e.g. optional, compile time (cython), test (pytest) and build dependencies (tox).

What I want to do is define version ranges in one place for all of the Sage project.

Distinguishing the kinds of dependencies (build, test, ...) should be orthogonal -- and the mechanism in #29041 should be able to put these version ranges into whatever file/format is needed.

(I don't think it would add sufficient value to Sage to have different version ranges for a given package in different roles, like "we need Cython >= 0.29.21 for build, but any Cython >= 0.27 is fine for runtime.)

sage-the library should be completely independent of sage-the-application (i.e. it should work to have them in different git repositories).

I don't like to go into this direction because it would add a big hurdle to our developer community. Having one git repository is key.

[tobiasdiez]

comment:6

Replying to @mkoeppe:

I have to admit though that I don't see the benefit of having all these text files over having a single src/setup.cfg (or pyproject for that matter). It feels like it's the same information, just in a different location.

With modularization, I expect to see dozens of setup.cfg files for the various subset distributions. Basically, I don't want to duplicate version ranges between these.

Sounds like a valid point.

Having one git repository is key.

I agree! What I meant is that it might be helpful to approach it like this, i.e. how would you solve this problem if sage-library would be sitting in a different git repo? I think, this will be helpful e.g for distribution as pipy-installable package where it would be helpful if src/ works independently of the make and build scripts.

One more remark: I would try to make the version range specifications as general as possible, i.e. only add restrictions if its known that that this particular version doesn't work with sage (this in particular applies to the upper limit). Otherwise the sage-library (and its subdistributions) would be harder to use (e.g. if one cannot upgrade/downgrade a dependency because other code relies on it). In this context, it may actually turn out to be helpful if you could could control the dependency versions for each submodule sepereatly, e.g. if ModuleA breaks with a newer version but ModuleB doesn't have these problems.

[mkoeppe]

comment:7

Replying to @tobiasdiez:

What I meant is that it might be helpful to approach it like this, i.e. how would you solve this problem if sage-library would be sitting in a different git repo? I think, this will be helpful e.g for distribution as pipy-installable package where it would be helpful if src/ works independently of the make and build scripts.

Sounds like we are basically in agreement here. One detail, however, is the bootstrapping phase. The way I see it, after running ./bootstrap, we would have standard Python package source trees (which will not depend on make and build) -- right now, src or build/pkgs/sagelib/src/, but many more after modularization (such as build/pkgs/sage-ntl/src/). By using scripts that are executed during bootstrapping, we can avoid duplicating a lot of information and thus reduce the maintenance burden.

[mkoeppe]

comment:8

Replying to @tobiasdiez:

I would try to make the version range specifications as general as possible, i.e. only add restrictions if its known that that this particular version doesn't work with sage (this in particular applies to the upper limit). Otherwise the sage-library (and its subdistributions) would be harder to use (e.g. if one cannot upgrade/downgrade a dependency because other code relies on it).

Yes, this is tricky to get right and will probably need some detailed discussion for individual packages. But I do think that some upper bounds on the version are always needed because we do not want our package to stop working if a new major version of some package is released on pypi. For example, Cython has an upcoming 3.x version which we have not tested with at all.

In this context, it may actually turn out to be helpful if you could could control the dependency versions for each submodule separately, e.g. if ModuleA breaks with a newer version but ModuleB doesn't have these problems.

Yes, this is conceivable but I would hope that if this ever becomes necessary, it can be implemented by overriding the default rather than defaulting to duplicate maintenance of the version ranges.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

85b2dcf	for a in *; do if [ -d $a -a ! -r $a/requirements.txt -a ! -r $a/install-requires.txt ]; then if grep -q -i sdh_pip $a/spkg-install.in >&/dev/null ; then ( if [ -r $a/package-version.txt ]; then echo "$a >=$(sed s/[.]p[0-9]$// $a/package-version.txt)"; else echo "$a"; fi) > $a/install-requires.txt && git add $a/install-requires.txt; fi; fi; done
f9ac70e	Add install-requires.txt for setuptools, pip

[sagetrac-git]

Changed commit from 11ce883 to f9ac70e

[mkoeppe]

comment:10

Autogenerated for now, setting current version as lower bound

[mkoeppe]

comment:12

I'm hoping to gather version information for our Python packages here.

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f6a2c56

Merge tag '9.3.beta1' into t/30719/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages

[sagetrac-git]

Changed commit from f9ac70e to f6a2c56

[kiwifb]

comment:17

That's a lot of packages to parse :(

Some stuff I can notice from a cursory inspection. I use sphinx 3.2.1 so the restriction <3.2 seem useless. Similarly I currently use networkx 2.5. python_openid was a sagenb dependency, I don't think we need it anymore. I certainly do not have it anymore on my system. Same with itsdangerous.

[tobiasdiez]

comment:18

The pipenv.lock file added in #30371 is a good indication, too.

The <1.0 constraint for cython seems to be not necessary (at least they are very far off from version 1.0).

[sagetrac-git]

Changed commit from f6a2c56 to ba1d913

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e61929d	build/pkgs/sphinx/install-requires.txt: Update from gentoo
8fe0e35	build/pkgs/python_openid: Unused, remove
ba1d913	build/pkgs/itsdangerous: Unused, remove

[mkoeppe]

comment:20

Replying to @tobiasdiez:

The <1.0 constraint for cython seems to be not necessary (at least they are very far off from version 1.0).

From comment 8:
For example, Cython has an upcoming 3.x version which we have not tested with at all.

[sagetrac-git]

Changed commit from ba1d913 to bb6c4ae

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

bb6c4ae

build/pkgs/tox/install-requires.txt: New

[sagetrac-git]

Changed commit from bb6c4ae to d507501

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d507501

build/pkgs/networkx/install-requires.txt: Update from gentoo

[mkoeppe]

comment:23

See also: Upgrade ticket #30611

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

82d0df3	build/pkgs/p_group_cohomology/install-requires.txt: Not on PyPI, remove
d5e9840	build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove

[sagetrac-git]

Changed commit from d507501 to d5e9840

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c3a9352

build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove

[sagetrac-git]

Changed commit from d5e9840 to c3a9352

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ebd4610

build/pkgs/pynac/install-requires.txt: New

[sagetrac-git]

Changed commit from c3a9352 to ebd4610

[mkoeppe]

comment:28

Good enough as a first approximation?

[mkoeppe]

comment:29

Missing: numpy

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6ec00dd

build/pkgs/{numpy,pillow}/install-requires.txt: New

[sagetrac-git]

Changed commit from ebd4610 to 6ec00dd

[dimpase]

comment:31

OK, good. (all tests of this branch pass on Fedora 30, too)

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:32

Thanks!

[vbraun]

Changed branch from u/mkoeppe/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages to 6ec00dd