Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI

[mkoeppe]

This task ticket, organizing the steps necessary to eventually provide sagelib via PyPI, has been superseded by Meta-ticket #29705: Modularize sagelib into separate distributions (distutils packages).

. . . . . . . . . . . . . . . . .

It would be used by a Python user as follows. Let's assume a user has already installed every package that sage-the-distribution provides on their distribution.
Then a simple

pip install sagelib

would install sagelib in the user's Python installation. Then the user could do from sage.all import *.

. . . . . . . . . . . . . . . . .

Here are the first steps:

• Meta-ticket: Clean up src/setup.py to bring it to standard setuptools behavior #21508: Clean up src/setup.py to bring it to standard distutils behavior
  • Make sagelib setup.py self-contained and independent of SAGE_ROOT #21480: Make sagelib setup.py self-contained, independent of SAGE_ROOT
  • Use custom build_ext to compile Cython code #21600: Use custom build_ext to compile Cython code
  • Cleaning up stale installed files in setup() #21604: Cleaning up stale installed files in setup()
  • src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory #21654: Disentangle cleaning of stale installed files in build directory and in install directory
  • Fix sagelib sdist (src/setup.py sdist) #21516: Fix sagelib sdist (src/setup.py sdist)
  • Make src/setup.py respect --build-base and --inplace, independent of SAGE_CYTHONIZED #21535: Make src/setup.py respect --build-base and --inplace, independent of SAGE_CYTHONIZED
  • build the sage library in place #12659: build the sage library in place
  • Make sure src/setup.py respects --prefix, --install-base, --root, --user #21573: Make sure src/setup.py respects --install-base and --root
  • Don't use make for autogenerated modules in sagelib #21613: Make setup.py not depend on make (follow-up: fix incompatibility with py3 in autogen/pari #22044, setup.py: run_autogen is ran too late #22094, setup.py: generate auto-generated files in setup() #22106)
  • Don't hard-code the path to thebe.js #22061: Don't hard-code the path to thebe.js (dup: Fix symbolic link to thebe.js #21527: Fix symbolic link to thebe.js)
  • Install Jupyter kernel in the correct prefix #25546: Install Jupyter kernel in the correct prefix
  • sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041: at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs
• Testsuite for src/setup.py #21678: Testsuite for src/setup.py
• Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile #21785: Installation of SAGE_SRC/ext/ in SAGE_LOCAL/share/sage/ext/ should be done by setup.py, not build/make/Makefile
• Use package_data instead of data_files in setup.py #20108: Use package_data instead of data_files in setup.py
• Add a separate "build_cython" command to setup.py #21682: Add a separate "cythonize" command to setup.py
• Install src/bin/* scripts via setup.py (scripts, console_scripts) #21569: Install src/bin/* scripts via setup.py (scripts, console_scripts)
• Move non-scripts of src/bin/ elsewhere (and also move their install location) #21570: Move non-scripts of src/bin/ elsewhere (and also move their install location)
• Install COPYING.txt #21571: Install COPYING.txt in SAGE_LOCAL and use it from misc/copying.py
• Improve Cython debugging #21509: Install cython_debug somewhere in SAGE_LOCAL
• Move runtime documentation python modules into src/sage #21732: Move runtime documentation python modules into src/sage
• Python package sage_conf: Provides optional configuration information for sagelib #29038: Python package sage_conf: Provides optional configuration information for sagelib

This defines milestone 1. sagelib is now a well-behaved Python package. It can be built and installed as follows (without invoking sage -sh):

export SAGE_LOCAL=/path/to/local/hierarchy/populated/by/sage/distribution
export SAGE_PKGS=/path/to/sage/distribution/source/directory/build/pkgs
$SAGE_LOCAL/bin/python setup.py install   # or pip install . 

. . . . . . . . . . . . . . . . .

Next steps:

• Remove the dependency on the environment variable SAGE_PKGS (Replace is_package_installed with Features #20382, ...)

This defines milestone 2. sagelib can now be built and installed as follows (without invoking sage -sh):

export SAGE_LOCAL=/path/to/local/hierarchy/populated/by/sage/distribution
$SAGE_LOCAL/bin/python setup.py install   # or pip install . 

. . . . . . . . . . . . . . . . .

Next steps:

• Remove the dependency on the SAGE_LOCAL environment variable
  • hardwired paths in src/sage #15105: hardwired paths in src/sage
  • Clean up unnecessary flint include directories #21803: env.py, misc/cython.py: Define and use FLINT_INCLUDE_DIR
  • Add an autoconf configure script for sagelib #29119: Add an autoconf configure script for sagelib
  • ...

This defines milestone 3. If SAGE_LOCAL is not set, then sagelib will discover system packages and Python packages installed in standard places.

python setup.py install   # or pip install . 

At this point, sagelib will be a standard pip-installable Python source package, ready for upload to PyPI.

Binary packages (using wheels etc.) is beyond the scope of this ticket.

. . . . . . . . . . . . . . . . .

See also:

• Meta-meta-ticket: Build, packaging, testing improvements #29133 META-META-TICKET: Build, packaging, testing improvements

. . . . . . . . . . . . . . References . . . . . . . . . . . . . . .

• https://docs.python.org/2/library/distutils.html
• https://setuptools.readthedocs.io/en/latest/setuptools.html#command-reference
• https://docs.python.org/2/distutils/sourcedist.html
• https://docs.python.org/2/install/
• https://pip.pypa.io/en/stable/reference/pip_install/#pip-install-examples
• http://askubuntu.com/questions/802544/is-sudo-pip-install-still-a-broken-practice
• http://stackoverflow.com/questions/33004708/osx-el-capitan-sudo-pip-install-oserror-errno-1-operation-not-permitted/33004920#33004920
• https://packaging.python.org/science/
• https://packaging.python.org/current/
• https://packaging.python.org/key_projects/#setuptools
• https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
• https://www.cac.cornell.edu/wiki/index.php?title=Python_Distutils_Tips
• https://blog.ionelmc.ro/2014/05/25/python-packaging/
• http://stackoverflow.com/questions/7522250/how-to-include-package-data-with-setuptools-distribute

CC: @jdemeyer @williamstein @vbraun @videlec @dimpase @kiwifb @embray @nexttime @sagetrac-aenge @nthiery @miguelmarco @kwankyu @robertwb @infinity0 @tobihan @defeo @slel @timokau @kevinywlui @isuruf @jhpalmieri

Component: build

Keywords: pip, PyPI

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/21507

[embray]

comment:10

Thanks for taking the lead on this. There's more still be to added to the list of issues, such as providing binary wheels and how that will be handled. I think it will be important to also make more progress on making some hard dependencies optional (that is, allow optional features to fail gracefully if a dependency for that feature is not found).

[mkoeppe]

comment:11

Replying to @embray:

There's more still be to added to the list of issues, such as providing binary wheels and how that will be handled.

Definitely. I'm hoping that we can use this ticket to organize it.

I think it will be important to also make more progress on making some hard dependencies optional (that is, allow optional features to fail gracefully if a dependency for that feature is not found).

I agree.

[mkoeppe]

comment:12

Here's a question for the distutils experts. Is a Python package allowed to install things in bin/? Do any other Python packages do that?
Where would one install helper scripts? Is there a Python equivalent of $prefix/libexec?

[williamstein]

comment:14

This Python package, which is part of SMC, installs many things to /usr/local/bin using completely standard approaches: https://github.com/sagemathinc/smc/tree/master/src/smc_pyutil. See, in particular, the console_scripts section of setup.py: https://github.com/sagemathinc/smc/blob/master/src/smc_pyutil/setup.py#L58

[mkoeppe]

comment:15

Thanks a lot! Following your pointer, I've found this documentation: http://python-packaging.readthedocs.io/en/latest/command-line-scripts.html, which I will refer to alongside the example that you pointed out.

[mkoeppe]

comment:17

This now #21569.

[embray]

comment:21

I don't like the phrasing "PyPI package". There's no such thing as a "PyPI package" per se.
Let's be more clear about what we want here--should it be buildable from a source package? Or are we providing binary wheels? Both? Neither?

[embray]

comment:22

I wonder if this work might not also benefit from adopting portions of astropy-helpers into Sage's build tools. I don't think it would necessarily be appropriate to use directly--although there's nothing very "astropy" specific in astropy-helpers, a lot of it does assume one is using the astropy project template, or something close to it. But large swaths of it are very generic, and I put a lot of work into developing tools for build sanity of large Python projects.

Some key features that I think might benefit sage include:

• per-subpackage setup_package.py modules, which can contain any number of a (mostly) well-documented hooks for building compiled modules in that sub-package. This provides a clear and explicit alternative to a monolithic module_list.py (see the discussion here), and would also make it easier to break off some of Sage's subpackages into standalone packages should the time ever come for that.
• one of the less-well-documented features is also that setup_package.py can contain hooks for setup.py commands. For example, if a module contains some generated code (a la sage_setup.autogen), it can include a pre_build_ext hook to make sure the generated code is up to date before running the ./setup.py build_ext command (either explicitly, or implicitly as a sub-command). This makes it relatively easy for individual submodules to put hooks into the build process without having to write complicated command subclasses cluttered with subpackage-specific crud.
• smarter handling of Cython modules--for example, when creating a source tarball it will make sure all Cython modules have been Cythonized, and it will include the resulting C/C++ sources in the source tarball. This ensures that the user does not need Cython to be installed in order to build from source (which is good, because even if they do have Cython it will often be the wrong version--and in Sage's case missing important patches as well).

Those are the main ones for now but I could probably think of more.

[mkoeppe]

comment:23

Replying to @embray:

I don't like the phrasing "PyPI package". There's no such thing as a "PyPI package" per se.
Let's be more clear about what we want here--should it be buildable from a source package? Or are we providing binary wheels? Both? Neither?

The scope of this ticket is a source package that is pip-installable and would be listed as such on PyPI.

Binary packaging (about which I don't know anything at the moment) is beyond the scope of this ticket.

[mkoeppe]

comment:24

Replying to @embray:

I wonder if this work might not also benefit from adopting portions of astropy-helpers into Sage's build tools.

This all sounds great and seems like it could be part of #21508 or be a follow up to it.

[mkoeppe]

comment:25

I've updated the description to explain the scope of the ticket better.

[slel]

comment:53

There is interest in this process, see
Ask Sage #40947: SageMath as a regular python package.

[jdemeyer]

comment:54

Added #25546

[mkoeppe]

comment:58

I propose that we get this task ticket done in 2020.

[Shlokatadistance]

comment:61

The major problem I feel with making something like Sage into a python package is owing to the enormous complexity of Sagemath, the application itself has such major features , and I am not sure how much would we need to modify the application in order to help the proper incorporation of Sage into ordinary python code.

[mkoeppe]

comment:62

Yes, the complexity is why we use a meta-ticket to organize the necessary steps.

If you want to help, you can work on a well-defined ticket such as #21516.

[Shlokatadistance]

comment:63

Hi sorry for the delay, I'll start working on this right away
Replying to @mkoeppe:

Yes, the complexity is why we use a meta-ticket to organize the necessary steps.

If you want to help, you can work on a well-defined ticket such as #21516.

[mkoeppe]

comment:69

Let's close this ticket. Meta-ticket #29705 describes the updated plan.

[dimpase]

Reviewer: Dima Pasechnik