diff --git a/src/doc/en/developer/coding_basics.rst b/src/doc/en/developer/coding_basics.rst index 7471a9d4022..00844de6f4a 100644 --- a/src/doc/en/developer/coding_basics.rst +++ b/src/doc/en/developer/coding_basics.rst @@ -89,9 +89,9 @@ In particular, Files and directory structure ============================= -Roughly, the Sage directory tree is layout like this. Note that we use -``SAGE_ROOT`` in the following as a shortcut for the (arbitrary) name -of the directory containing the Sage sources: +Roughly, the Sage directory tree is laid out like this. Note that we +use ``SAGE_ROOT`` in the following as a shortcut for the name of the +directory containing the Sage sources: .. CODE-BLOCK:: text @@ -104,7 +104,7 @@ of the directory containing the Sage sources: setup.py ... sage/ # Sage library - ext_data/ # extra Sage resources (formerly src/ext) + ext_data/ # extra Sage resources (legacy) bin/ # the scripts in local/bin that are tracked upstream/ # tarballs of upstream sources local/ # installed binaries @@ -149,15 +149,36 @@ Adding new top-level packages below :mod:`sage` should be done sparingly. It is often better to create subpackages of existing packages. -Non-Python Sage source code and supporting files can be included in one -of the following places: +Non-Python Sage source code and small supporting files can be +included in one of the following places: - In the directory of the Python code that uses that file. When the Sage library is installed, the file will be installed in the same - location as the Python code. For example, - ``SAGE_ROOT/src/sage/interfaces/maxima.py`` needs to use the file - ``SAGE_ROOT/src/sage/interfaces/maxima.lisp`` at runtime, so it refers - to it as :: + location as the Python code. This is referred to as "package data". + + The preferred way to access the data from Python is using the + `importlib.resources API + `_, + in particular the function :func:`importlib.resources.files`. + Using it, you can: + + - open a resource for text reading: ``fd = files(package).joinpath(resource).open('rt')`` + - open a resource for binary reading: ``fd = files(package).joinpath(resource).open('rb')`` + - read a resource as text: ``text = files(package).joinpath(resource).read_text()`` + - read a resource as bytes: ``bytes = files(package).joinpath(resource).read_bytes()`` + - open an xz-compressed resource for text reading: ``fd = lzma.open(files(package).joinpath(resource).open('rb'), 'rt')`` + - open an xz-compressed resource for binary reading: ``fd = lzma.open(files(package).joinpath(resource).open('rb'), 'rb')`` + + If the file needs to be used outside of Python, then the + preferred way is using the context manager + :func:`importlib.resources.as_file`. It should be imported in the + same way as shown above. + +- Older code in the Sage library accesses + the package data in more direct ways. For example, + ``SAGE_ROOT/src/sage/interfaces/maxima.py`` uses the file + ``SAGE_ROOT/src/sage/interfaces/maxima.lisp`` at runtime, so it + refers to it as:: os.path.join(os.path.dirname(__file__), 'sage-maxima.lisp') @@ -169,11 +190,39 @@ of the following places: from sage.env import SAGE_EXTCODE file = os.path.join(SAGE_EXTCODE, 'directory', 'file') -In both cases, the files must be listed (explicitly or via wildcards) in + This practice is deprecated, see :issue:`33037`. + +In all cases, the files must be listed (explicitly or via wildcards) in the section ``options.package_data`` of the file ``SAGE_ROOT/pkgs/sagemath-standard/setup.cfg.m4`` (or the corresponding file of another distribution). +Large data files should not be added to the Sage source tree. Instead, it +is proposed to do the following: + +- create a separate git repository and upload them there [2]_, + +- add metadata to the repository that make it a pip-installable + package (distribution package), as explained for example in the + `Python Packaging User Guide + `_, + +- `upload it to PyPI + `_, + +- create metadata in ``SAGE_ROOT/build/pkgs`` that make your new + pip-installable package known to Sage; see :ref:`chapter-packaging`. + +For guiding examples of external repositories that host large data +files, see https://github.com/sagemath/conway-polynomials, and +https://github.com/gmou3/matroid-database. + +.. [2] + + It is also suggested that the files are compressed, e.g., through + the command ``xz -e``. They can then be read via a command such as + ``lzma.open(file, 'rt')``. + Learn by copy/paste =================== diff --git a/src/doc/en/developer/coding_in_python.rst b/src/doc/en/developer/coding_in_python.rst index 6b3b936662d..fef7a1011f0 100644 --- a/src/doc/en/developer/coding_in_python.rst +++ b/src/doc/en/developer/coding_in_python.rst @@ -7,6 +7,7 @@ Coding in Python for Sage This chapter discusses some issues with, and advice for, coding in Sage. +.. _section-python-language-standard: Python language standard ========================