-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CEP: support for ABI3 packages #86
base: main
Are you sure you want to change the base?
Changes from 3 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,245 @@ | ||
# Support for abi3 python packages | ||
|
||
<table> | ||
<tr><td> Title </td><td> Support for abi3 python packages </td> | ||
<tr><td> Status </td><td> Draft </td></tr> | ||
<tr><td> Author(s) </td><td> Isuru Fernando <ifernando@quansight.com></td></tr> | ||
<tr><td> Created </td><td> July 01, 2023</td></tr> | ||
</table> | ||
|
||
## Abstract | ||
|
||
This CEP specifies how ABI3 python packages are supported in conda install tools | ||
(conda/mamba/micromamba/pixi) and how they are built in conda build tools | ||
(conda-build/rattler-build). | ||
|
||
## Motivation | ||
|
||
When building extensions for python, they might use python minor version | ||
specific symbols. This results in the extension being usable only on that minor | ||
version. These extensions are identified by the extension suffix. | ||
For eg: | ||
|
||
foo.cpython-310-x86_64-linux-gnu.so | ||
|
||
is an extension that support only CPython 3.10 on x86_64-linux-gnu platform. | ||
|
||
However some symbols are available in all python major.minor versions with some | ||
lower bound on the python version. These symbols are part of the | ||
[limited C API]([C_API_Stability]). It is guaranteed that the symbols Stable ABI | ||
introduced in Python 3.X are available in Python 3.Y for any `Y >= X`. | ||
Extensions using only these symbols are identified by the extension suffix | ||
`abi3.so`. For eg: | ||
|
||
foo.abi3.so | ||
|
||
These extensions only support the platform it was built for (for eg: | ||
`x86_64-linux-gnu`), but is not specified in the extension suffix. | ||
|
||
Note that the stable ABI is only specific to CPython and is not compatible with | ||
PyPy or other Python implementations. For a Python implementation independent | ||
ABI, see the [HPy project](HPy). | ||
|
||
The motivation for building abi3 packages is that we only need to build the | ||
extension for one python version and the extension will work for any python | ||
later version. This reduces build matrix from 4-5 python minor versions to one | ||
python minor version and reduces the maintenance burden of package builders. | ||
|
||
## noarch: python packages | ||
|
||
abi3 packages are python version independent and we will first look at | ||
`noarch: python` packages that are also python version independent and in addition | ||
are arch independent. | ||
|
||
`noarch: python` packages have several attributes to them: | ||
|
||
A1. They have `subdir: noarch` in `info/index.json`. | ||
|
||
A2. They have `noarch: python` in `info/index.json`. | ||
|
||
A3. Python files are in `<PREFIX>/site-packages`. | ||
|
||
A4. Entry points are recorded in `info/link.json`. | ||
|
||
A conda install tool does four things to support them: | ||
|
||
B1. Files in `<PREFIX>/site-packages` are moved to the correct location. Eg: | ||
`<PREFIX>/lib/python3.10/site-packages`. | ||
|
||
B2. python files (files ending with `*.py`) are compiled to `.pyc` files. Eg: | ||
`<PREFIX>/lib/python3.10/site-packages/foo.py` is compiled to | ||
`<PREFIX>/lib/python3.10/site-packages/__pycache__/foo.cpython-310.pyc`. | ||
|
||
B3. `.pyc` files created are recorded in `<PREFIX>/conda-meta/<pkg>.json` | ||
so that they are removed properly when the package is uninstalled. | ||
|
||
B4. Entry points in `info/link.json` are materialised. | ||
|
||
### info/link.json file | ||
An example `info/link.json` for `noarch: python` looks like | ||
|
||
```json | ||
{ | ||
"noarch": { | ||
"entry_points": [ | ||
"isympy = isympy:main" | ||
], | ||
"type": "python" | ||
}, | ||
"package_metadata_version": 1, | ||
"preferred_env": "foo" | ||
} | ||
``` | ||
|
||
An example for `info/link.json` for `noarch: generic` looks like | ||
``` | ||
{ | ||
"noarch": { | ||
"type": "generic" | ||
}, | ||
"package_metadata_version": 1 | ||
} | ||
``` | ||
|
||
|
||
Here `preferred_env` is ignored by conda since 2017 and is not supported by | ||
other conda install tools. Therefore `info/link.json` is used exclusively | ||
for `noarch` packages and out of the two types, `noarch: generic` packages | ||
does not require any special action. | ||
|
||
### info/index.json file | ||
|
||
An example for a `noarch: python` recipe. | ||
|
||
```json | ||
{ | ||
"arch": null, | ||
"build": "pyh2585a3b_103", | ||
"build_number": 103, | ||
"depends": [ | ||
"mpmath >=0.19", | ||
"python >=3.8" | ||
], | ||
"license": "BSD-3-Clause", | ||
"license_family": "BSD", | ||
"name": "sympy", | ||
"noarch": "python", | ||
"platform": null, | ||
"subdir": "noarch", | ||
"timestamp": 1718625708903, | ||
"version": "1.12.1" | ||
} | ||
``` | ||
|
||
### Current behaviour in solver tools | ||
|
||
Conda package upload tools like `anaconda-client` use `A1` to upload | ||
the package to the `noarch` subdir. | ||
|
||
Conda install tools have slightly different behaviour. | ||
|
||
Conda: | ||
1. Actions `B1, B2, B3` are applied for packages with `A3`. | ||
2. Action `B4` is applied for packages with `A4`. | ||
|
||
Micromamba: | ||
1. Actions `B1, B2, B3` are applied for packages with both `A2, A3`. | ||
2. Action `B4` is applied for packages with both `A2, A4`. | ||
|
||
|
||
## Implementation for abi3 packages in install tools. | ||
|
||
In order to support abi3 packages, we propose two methods. | ||
|
||
### `package_metadata_version = 1` | ||
|
||
We require the following attributes in abi3 packages: | ||
|
||
C1. They have `subdir: <platform>` where `<platform>` is the subdir | ||
that the package was built for. | ||
|
||
C2. They have `noarch: python`. | ||
|
||
C3. `A2, A3, A4` are applied. | ||
|
||
This is compatible with `conda/mamba/micromamba` install tools | ||
currently. This requires support from build tools to set `subdir: <platform>` | ||
only. | ||
|
||
In particular an option: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it planned that the conda-forge tooling would check for this option and then set up Windows/Mac/Linux builds even though There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, we can do that. |
||
``` | ||
build: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think for V1 recipes we would use: build:
python:
version_independent: true/false |
||
python_version_independent: true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i think something like python:
abi: abi3 leaves open enough space to support future expansion of other abis that may require special behavior for example python:
abi: hpy There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mariusvniekerk, both those options need the exact same thing. (B1-B4) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here's what a conda-build PR would look like conda/conda-build#5456 |
||
``` | ||
to imply a python version independent package and set `noarch: python` | ||
in `info/index.json`. | ||
|
||
### `package_metadata_version = 2` | ||
|
||
In this method, we require additional support from install tools. | ||
|
||
For `abi3` and `noarch: python` pacakges, we record the `entry_points` and | ||
the pure python files in `info/link,json`. | ||
isuruf marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```json | ||
{ | ||
"python": { | ||
"entry_points": [ | ||
"foo = foo:main" | ||
], | ||
"py_compile": [ | ||
"lib/python/site-packages/foo/main.py", | ||
"lib/python/site-packages/foo/__init__.py", | ||
] | ||
}, | ||
"package_metadata_version": 2, | ||
} | ||
``` | ||
|
||
We require the following support from install tools | ||
|
||
D1: Apply action `B4` if `python: entry_points` is present in `info/link.json`. | ||
|
||
D2: Apply actions `B2, B3` if `python: py_compile` is present in `info/link.json`. | ||
|
||
D3: Provide a `__supports_package_metadata_version=2` virtual package. | ||
|
||
Note that we do not require `B1` as package authors should depend on a python | ||
version that has a custom `site.py` that adds `lib/python/site-packages` to | ||
the path. | ||
Comment on lines
+221
to
+223
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Doesnt this imply that older python versions available on for instance conda-forge will never have the ability to use abi3 packages? If we are modifying the installer we could just as well move the Python files to the python specific site-packages directory. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, but we can also have the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But relying on a special package might also be problematic if you are not using conda-forge though. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm hoping we can add support in defaults too. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would not prefer this because it means the proper functioning of the installer is dependent on a specific package. I'd rather encode the behavior in the installer itself. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that the installer should do less special case stuff. But I also don't think that it should depend on specific packages to function properly. Even with this proposal the installer does already have special case behavior. However, without the dependency on a package that introduces a However, one could argue that the same holds for depending on It would be ideal if a recipe author doesn't have to care about these details. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could make the build tool add this package, then the installer does not have to care about it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we do this for other packages? Because then we tie the build-tool to specific package names which might also not be ideal. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We take the C compiler name from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh I forgot about |
||
|
||
We require additional support from build tools. | ||
|
||
E1. Move contents in `<SP_DIR>` to `<PREFIX>/lib/python/site-packages` | ||
instead of `<PREFIX>/site-packages`. | ||
|
||
E2. Record pure python `.py` files to be compiled in `info/link.json`. | ||
TODO: not sure if this is required. We can infer this from `info/index.json`. | ||
|
||
E3: For `noarch: generic` packages, we do not require `info/link.json` file and | ||
build tools are recommended to not produce a `info/link.json` file. | ||
|
||
E4: Adds a `__supports_package_metadata_version>=2` in `run`. | ||
|
||
## Alternatives considered | ||
|
||
### Apply all actions in a `post-link.sh` script. | ||
|
||
A draft work provided at [python-feedstock](python-pr-669) | ||
This was suggested by `@mbargull`, but some community members (@baszalmstra, | ||
@wolfv) does not prefer post-link scripts as they can be used for arbitrary | ||
code execution. However in the author's opinion, this attack vector is not a | ||
new one since the install tool uses the python executable in the host | ||
environment to compile the python files. | ||
|
||
### `noarch: python` packages with `__linux, __osx, __win` constrains. | ||
|
||
This suggestion by `@wolfv` is not ideal as this clutters `noarch` subdir | ||
`repodata.json` file with packages that are useless for the platform in question. | ||
|
||
<!--links--> | ||
[C_API_Stability]: https://docs.python.org/3/c-api/stable.html | ||
|
||
[HPy]: https://hpyproject.org | ||
|
||
[python-pr-669]: https://github.com/conda-forge/python-feedstock/pull/669 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For reference, rattler follows the same behavior as micromamba.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm interested what your suggestion is for this mismatch in behavior, should conda be adapted to follow mamba and rattler in this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My suggestion is to keep the existing mismatched behaviour in
package_metadata_version = 1
and make this uniform inpackage_metadata_version = 2