Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document OS-specific approaches for noarch packages #1839

Merged
merged 13 commits into from
Jun 14, 2023
119 changes: 119 additions & 0 deletions src/maintainer/knowledge_base.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1028,9 +1028,128 @@ In order to qualify as a noarch python package, all of the following criteria mu
which builds on Linux `and` Windows, with ``build_number`` offsets to create a pair packages, like
``dataclasses``.

.. hint::

You can build platform-specific ``noarch`` packages to include runtime requirements depending on the target OS.
See mini-tutorial below.

If an existing python package qualifies to be converted to a noarch package, you can request the required changes
by opening a new issue and including ``@conda-forge-admin, please add noarch: python``.

.. _os_specific_noarch:

Noarch packages with OS-specific dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is possible to build ``noarch`` packages with runtime requirements that depend on the target OS
(Linux, Windows, MacOS), regardless the architecture (amd64, ARM, PowerPC, etc). This approach
relies on three concepts:

1. `Virtual packages <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-virtual.html>`__.
Prefixed with a double underscore, they are used by conda to represent system properties as
constraints for the solver at install-time. We will use ``__linux``, ``__win`` or ``__osx``,
which are only present when the running platform is Linux, Windows, or MacOS, respectively.
``__unix`` is present in both Linux and MacOS. Note that this feature is **only fully available
on conda 4.10 or above**.
2. ``conda-forge.yml``'s :ref:`noarch_platforms` option.

The idea is to generate different noarch packages for each OS needing different dependencies.
Let's say you have a pure Python package, perfectly eligible for ``noarch: python``, but on Windows
it requires ``windows-only-dependency``. You might have something like:

.. code-block:: yaml
:caption: recipe/meta.yaml (original)

name: package
source:
# ...
build:
number: 0
requirements:
# ...
run:
- python
- numpy
- windows-only-dependency # [win]

Being non-noarch, this means that the build matrix will include at least 12 outputs: three platforms,
times four Python versions. This gets worse with arm64, aarch64 and ppc64le in the mix. We can get it down
to two outputs if replace it with this other approach!

.. code-block:: yaml+jinja
:caption: recipe/meta.yaml (modified)

name: package
source:
# ...
build:
number: 0
# You NEED to include the platform in the build string to avoid hash collisions
string: "unix_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [unix]
string: "win_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [win]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to avoid manipulating the string and instead rely on different hashes (note Jaime is mentioning something similar elsewhere)?

The main issue with string is pretty finicky. Users need to be very careful to replicate conda-build's behavior and if they missing something they could wind up with duplicate packages (which sometimes can both be accepted by Anaconda.org with neither being downloadable).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be needed until conda/conda-build#4606 lands and a new conda-build release is cut.

Is it ok to merge as is or would you rather have a little admonition mentioning this caveat?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding from the thread above is using conda_build_config.yaml meant we would not need this.

Fixing conda-build to not need either is certainly better. That said, between setting a string and using conda_build_config.yaml, would prefer the latter (fewer foot guns).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am partial to conda_build_config.yaml too as we transition to the new conda-build version.

noarch: python
requirements:
# ...
run:
- python
- numpy
isuruf marked this conversation as resolved.
Show resolved Hide resolved
- __unix # [unix]
- __win # [win]
- windows-only-dependency # [win]

Do not forget to specify the platform virtual packages with their selectors!
Otherwise, the solver will not be able to choose the variants correctly.

By default, conda-forge will only build ``noarch`` packages on a ``linux_64`` CI runner, so
only the ``# [unix]`` selectors would be true. However, we can change this behaviour using
the ``noarch_platforms`` option in ``conda-forge.yml``:

.. code-block:: yaml
:caption: conda-forge.yml

noarch_platforms:
- linux_64
- win_64

This will provide two runners per package! Perfect! All these changes require a
feedstock rerender to be applied. See :ref:`dev_update_rerender`.

If you need conditional dependencies on all three operating systems, this is how you do it:

.. code-block:: yaml+jinja
:caption: recipe/meta.yaml

name: package
source:
# ...
build:
number: 0
# You NEED to include the platform in the build string to avoid hash collisions
string: "linux_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [linux]
string: "osx_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [osx]
string: "win_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [win]
noarch: python
requirements:
# ...
run:
- python
- numpy
- __linux # [linux]
- __osx # [osx]
- __win # [win]
- linux-only-dependency # [linux]
- osx-only-dependency # [osx]
- windows-only-dependency # [win]

.. code-block:: yaml
:caption: conda-forge.yml

noarch_platforms:
- linux_64
- osx_64
- win_64

Again, remember to rerender after adding / modifying these files so the changes are applied.

Noarch generic
--------------
Expand Down