Document OS-specific approaches for noarch packages #1839

jaimergp · 2022-10-21T12:44:32Z

PR Checklist:

note any issues closed by this PR with closing keywords
put any other relevant information below

This creative approach is used by several feedstocks (10 or so, according to this Github CodeSearch query) now but wasn't documented. This approach alleviates the number of builds needed for platform-specific dependencies in noarch-eligible packages.

mattip · 2022-10-21T12:59:18Z

src/maintainer/knowledge_base.rst

+It is possible to build ``noarch`` packages with runtime requirements that depend on the target OS (Linux, Windows,
+MacOS), regardless the architecture (amd64, ARM, PowerPC, etc). This approach relies on four concepts:
+
+1.  Virtual packages. Prefixed with a double underscore, they are used by conda to represent properties of the running system 


Suggested change

1. Virtual packages. Prefixed with a double underscore, they are used by conda to represent properties of the running system

1. Virtual packages. Prefixed with a double underscore, they are used by conda

when running the dependency resolver to represent properties of the currently

activated environment

Same as below, technically they are environment independent unless overridden.

mattip · 2022-10-21T12:59:54Z

src/maintainer/knowledge_base.rst

+
+1.  Virtual packages. Prefixed with a double underscore, they are used by conda to represent properties of the running system 
+    as constraints for the solver. We will use ``__linux``, ``__win`` or ``__osx``, which are only present when
+    the running platform is Linux, Windows, or MacOS, respectively. ``__unix`` is present in both Linux and MacOS. Note


Suggested change

the running platform is Linux, Windows, or MacOS, respectively. ``__unix`` is present in both Linux and MacOS. Note

the activated environment is Linux, Windows, or MacOS, respectively. ``__unix`` is present in both Linux and MacOS. Note

I think one can only change the virtual packages with CONDA_OVERRIDE_* env variables. Otherwise, regardless the active environment, it will still report the native platform type.

OK, thanks for the clarification. I don't know if "running platform" is defined somewhere?

also, it is not clear to me from "as constraints for the solver" that this happens at runtime, and not while building the package.

I'll rephrase, thanks!

src/maintainer/knowledge_base.rst

jakirkham · 2022-10-21T15:30:53Z

Thanks for adding this, Jaime! 🙏

cc @minrk (in case you have thoughts on this 🙂)

src/maintainer/knowledge_base.rst

isuruf · 2022-10-25T14:12:53Z

src/maintainer/knowledge_base.rst

+      - __{{ target_os }}
+      {% if target_os == 'win' %}
+      - windows-only-dependency
+      {% endif %}


I don't understand why we need target_os.
We can just do

Suggested change

- __{{ target_os }}

{% if target_os == 'win' %}

- windows-only-dependency

{% endif %}

{% if win %}

- windows-only-dependency

- __win

{% else %}

- __unix

{% endif %}

Mmm possibly, I need to double check but indeed we might not need it. Thanks!

Ok my main concern about not using conda_build_config.yaml, @isuruf:

The package hashes will be the same (see Noarch click-feedstock#44), so the build string NEEDS to be prefixed with the platform value. If the user forgets about this, it will result in a filename collision. This should be checked by the linter, at the very least. I am unaware if sharing a hash is problematic for other cases.

I share this concern and would prefer not using string. Explained in more detail below ( #1839 (comment) )

See conda/conda-build#4606

jakirkham · 2022-10-29T10:43:29Z

src/maintainer/knowledge_base.rst

+    string: "unix_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}"  # [unix]
+    string: "win_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}"   # [win]


Would it be possible to avoid manipulating the string and instead rely on different hashes (note Jaime is mentioning something similar elsewhere)?

The main issue with string is pretty finicky. Users need to be very careful to replicate conda-build's behavior and if they missing something they could wind up with duplicate packages (which sometimes can both be accepted by Anaconda.org with neither being downloadable).

This will be needed until conda/conda-build#4606 lands and a new conda-build release is cut.

Is it ok to merge as is or would you rather have a little admonition mentioning this caveat?

My understanding from the thread above is using conda_build_config.yaml meant we would not need this.

Fixing conda-build to not need either is certainly better. That said, between setting a string and using conda_build_config.yaml, would prefer the latter (fewer foot guns).

I am partial to conda_build_config.yaml too as we transition to the new conda-build version.

jaimergp · 2022-11-08T16:21:41Z

After a conversation with @viniciusdc, we realized a simplified version of this approach can also be used to skip OS builds. A common use case that requires non-noarch is to skip Windows, so we can just add a __unix dependency to achieve the same result under a noarch scheme.

BastianZim · 2022-12-16T12:16:30Z

Is everyone happy with merging this? Would be good to have it in the docs.

src/maintainer/knowledge_base.rst

jaimergp · 2022-12-16T15:35:28Z

Is everyone happy with merging this? Would be good to have it in the docs.

We need to agree on the "best" solution to avoid the hash collision. See discussion here.

src/maintainer/knowledge_base.rst

Co-authored-by: Ryan May <rmay31@gmail.com>

hmaarrfk

thanks using this now.

src/maintainer/knowledge_base.rst

Co-authored-by: Mark Harfouche <mark.harfouche@gmail.com>

jaimergp · 2023-05-15T22:03:30Z

The conda-build PR we were waiting for was merged recently and should be available in the upcoming conda-build 3.25, which means we will be able to merge this longstanding PR once and for all!

asmeurer · 2023-05-15T22:16:40Z

The linter will need to be updated to recognize this approach. See for instance conda-forge/sympy-feedstock#52.

jaimergp · 2023-05-17T09:24:16Z

@asmeurer, it was updated a while ago (see conda-forge/conda-smithy#1682), but only for simple platform selectors (e.g. # [win]) not custom ones like in your example (# [with_gmpy]).

mikemhenry · 2023-06-12T21:40:16Z

Just a friendly bump, I was looking for this recently and it would be great for the documentation to be easier to find, any blockers left?

jaimergp · 2023-06-13T17:27:50Z

This is ready to go from my side. @conda-forge/core if someone has the time to click merge :) 🙏

Document OS-specific approaches for noarch packages

8af376f

jaimergp requested a review from a team as a code owner October 21, 2022 12:44

fix label

c33f553

mattip reviewed Oct 21, 2022

View reviewed changes

fix yaml blocks

d17fb46

mattip reviewed Oct 21, 2022

View reviewed changes

src/maintainer/knowledge_base.rst Outdated Show resolved Hide resolved

jaimergp added 4 commits October 21, 2022 15:09

add details about build string

99af8f6

use yaml+jinja

fa8b4ec

rephrase and rewrap

8dbc5e4

noarch_platforms uses underscores

a893f82

mattip reviewed Oct 22, 2022

View reviewed changes

src/maintainer/knowledge_base.rst Outdated Show resolved Hide resolved

mattip mentioned this pull request Oct 22, 2022

add noarch label conda-forge/blessed-feedstock#20

Merged

5 tasks

goanpeca approved these changes Oct 23, 2022

View reviewed changes

isuruf reviewed Oct 25, 2022

View reviewed changes

jakirkham mentioned this pull request Oct 25, 2022

Use noarch: python (with __win requirement) conda-forge/wincertstore-feedstock#24

Merged

1 task

isuruf mentioned this pull request Oct 26, 2022

Packages that can be noarch #1840

Open

use selectors and get rid of conda_build_config indirection

00ecadc

jakirkham reviewed Oct 29, 2022

View reviewed changes

jakirkham mentioned this pull request Oct 29, 2022

Noarch conda-forge/click-feedstock#44

Merged

5 tasks

jaimergp mentioned this pull request Oct 31, 2022

include virtual packages in hash contents conda/conda-build#4606

Merged

3 tasks

BastianZim mentioned this pull request Nov 7, 2022

Make package noarch conda-forge/spyder-kernels-feedstock#86

Merged

5 tasks

viniciusdc mentioned this pull request Nov 8, 2022

Add Nebari conda-forge/staged-recipes#21048

Merged

10 tasks

ocefpaf mentioned this pull request Nov 14, 2022

Added pynsist as a package conda-forge/staged-recipes#20138

Closed

10 tasks

BastianZim mentioned this pull request Dec 15, 2022

poetry v1.3.0 conda-forge/poetry-feedstock#77

Merged

3 tasks

BastianZim mentioned this pull request Dec 16, 2022

Make noarch conda-forge/poetry-feedstock#80

Merged

5 tasks

BastianZim reviewed Dec 16, 2022

View reviewed changes

src/maintainer/knowledge_base.rst Outdated Show resolved Hide resolved

dopplershift reviewed Dec 19, 2022

View reviewed changes

src/maintainer/knowledge_base.rst Outdated Show resolved Hide resolved

jaimergp and others added 2 commits December 20, 2022 16:04

Update src/maintainer/knowledge_base.rst

b43e58b

Co-authored-by: Ryan May <rmay31@gmail.com>

Update knowledge_base.rst

6e93d0b

jakirkham mentioned this pull request Feb 16, 2023

Can a package bind to a platform, but not a specific python version? #1900

Closed

jaimergp mentioned this pull request Mar 16, 2023

Gallery of toy recipes that implement "known conda-build patterns" #1908

Open

hmaarrfk reviewed Mar 18, 2023

View reviewed changes

src/maintainer/knowledge_base.rst Outdated Show resolved Hide resolved

asmeurer mentioned this pull request Mar 20, 2023

Can't install SymPy 1.10 in a Python 3.11 env conda-forge/sympy-feedstock#50

Open

1 task

This was referenced Apr 14, 2023

Make noarch conda-forge/hyperspyui-feedstock#16

Merged

Make recipe noarch conda-forge/pyebsdindex-feedstock#5

Merged

jaimergp mentioned this pull request Apr 23, 2023

Adding h5glance and htmlgen conda-forge/staged-recipes#22647

Merged

10 tasks

h-vetinari mentioned this pull request May 2, 2023

Include virtual packages in hash contents conda/conda-build#4861

Closed

2 tasks

mikemhenry mentioned this pull request May 2, 2023

Installing openmm-ml does not install nnpops openmm/openmm-ml#54

Open

isuruf and others added 2 commits May 15, 2023 15:32

Lower bound on python

7b58e00

Co-authored-by: Mark Harfouche <mark.harfouche@gmail.com>

Remove hashes for conda-build 3.25.0

ca54541

missing space

0340b8b

xhochy approved these changes Jun 14, 2023

View reviewed changes

beckermr merged commit 032898b into conda-forge:main Jun 14, 2023

h-vetinari mentioned this pull request Jul 31, 2023

Why isn't this package noarch? conda-forge/pythran-feedstock#77

Open

ocefpaf mentioned this pull request Aug 21, 2023

Added glue-qt package conda-forge/staged-recipes#23697

Merged

10 tasks

BastianZim mentioned this pull request Dec 8, 2024

poetry v1.8.5, and remove build strings conda-forge/poetry-feedstock#105

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document OS-specific approaches for noarch packages #1839

Document OS-specific approaches for noarch packages #1839

jaimergp commented Oct 21, 2022 •

edited

Loading

mattip Oct 21, 2022

jaimergp Oct 21, 2022

mattip Oct 21, 2022

jaimergp Oct 21, 2022

mattip Oct 21, 2022

mattip Oct 21, 2022

jaimergp Oct 21, 2022

jakirkham commented Oct 21, 2022

isuruf Oct 25, 2022

jaimergp Oct 26, 2022

jaimergp Oct 28, 2022

jakirkham Oct 29, 2022

isuruf Oct 29, 2022

jakirkham Oct 29, 2022

jaimergp Oct 31, 2022

jakirkham Oct 31, 2022

jaimergp Oct 31, 2022

jaimergp commented Nov 8, 2022

BastianZim commented Dec 16, 2022

jaimergp commented Dec 16, 2022 •

edited

Loading

hmaarrfk left a comment

jaimergp commented May 15, 2023

asmeurer commented May 15, 2023

jaimergp commented May 17, 2023

mikemhenry commented Jun 12, 2023

jaimergp commented Jun 13, 2023

	the running platform is Linux, Windows, or MacOS, respectively. ``__unix`` is present in both Linux and MacOS. Note
	the activated environment is Linux, Windows, or MacOS, respectively. ``__unix`` is present in both Linux and MacOS. Note

		string: "unix_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [unix]
		string: "win_pyh{{ PKG_HASH }}_{{ PKG_BUILDNUM }}" # [win]

Document OS-specific approaches for noarch packages #1839

Document OS-specific approaches for noarch packages #1839

Conversation

jaimergp commented Oct 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakirkham commented Oct 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaimergp commented Nov 8, 2022

BastianZim commented Dec 16, 2022

jaimergp commented Dec 16, 2022 • edited Loading

hmaarrfk left a comment

Choose a reason for hiding this comment

jaimergp commented May 15, 2023

asmeurer commented May 15, 2023

jaimergp commented May 17, 2023

mikemhenry commented Jun 12, 2023

jaimergp commented Jun 13, 2023

jaimergp commented Oct 21, 2022 •

edited

Loading

jaimergp commented Dec 16, 2022 •

edited

Loading