Piggyback migrator for stdlib-migration #2135

h-vetinari · 2024-02-07T10:41:21Z

As the next big step for conda-forge/conda-forge.github.io#1844, write a piggyback-migrator for adding {{ stdlib("c") }} to all our compiled recipes. For now this is just aimed at one migration, with the aim to enable it for all migrations afterwards (see here).

Done:

handle selectors
handle m2w64-compilers
replace sysroot_linux-64 with corresponding spec in CBC
rewrite MACOSX_DEPLOYMENT_TARGET in CBC, remove lines involving __osx from meta.yaml

To do resp. to decide:

Do we want to defensively add it even for pure fortran recipes? AFAIU even {{ compiler("fortran") }} might use symbols from the C stdlib.
- Current approach: defensively add stdlib
Same question for rust.
- Current approach: not added
Do we want to use {{ stdlib("c") }} also on windows (even if empty)? If so, what will c_stdlib{,_version} be? This influences if we would need to add/modify selectors, which I would very much like to avoid.
- Current approach: If {{ compiler(...) }} for the output has a selector, copy that to {{ stdlib("c") }}.
How do we want to deal with outputs that have no build section (e.g. because there's just one global build phase)?
- Current approach: defensively add stdlib.
Do we want to handle CB2 recipes?
- Current approach: not done, because I expect it to be rare, and because I don't know how CB2 recipes look in attrs["meta_yaml"]["requirements"]. Also, the current logic depends on finding the lines for build: / host: / run, so this would need deeper changes.
figure out what changes (if any) related to _stub handling are necessary in various files here, e.g. conda_forge_tick/{migrators/arch.py,make_graph.py,...}

We can easily add more tests if you point me at some feedstocks you want to check.

Reuses some useful functionality (for slicing outputs, resp. testing aproach) from #1668.

codecov · 2024-02-07T10:49:31Z

Codecov Report

Attention: Patch coverage is 97.48428% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 71.46%. Comparing base (1d8f232) to head (47c2ea1).

Files	Patch %	Lines
conda_forge_tick/auto_tick.py	0.00%	2 Missing ⚠️
conda_forge_tick/migrators/cstdlib.py	98.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2135      +/-   ##
==========================================
+ Coverage   71.03%   71.46%   +0.42%     
==========================================
  Files          97       99       +2     
  Lines        9674     9833     +159     
==========================================
+ Hits         6872     7027     +155     
- Misses       2802     2806       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

h-vetinari · 2024-02-07T10:50:09Z

The add-trailing-comma hook here produces a false positive:

 pat_compiler = re.compile(
     r"(?P<indent>\s*)-\s*"
     r"(?P<compiler>\{\{\s*compiler\([\"\'](c|cxx|fortran)[\"\']\)\s*\}\})"
-    r"\s*(?P<selector>\#\s+\[[\w\s]+\])?"
+    r"\s*(?P<selector>\#\s+\[[\w\s]+\])?",
 )

This is a string that should not get a comma. Black handles trailing commas already, so I don't get why there's a separate hook.

beckermr · 2024-02-07T16:17:10Z

yeah that comma won't hurt but is odd. You can remove that hook in another PR if you'd like.

h-vetinari · 2024-02-07T16:22:21Z

yeah that comma won't hurt but is odd. You can remove that hook in another PR if you'd like.

Done: #2136

beckermr

Really nice! I had a few questions in the comments. Also, we should add a test where the compiler has a selector. I didn't see that covered in the current tests though I may have missed it!

conda_forge_tick/auto_tick.py

conda_forge_tick/migrators/cstdlib.py

tests/test_yaml/stdlib_arrow_after_meta.yaml

almost all outputs gain a host dependence on stdlib (including `apache-arrow-proc`, where we insert the host section due to the assumption that an output without a build section is relying on the global build requirements). Note that `libarrow-all` does not gain a stdlib-dependence, as it _has_ a non-empty build section. ``` diff --git a/tests/test_yaml/stdlib_arrow_before_meta.yaml b/tests/test_yaml/stdlib_arrow_after_meta.yaml index 85771bad..60b4550b 100644 --- a/tests/test_yaml/stdlib_arrow_before_meta.yaml +++ b/tests/test_yaml/stdlib_arrow_after_meta.yaml @@ -55,6 +55,7 @@ requirements: - autoconf # [linux] - make # [linux] host: + - {{ stdlib("c") }} # for required dependencies, see # https://github.com/apache/arrow/blob/apache-arrow-11.0.0/cpp/cmake_modules/ThirdpartyToolchain.cmake#L46-L75 - clangdev {{ llvm_version }} @@ -94,6 +95,8 @@ outputs: number: {{ proc_build_number }} string: {{ build_ext }} requirements: + host: + - {{ stdlib("c") }} run_constrained: # avoid installation with old naming of proc package - arrow-cpp-proc <0.0a0 @@ -185,6 +188,7 @@ outputs: - {{ compiler("cxx") }} - {{ compiler("cuda") }} # [cuda_compiler_version != "None"] host: + - {{ stdlib("c") }} - aws-crt-cpp - aws-sdk-cpp - brotli @@ -286,6 +290,7 @@ outputs: - {{ compiler("c") }} - {{ compiler("cxx") }} host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", exact=True) }} run: - {{ pin_subpackage("libarrow", exact=True) }} @@ -330,6 +335,7 @@ outputs: - {{ compiler("c") }} - {{ compiler("cxx") }} host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", exact=True) }} - {{ pin_subpackage("libarrow-acero", exact=True) }} - {{ pin_subpackage("libparquet", exact=True) }} @@ -381,6 +387,7 @@ outputs: - libgrpc # [build_platform != target_platform] - libprotobuf # [build_platform != target_platform] host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", exact=True) }} - libabseil - libgrpc @@ -435,6 +442,7 @@ outputs: - libgrpc # [build_platform != target_platform] - libprotobuf # [build_platform != target_platform] host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", exact=True) }} - {{ pin_subpackage("libarrow-flight", exact=True) }} - libprotobuf @@ -482,6 +490,7 @@ outputs: - {{ compiler("c") }} - {{ compiler("cxx") }} host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", max_pin="x") }} - libutf8proc # gandiva requires shared libllvm; needs to match version used at build time @@ -538,6 +547,7 @@ outputs: - cmake - ninja host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", exact=True) }} - {{ pin_subpackage("libarrow-acero", exact=True) }} - {{ pin_subpackage("libarrow-dataset", exact=True) }} @@ -588,6 +598,7 @@ outputs: - {{ compiler("c") }} - {{ compiler("cxx") }} host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow", max_pin="x") }} - openssl - thrift-cpp @@ -647,6 +658,7 @@ outputs: - cmake - ninja host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow-all", exact=True) }} - clangdev {{ llvm_version }} - llvmdev {{ llvm_version }} @@ -738,6 +750,7 @@ outputs: - cmake - ninja host: + - {{ stdlib("c") }} - {{ pin_subpackage("libarrow-all", exact=True) }} - {{ pin_subpackage('pyarrow', exact=True) }} - clangdev {{ llvm_version }} ```

``` diff --git a/tests/test_yaml/stdlib_polars_before_meta.yaml b/tests/test_yaml/stdlib_polars_after_meta.yaml index 4221271f..ae72755e 100644 --- a/tests/test_yaml/stdlib_polars_before_meta.yaml +++ b/tests/test_yaml/stdlib_polars_after_meta.yaml @@ -30,6 +30,7 @@ requirements: - make # [unix] - cargo-bundle-licenses host: + - {{ stdlib("c") }} # this is a hacky way to do cross-compilation from linux to windows - python # [not (build_platform == "linux-64" and target_platform == "win-64")] - pip # [not (build_platform == "linux-64" and target_platform == "win-64")] ```

due to limitations of the testing framework here, we still have different files because we need to go through a fake version migrator to test the piggyback.

h-vetinari · 2024-02-07T16:45:32Z

Thanks for the quick review!

Also, we should add a test where the compiler has a selector. I didn't see that covered in the current tests though I may have missed it!

Indeed, there aren't that many variations tested yet (though the code is written at least) - I didn't have a feedstock to hand for that, but I can look for one.

tests/test_yaml/stdlib_arrow_after_meta.yaml

added characters until the following search was (essentially) empty: https://github.com/search?q=org%3Aconda-forge+%2F%23+%5C%5B.*%5B%5E%5Cw%5Cs%28%29%3C%3E%21%3D.%2C%5C-%5C%27%5C%22%5D.*%5C%5D%2F+path%3Arecipe%2Fmeta.yaml&type=code

h-vetinari · 2024-02-08T14:16:21Z

Can we write to conda_build_config.yaml from a piggyback migrator? @beckermr

I was looking at all the recipes containing - sysroot_linux-64 2.17 already; it'd be nice if we could remove those lines and replace them with something like:

c_stdlib_version:   # [linux]
  - "2.17"          # [linux]

in the CBC.

h-vetinari · 2024-02-08T14:17:00Z

Do we want to use {{ stdlib("c") }} also on windows (even if empty)? If so, what will c_stdlib{,_version} be? This influences if we would need to add/modify selectors, which I would very much like to avoid.

@isuruf, would appreciate your input on this! 🙏

beckermr · 2024-02-08T14:51:06Z

Yes we can write to the conda build config if needed.

h-vetinari · 2024-02-08T15:18:33Z

Yes we can write to the conda build config if needed.

Cool, how would I do that? So far I've just been rewriting meta.yaml lines. 😅

beckermr · 2024-02-08T15:25:08Z

The entire feedstock is pulled via git and the bot's cwd is in the feedstock. Thus you should be to detect if a cbc yaml is there and make the appropriate changes and/or create one as needed. I forget if you need to git add the file or not, but I suspect a Quick Look at the code will indicate what to do.

h-vetinari · 2024-02-13T06:39:59Z

Ok. Don't let me block you on this in any case!

Mainly I'm waiting for input from Isuru, as I'm implementing his idea, without having necessarily grasped all the details.

Though the one thing you could help me with is the last point from the OP. I needed to teach the recipe parser how to handle the new jinja function (i.e. once recipes start using it), and so far I've only added it in conda_forge_tick/utils.py (see 4f2c04c), because that was the minimal thing to get the skip-migration test to work. If there are other parts of this repo that need to know about {{ stdlib(...) }}, please let me know!

beckermr · 2024-02-13T15:15:50Z

Right so if the stdlib packages have run exports attached to them, then we need to make sure those get accounted for in make_graph.py. I'd be good to add a test with the new jinja2 to the check solvable repo as well (https://github.com/regro/conda-forge-feedstock-check-solvable).

isuruf · 2024-02-13T16:03:02Z

For windows we can just do

c_stdlib:
  - vs

h-vetinari · 2024-02-13T16:51:52Z

For windows we can just do
c_stdlib:
  - vs

OK cool, thanks.

Right so if the stdlib packages have run exports attached to them, then we need to make sure those get accounted for in make_graph.py.

For now I just added this to COMPILER_STUBS_WITH_STRONG_EXPORTS in ec6f2ab - sounds like this should be enough? The stubs get created in the same way as for the compilers (again due to 4f2c04c).

It'd be good to add a test with the new jinja2 to the check solvable repo as well (https://github.com/regro/conda-forge-feedstock-check-solvable).

I think the next step for that would be to add c_stdlib: to the global pinning so it gets correctly populated when trying to resolve. Since we now have all three metapackages together, that should be feasible without breaking anything

beckermr · 2024-02-13T22:36:22Z

yep LGTM on adding to the compiler stubs with exports list. Surely we'll find other issues later on, but I think we're good to go for now and not doing anything obviously dumb for the moment!

h-vetinari · 2024-03-25T09:23:49Z

~~@conda-forge/core, the blockers for starting this are all close to being resolved. In case someone still has feedback here, please let me know!~~

Never mind, cannot ping across organisations...

beckermr · 2024-03-25T12:21:22Z

conda_forge_tick/migrators/cstdlib.py

+)
+pat_stdlib = re.compile(r".*\{\{\s*stdlib\([\"\']c[\"\']\)\s*\}\}.*")
+# no version other than 2.17 currently available (except 2.12 as default on linux-64)
+pat_sysroot_217 = re.compile(r"- sysroot_linux-64\s*=?=?2\.17")


Suggested change

pat_sysroot_217 = re.compile(r"- sysroot_linux-64\s*=?=?2\.17")

pat_sysroot = re.compile(r"- sysroot_linux-64\s*=?=?2\.(17|28)")

We have 2.28 in the wild right now and formally people can build with it if they do not use cdts. Maybe we should account for it now?

I had checked at the time and there were no users of 2.28. Now that I've expanded the search a bit, there's only cudnn (which seems misspecified actually because it pins both 2.17 & 2.28 on aarch).

Yep. As I meant to say totally optional.

beckermr · 2024-03-25T12:24:33Z

conda_forge_tick/migrators/cstdlib.py

+        needs_stdlib |= any(pat_compiler.search(line) for line in build_reqs)
+
+    if not needs_stdlib:
+        if any(pat_sysroot_217.search(line) for line in lines):


Suggested change

if any(pat_sysroot_217.search(line) for line in lines):

if any(pat_sysroot.search(line) for line in lines):

beckermr · 2024-03-25T12:34:48Z

conda_forge_tick/migrators/cstdlib.py

+
+    # check if someone specified a newer sysroot in recipe already,
+    # leave indicator to migrate() function that we need to write spec to CBC
+    if any(pat_sysroot_217.search(line) for line in lines):


Suggested change

if any(pat_sysroot_217.search(line) for line in lines):

if any(pat_sysroot.search(line) for line in lines):

beckermr · 2024-03-25T12:35:07Z

conda_forge_tick/migrators/cstdlib.py

+        lines = attrs["raw_meta_yaml"].splitlines()
+        already_migrated = any(pat_stdlib.search(line) for line in lines)
+        has_compiler = any(pat_compiler.search(line) for line in lines)
+        has_sysroot = any(pat_sysroot_217.search(line) for line in lines)


Suggested change

has_sysroot = any(pat_sysroot_217.search(line) for line in lines)

has_sysroot = any(pat_sysroot.search(line) for line in lines)

beckermr

This one is ready to go. I made one set of comments but they are probably overkill.

h-vetinari · 2024-03-25T21:47:12Z

Thanks a lot for the collaboration and help here! 🙏

h-vetinari · 2024-03-26T03:49:31Z

First PRs are being opened, so far looking like it's working as intended 🤞

h-vetinari requested review from xhochy, isuruf and beckermr February 7, 2024 10:41

h-vetinari force-pushed the stdlib branch from a0c0d6e to 2988109 Compare February 7, 2024 10:46

beckermr requested changes Feb 7, 2024

View reviewed changes

conda_forge_tick/auto_tick.py Show resolved Hide resolved

conda_forge_tick/migrators/cstdlib.py Outdated Show resolved Hide resolved

tests/test_yaml/stdlib_arrow_after_meta.yaml Show resolved Hide resolved

h-vetinari added 6 commits February 7, 2024 17:39

ENH: piggyback migrator for adding {{ stdlib("c") }}

b605588

add test setup for testing stdlib-migration

edb1793

ensure we can parse recipes containing stdlib-jinja

4f2c04c

add test for stdlib-migration: skip already migrated feedstocks

4daba02

due to limitations of the testing framework here, we still have different files because we need to go through a fake version migrator to test the piggyback.

h-vetinari force-pushed the stdlib branch from 2988109 to 4daba02 Compare February 7, 2024 16:43

isuruf reviewed Feb 7, 2024

View reviewed changes

tests/test_yaml/stdlib_arrow_after_meta.yaml Outdated Show resolved Hide resolved

expand pattern for valid selectors

d2c804e

added characters until the following search was (essentially) empty: https://github.com/search?q=org%3Aconda-forge+%2F%23+%5C%5B.*%5B%5E%5Cw%5Cs%28%29%3C%3E%21%3D.%2C%5C-%5C%27%5C%22%5D.*%5C%5D%2F+path%3Arecipe%2Fmeta.yaml&type=code

h-vetinari added 6 commits February 10, 2024 16:14

insert stdlib after compiler, i.e. in build, not host

f472dd6

prefer C compilers for indent/line/selector

8dc7307

be a bit stricter in line-matching patterns

6d8ba9b

prefer requirements.build over build.{number,...} for line_build

9086907

adjust test expectations to shifting stdlib from host: to build:

3259924

expand keys used to detect non-requirement build: section

840a09e

h-vetinari mentioned this pull request Feb 12, 2024

DISCUSS: Raise MacOS minimum target from 10.9 to 10.13? conda-forge/conda-forge.github.io#1844

Closed

9 tasks

ensure stdlib run-exports get taken care of in make_graph.py

ec6f2ab

h-vetinari mentioned this pull request Feb 13, 2024

add stdlib metapackages to pinning conda-forge/conda-forge-pinning-feedstock#5499

Merged

This was referenced Feb 16, 2024

ENH: correctly rerender CI configs with new stdlib jinja-function conda-forge/conda-smithy#1840

Closed

ENH: Improve stdlib-handling conda/conda-build#5190

Closed

This was referenced Feb 28, 2024

Fix stdlib being recognized in variant hash inputs conda/conda-build#5195

Merged

Backport patch fixing stdlib detection conda-forge/conda-build-feedstock#220

Closed

h-vetinari mentioned this pull request Mar 8, 2024

META: {{ stdlib("c") }} migration conda-forge/conda-forge.github.io#2102

Open

57 tasks

Merge branch 'master' into stdlib

65afee3

beckermr reviewed Mar 25, 2024

View reviewed changes

beckermr approved these changes Mar 25, 2024

View reviewed changes

Merge branch 'master' into stdlib

47c2ea1

h-vetinari merged commit a5ce6f2 into regro:master Mar 25, 2024
5 checks passed

h-vetinari deleted the stdlib branch March 25, 2024 21:47

h-vetinari mentioned this pull request Mar 25, 2024

Update pin for boost conda-forge/conda-forge-pinning-feedstock#5191

Merged

2 tasks

This was referenced Mar 26, 2024

Fix corner case in stdlib piggyback #2317

Merged

Stdlib migration issues #2328

Closed

h-vetinari mentioned this pull request Apr 7, 2024

switch on stdlib-piggyback for all migrations #2372

Merged

h-vetinari mentioned this pull request Apr 20, 2024

add mini-piggyback migrator for removing pin_compatible("numpy"...) #2470

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Piggyback migrator for stdlib-migration #2135

Piggyback migrator for stdlib-migration #2135

h-vetinari commented Feb 7, 2024 •

edited

Loading

codecov bot commented Feb 7, 2024 •

edited

Loading

h-vetinari commented Feb 7, 2024

beckermr commented Feb 7, 2024

h-vetinari commented Feb 7, 2024

beckermr left a comment

h-vetinari commented Feb 7, 2024 •

edited

Loading

h-vetinari commented Feb 8, 2024

h-vetinari commented Feb 8, 2024

beckermr commented Feb 8, 2024

h-vetinari commented Feb 8, 2024

beckermr commented Feb 8, 2024

h-vetinari commented Feb 13, 2024

beckermr commented Feb 13, 2024

isuruf commented Feb 13, 2024 •

edited

Loading

h-vetinari commented Feb 13, 2024

beckermr commented Feb 13, 2024

h-vetinari commented Mar 25, 2024 •

edited

Loading

beckermr Mar 25, 2024

h-vetinari Mar 25, 2024

beckermr Mar 25, 2024

beckermr Mar 25, 2024

beckermr Mar 25, 2024

beckermr Mar 25, 2024

beckermr left a comment

h-vetinari commented Mar 25, 2024

h-vetinari commented Mar 26, 2024

	pat_sysroot_217 = re.compile(r"- sysroot_linux-64\s*=?=?2\.17")
	pat_sysroot = re.compile(r"- sysroot_linux-64\s*=?=?2\.(17\|28)")

	if any(pat_sysroot_217.search(line) for line in lines):
	if any(pat_sysroot.search(line) for line in lines):

	has_sysroot = any(pat_sysroot_217.search(line) for line in lines)
	has_sysroot = any(pat_sysroot.search(line) for line in lines)

Piggyback migrator for stdlib-migration #2135

Piggyback migrator for stdlib-migration #2135

Conversation

h-vetinari commented Feb 7, 2024 • edited Loading

codecov bot commented Feb 7, 2024 • edited Loading

Codecov Report

h-vetinari commented Feb 7, 2024

beckermr commented Feb 7, 2024

h-vetinari commented Feb 7, 2024

beckermr left a comment

Choose a reason for hiding this comment

h-vetinari commented Feb 7, 2024 • edited Loading

h-vetinari commented Feb 8, 2024

h-vetinari commented Feb 8, 2024

beckermr commented Feb 8, 2024

h-vetinari commented Feb 8, 2024

beckermr commented Feb 8, 2024

h-vetinari commented Feb 13, 2024

beckermr commented Feb 13, 2024

isuruf commented Feb 13, 2024 • edited Loading

h-vetinari commented Feb 13, 2024

beckermr commented Feb 13, 2024

h-vetinari commented Mar 25, 2024 • edited Loading

beckermr Mar 25, 2024

Choose a reason for hiding this comment

h-vetinari Mar 25, 2024

Choose a reason for hiding this comment

beckermr Mar 25, 2024

Choose a reason for hiding this comment

beckermr Mar 25, 2024

Choose a reason for hiding this comment

beckermr Mar 25, 2024

Choose a reason for hiding this comment

beckermr Mar 25, 2024

Choose a reason for hiding this comment

beckermr left a comment

Choose a reason for hiding this comment

h-vetinari commented Mar 25, 2024

h-vetinari commented Mar 26, 2024

h-vetinari commented Feb 7, 2024 •

edited

Loading

codecov bot commented Feb 7, 2024 •

edited

Loading

h-vetinari commented Feb 7, 2024 •

edited

Loading

isuruf commented Feb 13, 2024 •

edited

Loading

h-vetinari commented Mar 25, 2024 •

edited

Loading