rebuild 1.46.4 with abseil 20211102.0 #222

ngam · 2022-08-16T12:59:58Z

fixes #221

Comment:

rebuild: grpc-cpp 1.46.4 with abseil 20211102.0

motivation: mamba install grpc-cpp==1.46.4 yields:

package tensorflow-base-2.9.1-cpu_py310h048a3e3_0 requires abseil-cpp >=20211102.0,<20211102.1.0a0, but none of the providers can be installed

These pins are crazy wild... :tear

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

ngam · 2022-08-16T13:00:05Z

@conda-forge-admin, please rerender.

conda-forge-linter · 2022-08-16T13:00:06Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

github-actions · 2022-08-16T13:01:25Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you but ran into some issues. Please check the output logs of the latest rerendering GutHub actions workflow run for errors. You can also ping conda-forge/core for further assistance or try re-rendering locally.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/grpc-cpp-feedstock/actions/runs/2868115265.

ngam · 2022-08-16T13:02:09Z

@h-vetinari or should I do libabseil here instead?

github-actions · 2022-08-16T13:02:14Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you but ran into some issues. Please check the output logs of the latest rerendering GutHub actions workflow run for errors. You can also ping conda-forge/core for further assistance or try re-rendering locally.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/grpc-cpp-feedstock/actions/runs/2868120800.

github-actions · 2022-08-16T13:06:52Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/grpc-cpp-feedstock/actions/runs/2868138017.

h-vetinari · 2022-08-16T13:11:43Z

I'd like to understand first why 1.46.4 needs to be rebuilt specifically, and the 1.46.3 builds aren't usable

ngam · 2022-08-16T13:15:36Z

I'd like to understand first why 1.46.4 needs to be rebuilt specifically, and the 1.46.3 builds aren't usable

I believe the general pins are wrong somehow because they tend to pin for the patch...

So, we have conditions on jaxlib and tensorflow that are exactly identical, right?

But for some bizarre reason, now they cannot coexist, even though nothing has changed:

tensorflow-base built with hard pin on abseil-cpp 20211102.0 zipped with grpc-cpp 1.46 ends up with a condition on grpc-cpp like grpc-cpp >=1.46.3,<1.47 --- all good, right?
jaxlib build with the same exact hard pin on abseil-cpp 20211102.0 zipped with grpc-cpp 1.46 ends up with a condition on grpc-cpp like grpc-cpp >=1.46.4,<1.47

So, somehow the pin is taking a floor in the patch, rather than a minor. Hence, the inconsistency.

Now the problem is, the latter grpc-cpp 1.46.4 has not been built with abseil-cpp and for some reason that just worked because jaxlib didn't actually use abseil-cpp (for complicated reasons). I am happy resubmit the builds but force jaxlib to confront abseil-cpp and so it will be forced to downgrade grpc-cpp, but that may cause further problems (jaxlib takes 5 hours to build and it's fragile).

We could just rebuild grpc-cpp to keep it completely in sync instead.

Thoughts?

Either way, a rebuild somewhere is necessary to avoid this jam. I am happy to pursue a build in jaxlib if you object to yet another build here :)

ngam · 2022-08-16T13:21:26Z

My concern is, we really have to keep tiptoeing around for these fragile yet egomaniacal packages like tensorflow, jaxlib, pytorch, and co. Such a pain 😿

…nda-forge-pinning 2022.08.16.09.27.01

ngam · 2022-08-16T13:26:43Z

I never understood these pins and I raised some issue sin the past (I will try to find them) but I was told to relax because that was how they were supposed to work. I am so confused why the pins work by taking the current patch as minimum and keeping the next minor as maximum. I would think they should stay minor to minor, but... 🤷

h-vetinari · 2022-08-16T13:29:55Z

So in general, this situation should hopefully improve substantially, now that we have the wildcard of the abseil ABI better under control. But there's a whole bunch of transition pain, and I'm sorry about that. The silent run-dependence of grpc already plays a big role in that; that at least would be solved by my PR to mark those builds broken.

The lower bound that you get from pinning against 1.46.4 is correct. It's always >= the version used to build (imagine an ABI-relevant thing being fixed between x.y.0 and x.y.1); the interesting part is how far it goes up.

In this case, you'd have to either add a separate bound to not pick up the 20220623-only grpc 1.46.4, make the dependence on abseil explicit, or migrate to the newest abseil.

Or we merge this PR... It's not ideal to undo a migration on a branch, but in this case I guess we can "soft-undo" it in cbc & then revert for the next PR.

ngam · 2022-08-16T13:31:17Z

I am happy to submit a jaxlib rebuild if you don't want to upset the order here. It is straightforward there, it will just take 5 hours, that's all.

Let me know what you prefer.

ngam · 2022-08-16T13:32:40Z

But there's a whole bunch of transition pain, and I'm sorry about that.

PLEASE. No pain at all! This is the fault of the silent dependency. I like how this is really organized and streamlined. It's great! The fact that I could easily find the place where the problem was coming from is a testament to your good work!

h-vetinari · 2022-08-16T13:32:01Z

recipe/conda_build_config.yaml

@@ -2,3 +2,6 @@ c_compiler_version:    # [linux]
 - '11'                 # [linux]
 cxx_compiler_version:  # [linux]
 - '11'                 # [linux]
+
+libabseil:             # [unix]
+- '20211102.0'         # [unix] 


You need to add the same for libabseil_static

Ah, then that's why it didn't rerender

No, it didn't get picked up because you added pins in meta.yaml. The rerender will only pick up completely unpinned lines (well, except selectors) that have an entry in the global pinning or an ongoing migration.

I tried before that...

without libabseil rebuild 1.46.4 with abseil 20211102.0 #222 (comment)

with libabseil rebuild 1.46.4 with abseil 20211102.0 #222 (comment)

See it returned to 2022 ... 46fbfd3

I think the migration file takes precedence. Let me remove it and test... but this is changing a lot of stuff now in this branch

That's strange to me... The local cbc always should take precedence

h-vetinari · 2022-08-16T13:32:26Z

recipe/meta.yaml

@@ -34,13 +34,13 @@ outputs:
        - libprotobuf
        - ninja
        # We need all host deps also in build for cross-compiling
-        - libabseil   # [build_platform != target_platform]
+        - libabseil 20211102.0   # [build_platform != target_platform]


Remove these pins please. Should be done by cbc+rerender

Let's see if it works now after adding -static

ngam · 2022-08-16T13:37:00Z

Sorry, I accidentally closed this

…nda-forge-pinning 2022.08.16.09.27.01

h-vetinari · 2022-08-16T13:49:47Z

Well, not least since we're gonna revert this again right after, this doesn't need to win a beauty or purity contest. I suggest to skip touching the migration & the cbc, and just force the pin in the meta.yaml like you had done before. Sorry for the back and forth

h-vetinari · 2022-08-16T13:50:36Z

Just also add the pin to libabseil-static so we do this homogeneously across platforms

ngam · 2022-08-16T13:55:02Z

I will start a cleaner PR with meta.yaml only changes.

pin to 20211102.0

9e49edc

increase bld no

92e9e9a

ngam marked this pull request as ready for review August 16, 2022 13:00

ngam requested review from h-vetinari, mariusvniekerk, pitrou, wesm and xhochy as code owners August 16, 2022 13:00

try libabseil

4684995

try adding pin in meta

3d2dbd8

MNT: Re-rendered with conda-build 3.22.0, conda-smithy 3.21.1, and co…

907cfbb

…nda-forge-pinning 2022.08.16.09.27.01

ngam mentioned this pull request Aug 16, 2022

minor request #221

Closed

ngam closed this Aug 16, 2022

ngam reopened this Aug 16, 2022

h-vetinari requested changes Aug 16, 2022

View reviewed changes

add static

8f90068

ngam and others added 4 commits August 16, 2022 09:38

remove meta pin

5100526

MNT: Re-rendered with conda-build 3.22.0, conda-smithy 3.21.1, and co…

46fbfd3

…nda-forge-pinning 2022.08.16.09.27.01

Delete abseil_cpp202206230.yaml

b428479

MNT: Re-rendered with conda-build 3.22.0, conda-smithy 3.21.1, and co…

6e60e97

…nda-forge-pinning 2022.08.16.09.27.01

ngam closed this Aug 16, 2022

ngam mentioned this pull request Aug 16, 2022

rebuild grpc 1.46.4 with abseil 20211102 #223

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rebuild 1.46.4 with abseil 20211102.0 #222

rebuild 1.46.4 with abseil 20211102.0 #222

ngam commented Aug 16, 2022 •

edited

Loading

ngam commented Aug 16, 2022 •

edited

Loading

conda-forge-linter commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

ngam commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

ngam commented Aug 16, 2022 •

edited

Loading

ngam commented Aug 16, 2022

ngam commented Aug 16, 2022

h-vetinari commented Aug 16, 2022 •

edited

Loading

ngam commented Aug 16, 2022

ngam commented Aug 16, 2022

h-vetinari Aug 16, 2022

ngam Aug 16, 2022

h-vetinari Aug 16, 2022

ngam Aug 16, 2022

ngam Aug 16, 2022 •

edited

Loading

ngam Aug 16, 2022

h-vetinari Aug 16, 2022

h-vetinari Aug 16, 2022

ngam Aug 16, 2022

ngam commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

ngam commented Aug 16, 2022

rebuild 1.46.4 with abseil 20211102.0 #222

rebuild 1.46.4 with abseil 20211102.0 #222

Conversation

ngam commented Aug 16, 2022 • edited Loading

Comment:

ngam commented Aug 16, 2022 • edited Loading

conda-forge-linter commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

ngam commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

ngam commented Aug 16, 2022 • edited Loading

ngam commented Aug 16, 2022

ngam commented Aug 16, 2022

h-vetinari commented Aug 16, 2022 • edited Loading

ngam commented Aug 16, 2022

ngam commented Aug 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngam Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngam commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

h-vetinari commented Aug 16, 2022

ngam commented Aug 16, 2022

ngam commented Aug 16, 2022 •

edited

Loading

ngam commented Aug 16, 2022 •

edited

Loading

ngam commented Aug 16, 2022 •

edited

Loading

h-vetinari commented Aug 16, 2022 •

edited

Loading

ngam Aug 16, 2022 •

edited

Loading