Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLD Fixes osx build by downgrading to 11.X [cd build] #21227

Merged
merged 1 commit into from
Oct 7, 2021

Conversation

thomasjpfan
Copy link
Member

@thomasjpfan thomasjpfan commented Oct 3, 2021

Reference Issues/PRs

Fixes #21182

What does this implement/fix? Explain your changes.

There are issues with libomp 12 on OSX. This PR downgrades the libomp version to 11.0.1.

  1. I tested the wheel built by github actions on my osx python 3.9 and this fix works.

  2. When testing this locally, the wheels work when numpy is installed through pip and fails when numpy is installed through conda. The biggest different I see is that with conda, llvm-openmp==12.0.1 is installed.

  3. The CI does not catch this error because everything is installed through pip.

XREF: microsoft/LightGBM#4229

@thomasjpfan
Copy link
Member Author

thomasjpfan commented Oct 3, 2021

Some experiments with different combinations:

  1. conda-forge numpy + llvm-openmp==11.1.0 + scikit-learn on pypi works.
  2. conda-forge numpy + llvm-openmp==12.0.1 + scikit-learn on conda-forge works.
  3. conda-forge numpy + llvm-openmp==12.0.1 + scikit-learn on pypi segfaults
  4. pypi numpy + scikit-learn on pypi works
  5. pypi lightgbm which links to homebrew's installed libomp 13.0.0 or 12.0.1 + scikit-learn on pypi segfaults.

In all cases the wheels built with this PR using libomp 11.0.1 works.

Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the investigation. Looks good to me as a stopgap solution.

For the longer term we might have to standardize the wheel packages of the scipy stack to all use the same version of the OpenMP runtimes for each platform...

That might involve some coordination effort via a dedicated SPEC document. This could also help with OpenBLAS / OpenMP interactions that we struggle to understand (e.g. #20642).

@ogrisel
Copy link
Member

ogrisel commented Oct 7, 2021

But reading the discussion in #21182, maybe this is just a bug in the new 12 and 13 versions of libomp, see: https://bugs.llvm.org/show_bug.cgi?id=50579#c1

@thomasjpfan next time you conduct such investigations, it might be interesting to log the output of python -m threadpoolctl -i sklearn or python -m threadpoolctl -i sklearn -i lightgbm in each env to better understand the state of each environment.

Copy link
Member

@jeremiedbb jeremiedbb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do that until libomp 12 gets fixed

@jeremiedbb jeremiedbb merged commit 686ab3e into scikit-learn:main Oct 7, 2021
@ogrisel ogrisel added the To backport PR merged in master that need a backport to a release branch defined based on the milestone. label Oct 7, 2021
jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Oct 8, 2021
@glemaitre glemaitre mentioned this pull request Oct 23, 2021
10 tasks
glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Oct 23, 2021
samronsin pushed a commit to samronsin/scikit-learn that referenced this pull request Nov 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocker Regression To backport PR merged in master that need a backport to a release branch defined based on the milestone.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG fresh install on OSX conda env with pip gives a segfault
3 participants