Skip to content

Commit

Permalink
DOC Warn if coexisting libomp / libiomp on MacOS (#142)
Browse files Browse the repository at this point in the history
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
  • Loading branch information
jeremiedbb and ogrisel authored Jul 11, 2023
1 parent 7b79693 commit 57a59e4
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 14 deletions.
5 changes: 5 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@

- Dropped support for Python 3.6 and 3.7.

- A warning is raised on macOS when threadpoolctl finds both Intel OpenMP and LLVM
OpenMP runtimes loaded simultaneously by the same Python program. See details and
workarounds at https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md.
https://github.com/joblib/threadpoolctl/pull/142

3.1.0 (2022-01-31)
==================

Expand Down
22 changes: 12 additions & 10 deletions multiple_openmp.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,35 +29,37 @@ program**. For instance, on Linux, we never observed any issue between
`libgomp` and `libiomp`, which is the most common mix (NumPy with MKL + a
package compiled with GCC, the most widely used C compiler on that platform).

## Incompatibility between Intel OpenMP and LLVM OpenMP under Linux
## Incompatibility between Intel OpenMP and LLVM OpenMP

The only unrecoverable incompatibility we encountered happens when loading a
mix of compiled extensions linked with **`libomp` (LLVM/Clang) and `libiomp`
(ICC), on Linux**, manifested by crashes or deadlocks. It can happen even with
the simplest OpenMP calls like getting the maximum number of threads that will
be used in a subsequent parallel region. A possible explanation is that
(ICC), on Linux and macOS**, manifested by crashes or deadlocks. It can happen
even with the simplest OpenMP calls like getting the maximum number of threads
that will be used in a subsequent parallel region. A possible explanation is that
`libomp` is actually a fork of `libiomp` causing name colliding for instance.
Using `threadpoolctl` may crash your program in such a setting.

**Fortunately this problem is very rare**: at the time of writing, all major
binary distributions of Python packages for Linux use either GCC or ICC to
build the Python scientific packages. Therefore this problem would only happen
if some packagers decide to start shipping Python packages built with
LLVM/Clang instead of GCC.

Surprisingly, we never encountered this kind of issue on macOS, where this mix
is the most frequent (Clang being the default C compiler on macOS).
LLVM/Clang instead of GCC (this is the case for instance with conda's default channel).

## Workarounds for Intel OpenMP and LLVM OpenMP case

As far as we know, the only workaround consists in making sure only of one of
the two incompatible OpenMP libraries is loaded. For example:

- Tell MKL (used by NumPy) to use the GNU OpenMP runtime instead of the Intel
OpenMP runtime by setting the following environment variable:
- Tell MKL (used by NumPy) to use another threading implementation instead of the Intel
OpenMP runtime. It can be the GNU OpenMP runtime on Linux or TBB on Linux and MacOS
for instance. This is done by setting the following environment variable:

export MKL_THREADING_LAYER=GNU

or, if TBB is installed:

export MKL_THREADING_LAYER=TBB

- Install a build of NumPy and SciPy linked against OpenBLAS instead of MKL.
This can be done for instance by installing NumPy and SciPy from PyPI:

Expand Down
4 changes: 0 additions & 4 deletions threadpoolctl.py
Original file line number Diff line number Diff line change
Expand Up @@ -742,10 +742,6 @@ def _check_prefix(self, library_basename, filename_prefixes):

def _warn_if_incompatible_openmp(self):
"""Raise a warning if llvm-OpenMP and intel-OpenMP are both loaded"""
if sys.platform != "linux":
# Only raise the warning on linux
return

prefixes = [lib_controller.prefix for lib_controller in self.lib_controllers]
msg = textwrap.dedent(
"""
Expand Down

0 comments on commit 57a59e4

Please sign in to comment.