TBB linking issue on Linux when installed with pip

[bdice]

Describe the bug
Reported by two users on the forum: https://groups.google.com/u/1/g/freud-users/c/G4NPojpOv9U

To Reproduce
Steps to reproduce the behavior:

1. docker run -it ubuntu:20.04
2. apt update
3. apt install python3 python3-pip
4. pip3 install freud-analysis
5. python3 -c "import freud"

Error output

root@f8d7353570b3:~# python3 -c "import freud"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/freud/__init__.py", line 4, in <module>
    from . import box
ImportError: libtbb.so.2: cannot open shared object file: No such file or directory

Here's the ldd output:

root@f8d7353570b3:/usr/local/lib/python3.8/dist-packages/freud# ldd libfreud.so
        linux-vdso.so.1 (0x00007ffc76ff5000)
        libtbb.so.2 => not found
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe44d50f000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe44d3c0000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe44d3a5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe44d1b3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe44de3c000)

It appears the bundled TBB library is installed to freud_analysis.libs but cannot be found:

root@f8d7353570b3:/usr/local/lib/python3.8/dist-packages# tree freud*
freud
|-- __init__.py
|-- __pycache__
|   |-- __init__.cpython-38.pyc
|   |-- data.cpython-38.pyc
|   |-- errors.cpython-38.pyc
|   `-- plot.cpython-38.pyc
|-- box.cpython-38-x86_64-linux-gnu.so
|-- cluster.cpython-38-x86_64-linux-gnu.so
|-- data.py
|-- density.cpython-38-x86_64-linux-gnu.so
|-- diffraction.cpython-38-x86_64-linux-gnu.so
|-- environment.cpython-38-x86_64-linux-gnu.so
|-- errors.py
|-- interface.cpython-38-x86_64-linux-gnu.so
|-- libfreud.so
|-- locality.cpython-38-x86_64-linux-gnu.so
|-- msd.cpython-38-x86_64-linux-gnu.so
|-- order.cpython-38-x86_64-linux-gnu.so
|-- parallel.cpython-38-x86_64-linux-gnu.so
|-- plot.py
|-- pmft.cpython-38-x86_64-linux-gnu.so
`-- util.cpython-38-x86_64-linux-gnu.so
freud_analysis-2.4.1.dist-info
|-- INSTALLER
|-- LICENSE
|-- METADATA
|-- RECORD
|-- WHEEL
`-- top_level.txt
freud_analysis.libs
`-- libtbb-d3218813.so.2 

System configuration (please complete the following information):

• OS: Ubuntu 20.04
• Version of Python: 3.8
• Version of freud: 2.4.1

Additional context
freud 2.3.0 uses the previous build system and is a viable workaround for the short term. I also think that the freud package from conda-forge should work fine.

[bdice]

I see that the RPATH is correct for the Python modules but not libfreud.so.

The RPATH appears in the Python modules...

root@f8d7353570b3:/usr/local/lib/python3.8/dist-packages/freud# readelf -d cluster.cpython-38-x86_64-linux-gnu.so
Dynamic section at offset 0x86000 contains 30 entries:
Tag        Type                         Name/Value
0x000000000000000f (RPATH)              Library rpath: [$ORIGIN:$ORIGIN/../freud_analysis.libs]
0x0000000000000001 (NEEDED)             Shared library: [libfreud.so]
0x0000000000000001 (NEEDED)             Shared library: [libtbb-d3218813.so.2]
0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
0x000000000000000c (INIT)               0x81f8
0x000000000000000d (FINI)               0x41c84
0x0000000000000019 (INIT_ARRAY)         0x253928
0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
0x000000000000001a (FINI_ARRAY)         0x253930
0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
0x000000006ffffef5 (GNU_HASH)           0x25c380
0x0000000000000005 (STRTAB)             0x25d230
0x0000000000000006 (SYMTAB)             0x25afe8
0x000000000000000a (STRSZ)              4111 (bytes)
0x000000000000000b (SYMENT)             24 (bytes)
0x0000000000000003 (PLTGOT)             0x254000
0x0000000000000002 (PLTRELSZ)           3528 (bytes)
0x0000000000000014 (PLTREL)             RELA
0x0000000000000017 (JMPREL)             0x7430
0x0000000000000007 (RELA)               0x2798
0x0000000000000008 (RELASZ)             19608 (bytes)
0x0000000000000009 (RELAENT)            24 (bytes)
0x000000006ffffffe (VERNEED)            0x2728
0x000000006fffffff (VERNEEDNUM)         3
0x000000006ffffff0 (VERSYM)             0x2584
0x000000006ffffff9 (RELACOUNT)          740
0x0000000000000000 (NULL)               0x0

... while RPATH is missing from libfreud.so:

root@f8d7353570b3:/usr/local/lib/python3.8/dist-packages/freud# readelf -d libfreud.so
Dynamic section at offset 0x52e820 contains 29 entries:
Tag        Type                         Name/Value
0x0000000000000001 (NEEDED)             Shared library: [libtbb.so.2]
0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
0x000000000000000e (SONAME)             Library soname: [libfreud.so]
0x000000000000000c (INIT)               0x2b9a70
0x000000000000000d (FINI)               0x453f2c
0x0000000000000019 (INIT_ARRAY)         0x72b788
0x000000000000001b (INIT_ARRAYSZ)       64 (bytes)
0x000000000000001a (FINI_ARRAY)         0x72b7c8
0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
0x000000006ffffef5 (GNU_HASH)           0x1f0
0x0000000000000005 (STRTAB)             0x7c518
0x0000000000000006 (SYMTAB)             0x1c068
0x000000000000000a (STRSZ)              1994067 (bytes)
0x000000000000000b (SYMENT)             24 (bytes)
0x0000000000000003 (PLTGOT)             0x72f000
0x0000000000000002 (PLTRELSZ)           284520 (bytes)
0x0000000000000014 (PLTREL)             RELA
0x0000000000000017 (JMPREL)             0x274308
0x0000000000000007 (RELA)               0x26b380
0x0000000000000008 (RELASZ)             36744 (bytes)
0x0000000000000009 (RELAENT)            24 (bytes)
0x000000006ffffffe (VERNEED)            0x26b2d0
0x000000006fffffff (VERNEEDNUM)         4
0x000000006ffffff0 (VERSYM)             0x26326c
0x000000006ffffff9 (RELACOUNT)          292
0x0000000000000000 (NULL)               0x0

[bdice]

Here are a few workarounds for now:

1. Install from the source distribution on PyPI. This requires a system installation of TBB to build the package from source.

sudo apt install libtbb2 libtbb-dev
pip3 install freud-analysis --no-binary freud-analysis

2. Install from conda-forge.

conda install -c conda-forge freud

3. Install an older binary that uses the correct RPATHs.

pip3 install freud-analysis==2.3.0

[bdice]

This issue is caused by an upstream issue in auditwheel: pypa/auditwheel#136

The core problem is that libfreud.so is categorized as a "non-Python" library but it is a "needed lib", and is thus not patched by auditwheel when locating and repairing libraries' RPATHs. My guess is that the original code author expected that all "needed libs" were Python extensions?

This pull request to auditwheel would solve the problem but has stalled for some time: pypa/auditwheel#139

[bdice]

I created a new pull request to auditwheel that will solve this problem: pypa/auditwheel#283

[bdice]

Resolved in freud v2.5.0 (#734). The final PR to auditwheel is here, for future reference: pypa/auditwheel#298