RPATH not getting set for linux wheel for release 2.6.1

[klarh]

Describe the bug

It seems that the RPATH isn't getting set for the 2.6.1 wheel on linux. I encountered this while testing the flowws-live containers, that can be used to reproduce/inspect the behavior if you like.

To Reproduce

Version 2.5.1 (working):

$ docker run --rm mspells/flowws-live:latest bash -c 'pip install -U freud-analysis==2.5.1 && readelf -d /home/user/env/lib/python3.9/site-packages/freud/libfreud.so'
Requirement already satisfied: freud-analysis==2.5.1 in /home/user/env/lib/python3.9/site-packages (2.5.1)
Collecting freud-analysis==2.5.1
  Downloading freud_analysis-2.5.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (5.6 MB)
Requirement already satisfied: scipy>=1.1 in /usr/lib/python3/dist-packages (from freud-analysis==2.5.1) (1.6.0)
Requirement already satisfied: rowan>=1.2.1 in /home/user/env/lib/python3.9/site-packages (from freud-analysis==2.5.1) (1.3.0.post1)
Requirement already satisfied: numpy>=1.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.5.1) (1.19.5)
Requirement already satisfied: cython>=0.29.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.5.1) (0.29.21)

Dynamic section at offset 0xb5c000 contains 30 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../freud_analysis.libs]
 0x0000000000000001 (NEEDED)             Shared library: [libtbb-b3f1ab55.so.12.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libfreud.so]
(other output clipped)

Version 2.6.1 (broken):

$ docker run --rm mspells/flowws-live:latest bash -c 'pip install -U freud-analysis==2.6.1 && readelf -d /home/user/env/lib/python3.9/site-packages/freud/libfreud.so'
Collecting freud-analysis==2.6.1
  Downloading freud_analysis-2.6.1-cp39-cp39-manylinux2010_x86_64.whl (5.5 MB)
Requirement already satisfied: cython>=0.29.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (0.29.21)
Requirement already satisfied: rowan>=1.2.1 in /home/user/env/lib/python3.9/site-packages (from freud-analysis==2.6.1) (1.3.0.post1)
Requirement already satisfied: numpy>=1.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (1.19.5)
Requirement already satisfied: scipy>=1.1 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (1.6.0)
Installing collected packages: freud-analysis
  Attempting uninstall: freud-analysis
    Found existing installation: freud-analysis 2.5.1
    Uninstalling freud-analysis-2.5.1:
      Successfully uninstalled freud-analysis-2.5.1
Successfully installed freud-analysis-2.6.1

Dynamic section at offset 0x5c5840 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libtbb.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libfreud.so]
 0x000000000000000f (RPATH)              Library rpath: [/usr/local/lib64]
(other output clipped)

Error output

$ docker run --rm mspells/flowws-live:latest bash -c 'pip install -U freud-analysis==2.6.1 && python -c "import freud"'                                            
Collecting freud-analysis==2.6.1
  Downloading freud_analysis-2.6.1-cp39-cp39-manylinux2010_x86_64.whl (5.5 MB)
Requirement already satisfied: scipy>=1.1 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (1.6.0)
Requirement already satisfied: numpy>=1.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (1.19.5)
Requirement already satisfied: rowan>=1.2.1 in /home/user/env/lib/python3.9/site-packages (from freud-analysis==2.6.1) (1.3.0.post1)
Requirement already satisfied: cython>=0.29.14 in /usr/lib/python3/dist-packages (from freud-analysis==2.6.1) (0.29.21)
Installing collected packages: freud-analysis
  Attempting uninstall: freud-analysis
    Found existing installation: freud-analysis 2.5.1
    Uninstalling freud-analysis-2.5.1:
      Successfully uninstalled freud-analysis-2.5.1
Successfully installed freud-analysis-2.6.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/env/lib/python3.9/site-packages/freud/__init__.py", line 4, in <module>
    from . import (
ImportError: libtbb.so.12: cannot open shared object file: No such file or directory

System configuration (please complete the following information):

• OS: debian testing container
• Version of Python: 3.9
• Version of freud: 2.6.1 binary wheel

Additional context

It's worth noting that TBB is also installed at the system level in these containers, but that gives a libtbb.so.2, not 12.

[bdice]

The freud wheels are supposed to include a copy of TBB 2021 in freud_analysis.libs, which is supposed to be shipped in the wheel. I am working on my dissertation and won't be able to investigate/solve this issue for 2-3 weeks. I would suggest checking into the following:

1. Is the TBB library being included in the wheel?
2. Is auditwheel being called by cibuildwheel? Does it show any errors? (Does the error affect macOS, which uses delocate instead of auditwheel?)
3. Are there issues in the CMake configuration that are preventing the RPATH from being set correctly?

The change in name from libtbb.so.2 to libtbb.so.12 is related to the TBB 2020 / TBB 2021 change, I believe.

[klarh]

For question 1, at least, the library is getting included (although it is libtbb-0d8fc6d2.so.12.2, not libtbb.so.12).

[bdice]

@klarh Good -- that indicates that auditwheel is packaging it. The name obfuscation is intentionally done by auditwheel to avoid library conflicts.

Please check readelf -d on the other libraries, like the built locality module (named something like locality.cp38-some-stuff.so. If that has the right RPATH and only libfreud.so has the wrong RPATH, then perhaps this is related to this issue in auditwheel (which I thought was solved): pypa/auditwheel#298

Those auditwheel changes were patched upstream into cibuildwheel before the fixes were released in the subsequent auditwheel version. Maybe we just need to update cibuildwheel or auditwheel?

[klarh]

Ah, you're right! The cython-built extensions all have the correct RPATH set up, so maybe it is just a matter of updating.

[bdice]

I did a little bit of digging to help identify the problem here. I misremembered, the auditwheel changes weren't patched upstream into cibuildwheel. What I should have said was that one user of cibuildwheel patched their own Docker image: pypa/auditwheel#283

The latest auditwheel release (4.0.0) supports the fix from pypa/auditwheel#298. However, the manylinux Docker images that cibuildwheel used for Linux builds haven't been updated to use auditwheel 4.0.0 yet. I checked the latest manylinux tag (quay.io/pypa/manylinux2010_x86_64:2021-06-07-00faba2) and it uses auditwheel 4.0.0.

The last release of freud has its CI logs here: https://github.com/glotzerlab/freud/runs/2895373537

This shows it used cibuildwheel 1.11.1. That version is pinned to this Docker image, which is too old to have auditwheel 4.0.0: https://github.com/pypa/cibuildwheel/blob/5ea360607df3d271239b364257eb636411d6a8e8/cibuildwheel/resources/pinned_docker_images.cfg#L3

The newest release of cibuildwheel (1.12.0) still isn't pinned high enough, but cibuildwheel 2.0.0 will be pinned higher once it is released.

For now I am creating a PR that will attempt to use a newer manylinux image with auditwheel 4.0.0.