cython project cannot open shared object file for third party .so

[levyfan]

Hi, I have a third party .so named libcuBERT.so and write cython project to wrap it. I successfully build and auditwheel it as:

% unzip -l wheelhouse/cuBERT-0.0.1-cp27-cp27mu-manylinux1_x86_64.whl
Archive:  wheelhouse/cuBERT-0.0.1-cp27-cp27mu-manylinux1_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
   302160  2019-05-14 14:06   cuBERT.so
  5329968  2019-05-14 14:06   .libscuBERT/libcuBERT-d011984c.so
 61277072  2019-05-14 14:06   .libscuBERT/libcublas-b9871408.so.9.0.480
    95040  2019-05-14 14:06   .libscuBERT/libgomp-91278daa.so.1.0.0
      319  2019-05-13 09:37   cuBERT-0.0.1.dist-info/metadata.json
       10  2019-05-13 09:37   cuBERT-0.0.1.dist-info/DESCRIPTION.rst
      846  2019-05-14 14:06   cuBERT-0.0.1.dist-info/RECORD
      110  2019-05-14 14:06   cuBERT-0.0.1.dist-info/WHEEL
      202  2019-05-13 09:37   cuBERT-0.0.1.dist-info/METADATA
        7  2019-05-13 09:37   cuBERT-0.0.1.dist-info/top_level.txt
---------                     -------
 67005734                     10 files

But after I install the wheel as pip install, and import it from python, the libcuBERT.so still can not be found:

Traceback (most recent call last):
  File "cuBERT_test.py", line 3, in <module>
    import cuBERT
ImportError: libcuBERT.so: cannot open shared object file: No such file or directory

I have checked the audited cuBERT.so, and it seems nothing wrong:

readelf --dynamic wheelhouse/cuBERT.so 

Dynamic section at offset 0x491c0 contains 31 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/.libscuBERT]
 0x0000000000000001 (NEEDED)             Shared library: [libcuBERT-d011984c.so]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x3070
 0x000000000000000d (FINI)               0x11124
 0x0000000000000019 (INIT_ARRAY)         0x214000
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x214008
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x1b8
 0x0000000000000005 (STRTAB)             0x2173f0
 0x0000000000000006 (SYMTAB)             0x1f8
 0x000000000000000a (STRSZ)              2138 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x214350
 0x0000000000000002 (PLTRELSZ)           1920 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x28f0
 0x0000000000000007 (RELA)               0x1678
 0x0000000000000008 (RELASZ)             4728 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x1658
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x1568
 0x000000006ffffff9 (RELACOUNT)          162
 0x0000000000000000 (NULL)               0x0

The cython setup.py looks like:

import setuptools
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

import numpy as np
import platform

setup(
    name='cuBERT',
    ext_modules = cythonize([Extension("cuBERT", 
                                       sources=["cuBERT.pyx"], 
                                       libraries=["cuBERT"],
                                       library_dirs=["path/to/libcuBERT.so"],
                                       include_dirs=[np.get_include()],
                                       language="c++")])
)

And the original wheel package before auditwheel is like:

unzip -l dist/cuBERT-0.0.1-cp27-cp27mu-linux_x86_64.whl 
Archive:  dist/cuBERT-0.0.1-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
   291608  2019-05-13 09:37   cuBERT.so
        7  2019-05-14 06:36   cuBERT-0.0.1.dist-info/top_level.txt
      105  2019-05-14 06:36   cuBERT-0.0.1.dist-info/WHEEL
      202  2019-05-14 06:36   cuBERT-0.0.1.dist-info/METADATA
      366  2019-05-14 06:36   cuBERT-0.0.1.dist-info/RECORD
---------                     -------
   292288                     5 files

[levyfan]

The libcuBERT.so is placed under a customized directory. I run auditwheel by LD_LIBRARY_PATH=/path/to/libcuBERT.so auditwheel repair dist/cuBERT-xxx.whl. And the cuBERT.so before audit is like:

readelf --dynamic cuBERT.so 

Dynamic section at offset 0x14018 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libcuBERT.so]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
...

[njsmith]

Can you post the output of LD_DEBUG=all python -c 'import cuBERT'?

[levyfan]

Sure, I attach the output here.
import.txt

[njsmith]

So it says that it's loading ./cuBERT.so, and that's what's linking to libcuBERT.so:

     26422:	file=libcuBERT.so [0];  needed by ./cuBERT.so [0]

Is it possible that you're running python in the same directory as your original un-repaired cuBERT.so, and that python is picking that up first instead of the repaired version?

[levyfan]

Oh yes, that is the reason!

But after I remove the original cuBERT.so and run import again, it says

Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!

How to fix that?

[levyfan]

This time, the output for LD_DEBUG=all python -c 'import cuBERT' is
import.txt

[levyfan]

I find out that I use patchelf 0.9 which has a bug in NixOS/patchelf#84. After upgraded to 0.10, it is OK now.

Thanks for your help!