Cannot install ctypes wrapper

[mzechmeister]

Environment

• pip version: pip3 19.1.1
• Python version: 3.6
• OS: x86_64-linux

Description
It is not possible for me to distribute a ctypes wrapper with pip3.

Expected behavior
As for python setup.py install --user, which works.

How to Reproduce

svn export https://github.com/mzechmeister/ke/trunk/cordic_dbl/python ke_cordic_dbl
cd ke_cordic_dbl
pip3 install . --user

The installation process gives no error. But then calling
import ke_dbl_i
raises:
OSError: ~/.local/lib/python3.6/site-packages/lib/ke_dbl_i.so: cannot open shared object file: No such file or directory

The problem is that pip3 appends the platform tag to the generated shared library:
~/.local/lib/python3.6/site-packages/lib/ke_dbl_i.cpython-36m-x86_64-linux-gnu.so.
But the shared library is basically loaded with ctypes.CDLL('lib/ke_dbl_i.so').
pip2 does not append a platform tag and runs without error.
python3 setup.py tags a parent folder ~/.local/lib/python3.6/site-packages/ke_dbl-0.0.0-py3.6-linux-x86_64.egg with the name argument of setup. Such a parent yields, in my opinion, a cleaner structure.

[mzechmeister]

Maybe related #5453 or https://stackoverflow.com/questions/24146840/how-do-i-package-for-distribution-a-python-module-that-uses-a-shared-library

[pradyunsg]

Hi! Thanks for filing this but this isn't the right place for this issue. pip doesn't handle system libraries or .so files directly, and only moves them around (or runs a build system, AKA setup.py) for them.

I think that python-list mailing lists, #python on Freenode (IRC) or StackOverflow would be better channels to get support for the issue you're facing.

[mzechmeister]

pip doesn't handle system libraries or .so files directly, and only moves them around (or runs a build system, AKA setup.py) for them.

Thanks for the comment.

I would not say this is the wrong project (the linked project is just a minimal example).
ctypes is builtin python functionality and there are not guidelines how to distribute with pip. The .so files are compiled during building by pip! IMO that is among the job of pip.

So is the general answer that pip cannot distribute projects based on ctypes wrapper?

[pfmoore]

It sounds like the problem is with the way you build the wheel - pip will simply install a wheel by unpacking it as defined in the wheel spec. It is the build backend's responsibility to lay out the wheel in a way that can be installed using that process. In this case, I assume you are using setuptools, so it's probably going to need some custom code in your setup.py to handle this (unless you can get the setuptools project to consider your use case as sufficiently important to justify built in support in setuptools).

[mzechmeister]

I'm have little experience with creating a setup.py and have trouble to understand what you mean with wheel, build backend, etc.
My setup.py is working with pip2.

I would appreciate a minimal working example showing how to distribute a hello world ctypes wrapper with pip3.
I think this is a common case. But I can find any doc/help on this.

[pfmoore]

Looking at the code, it appears that setup.py is building a C extension, but then the Python code is loading it as a raw DLL here. That line assumes a certain naming convention for C extensions, which is not necessarily true (the naming convention is an internal detail, so maybe it changed in some Python version), So this is definitely an issue with how the library itself is packaged/written, and is something you'll need to fix yourself.

Note - I don't think this is common at all. Normally C extensions should be written to be importable directly, loading them via ctypes is very unusual in my experience.

[mzechmeister]

Thanks for your patience.

Importing raw DLL seems to be the normal way in ctypes.
So I don't know how to fix it. Do I need to search for the file with glob or are there any arguments for the installation?

Note - There pros and cons for ctypes and C extension (high maintenance cost) (see https://scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html). It seems like C-extensions should be written only by truly experts.

[pfmoore]

If you have a normal C shared library, you shouldn't be building it with the setuptools Extension class (which is for C extensions). Of course the problem is that setuptools doesn't have support for building normal C shared libraries. So you trade off one complexity for another.

I don't have an answer for you, I'm afraid. This isn't a pip issue (or even a packaging issue, to be honest!) - it's basically a question of how to integrate C code and Python, and you probably need to ask somewhere else, where people with experience in this type of problem will be able to help. Judging from your link to a SciPy lectures site, and the fact that you seem to be integrating with numpy in your code, maybe some form of scientific Python users forum would be your best bet.

[mzechmeister]

Thanks for clarification.