Windows 10 x64 ImportError: DLL load failed - location issue or binary compatibility issue?

[avikde]

System info:

• using Python 3.7.2 official installation
• have VS2019 build tools (cl.exe prints Microsoft (R) C/C++ Optimizing Compiler Version 19.21.27702.2 for x86)

Steps taken:

• used the latest master branch of cyipopt (0.2.0-dev0), and downloaded the linked Ipopt binary versions
• python setup.py install runs fine ...Finished processing dependencies for ipopt==0.2.0.dev0
• however, import ipopt prints

>python
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipopt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\ipopt-0.2.0.dev0-py3.7-win-amd64.egg\ipopt\__init__.py", line 43, in <module>
    from cyipopt import *
ImportError: DLL load failed: The specified module could not be found.

• some DLLs are in this location

>ls C:\Users\...\AppData\Local\Programs\Python\Python37\Lib\site-packages\ipopt-0.2.0.dev0-py3.7-win-amd64.egg
EGG-INFO        IpOptFSS.dll   __pycache__                 cyipopt.py  msvcp100.dll
IpOpt-vc10.dll  Ipopt-vc8.dll  cyipopt.cp37-win_amd64.pyd  ipopt       msvcr100.dll

Debugging so far:

• I have so far looked through Cannot find cyipopt module after installation #21 as well as searched through all the issues in this repo for DLL and looked through the solutions.
• Is this due to a mismatch in the C compiler that produced the Ipopt binaries and the one that compiled cyipopt on my system? If so, is there a way to know which compiler was used for that, or any link to instructions for recompiling Ipopt?
• Potentially Build wheels for cyipopt #41 would alleviate this problem

[moorepants]

Thanks for the nice error report.

I don't think that other successful builds have been using the official Python. It seems most have been using Anaconda/Conda based Pythons.

Is this due to a mismatch in the C compiler that produced the Ipopt binaries and the one that compiled cyipopt on my system?

Could be. I recommend investigating the ipopt documentation, etc., for the answer.

[avikde]

Adding some further debugging information (not a final solution):

• the MSVC version that must be used corresponding to a particular Python version on windows is listed here https://wiki.python.org/moin/WindowsCompilers. For e.g. for 3.7, it is VS2019 (which is what I have on my system).
• however, prebuilt Ipopt was likely not built with that compiler based on what it says here https://coin-or.github.io/Ipopt/INSTALL.html, but likely with mingw (would be great if anyone who knows can confirm).

[moorepants]

We do not build cyipopt with mingw on appveyor, we use Microsoft's compiler. I think the only difference in your build would be that you are using Python.org's Python and we use Anaconda.org's.

[moorepants]

Here is the relevant call to the compiler on appveyor:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\projects\cyipopt\include\coin -IC:\Miniconda37-x64\lib\site-packages\numpy\core\include -IC:\Miniconda37-x64\include -IC:\Miniconda37-x64\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" /Tcsrc\cyipopt.c /Fobuild\temp.win-amd64-3.7\Release\src\cyipopt.obj
cyipopt.c
c:\miniconda37-x64\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
src\cyipopt.c(3887): warning C4244: '=': conversion from 'Py_ssize_t' to 'Index', possible loss of data
src\cyipopt.c(3945): warning C4244: '=': conversion from 'Py_ssize_t' to 'Index', possible loss of data
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\projects\cyipopt\lib\x64\ReleaseMKL /LIBPATH:C:\Miniconda37-x64\libs /LIBPATH:C:\Miniconda37-x64\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" Ipopt-vc8.lib IpOptFSS.lib IpOpt-vc10.lib /EXPORT:PyInit_cyipopt build\temp.win-amd64-3.7\Release\src\cyipopt.obj /OUT:build\lib.win-amd64-3.7\cyipopt.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\src\cyipopt.cp37-win_amd64.lib
cyipopt.obj : warning LNK4197: export 'PyInit_cyipopt' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.7\Release\src\cyipopt.cp37-win_amd64.lib and object build\temp.win-amd64-3.7\Release\src\cyipopt.cp37-win_amd64.exp

[apommel]

I can't test for now, but this might be due to the fact that you seem to be using x86 VS2019 build tools, whereas your Python version is x64, same as the Ipopt dll.
Also, could you confirm me that ipopt-0.2.0.dev0-py3.7-win-amd64.egg is a folder and not a file?

[avikde]

@apommel Thanks for your reply.

• The .egg file was indeed a folder (I listed the results of ls on that directory in Windows 10 x64 ImportError: DLL load failed - location issue or binary compatibility issue? #52 (comment)).
• Good thought about the x64 mismatch, but I opened an "x64 Native Tools Command Prompt" where you get

>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.21.27702.2 for x64

• ran >pip uninstall ipopt==0.2.0.dev0 (to remove the previous version) and ran cyipopt>python setup.py install which succeeded as before again. However the result was the same when trying to import ipopt

    from cyipopt import *
ImportError: DLL load failed: The specified module could not be found.

@moorepants

Thanks for your comment. Over the next couple of days I will try to (in order)

• recompile Ipopt with the same compiler and the same flags you listed (BTW where could I have found that? https://github.com/matthias-k/cyipopt/blob/master/appveyor.yml seems to download the binaries)
• try with start from scratch with conda python (hesitating a little since I depend on python quite a bit and am afraid of potential problems)

[moorepants]

The appveyor builds can be found here: https://ci.appveyor.com/project/moorepants/cyipopt

Note that we are not compiling Ipopt ourselves on appveyor, we simply download the prebuild ipopt binaries.

[apommel]

@avikde
To add a bit more to your points:

• The Ipopt DLL indeed seem to be built with mingw, but building the library with VS build tools shouldn't be a problem. On the contrary, cyipopt seems to be impossible to build with mingw from my testing.

• Your problem might indeed be resolved by using Anaconda, as some dependencies are included in it but not in standard Python, such as MKL with which we have had some compatibility problems before. These dependencies seem to be difficult to install only through pip, otherwise I think it would be great to have support for the standard Python distribution. By the way, how did you install the dependencies on your system?

All in all, I would advise you to try with Anaconda, and give your feedback about that. You can actually install it in a pretty unintrusive way by unchecking the options Add Anaconda to my PATH environment variable and Register Anaconda as my default Python in the installer. This way, your environment will be untouched and you will be able to use Anaconda only in the included Anaconda Prompt and IDE Spyder.
If you think that Anaconda is too cluttered and would take too much space on your system, you can install Miniconda instead which will only include the basic dependencies.

[avikde]

Just tried a few more steps with miniconda: recording all the steps here:

• install miniconda3 64-bit from the installer
• open "Anaconda Powershell Prompt (Miniconda3)"
• check python in this prompt is the new one

> python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

• pip install cython and pip install numpy
• cyipopt> python .\setup.py install -> completes successfully; prints at the end

Using c:\users\...\miniconda3\lib\site-packages
Finished processing dependencies for ipopt==0.2.0.dev0

• > python and

>>> import ipopt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\...\Miniconda3\lib\site-packages\ipopt-0.2.0.dev0-py3.7-win-amd64.egg\ipopt\__init__.py", line 43, in <module>
    from cyipopt import *
ImportError: DLL load failed: The specified module could not be found.

• This is the same error as before, unfortunately.
• checking the install directory

> ls


    Directory: C:\users\...\miniconda3\lib\site-packages\ipopt-0.2.0.dev0-py3.7-win-amd64.egg


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        1/11/2020   5:44 PM                EGG-INFO
d-----        1/11/2020   5:44 PM                ipopt
d-----        1/11/2020   5:44 PM                __pycache__
-a----        1/11/2020   5:44 PM         184320 cyipopt.cp37-win_amd64.pyd
-a----        1/11/2020   5:44 PM            306 cyipopt.py
-a----        1/11/2020   5:44 PM        2078720 IpOpt-vc10.dll
-a----        1/11/2020   5:44 PM        2774528 Ipopt-vc8.dll
-a----        1/11/2020   5:44 PM        8349696 IpOptFSS.dll
-a----        1/11/2020   5:44 PM         608080 msvcp100.dll
-a----        1/11/2020   5:44 PM         829264 msvcr100.dll

I am not sure which C compiler the people who had success were using, but I believe perhaps that is the problem. I looked into compiling ipopt but it seems quite involved... https://coin-or.github.io/Ipopt/INSTALL.html.

BTW, this isn't directly relevant, but I also use Julia and IPOPT.jl installs and works without any hassle. I wonder if they compile Ipopt from source (probably using clang, if so).

EDIT: To respond to your question about my packages on official python, normally I install my python packages from wheels here https://www.lfd.uci.edu/~gohlke/pythonlibs/ - numpy comes with mkl for instance, and all the installs are super fast.

[moorepants]

If you are using conda, you should install the dependencies with conda, not pip.

[moorepants]

These are the relevant conda commands we are using to build cyipopt on windows on appveyor:

conda.exe clean --yes --all
conda.exe update --yes conda
conda.exe update --yes --all
conda.exe config --prepend channels conda-forge
conda.exe install --yes numpy cython future six setuptools sphinx numpydoc mkl
conda.exe update -y --all
conda.exe info
conda.exe list

Note that we are installing binaries from conda-forge as the primary channel and we install mkl.

[moorepants]

Also, note that you are in uncharted territory, as we do not have a working Windows build for the official Python distribution. If you want to build compatible binaries with Chris Gohlke's binaries, you need to match his build process. We only have a conda based build working on Windows here because it is non-trivial to build IPOPT on Windows and non-trivial to build cyipopt. Your Julia installs work because someone has taken the time and effort to build compatible binaries of IPOPT and Julia for Windows.

We are very close to having conda install cyipopt working on Windows now that we have a conda-forge Windows IPOPT build here: conda-forge/ipopt-feedstock#34.

[avikde]

Thanks a lot, Jason. After following the steps you listed in #52 (comment), python setup.py install worked, and so did import ipopt. I think it would be great to list those steps in the README (unless they are there and I missed them).

[moorepants]

Great!

Maybe you'd like to submit a pull request to add instructions for installing on Windows? Theoretically only these two commands would be needed after downloading and extracting the IPOPT dlls:

conda.exe install -c conda-forge numpy cython future six setuptools mkl
python setup.py install

[moorepants]

The instructions are here:

https://github.com/matthias-k/cyipopt#from-source-on-windows

But maybe need a little improvement.