Segmentation Fault: 11 - OSX high sierra - please Help

[Dinuz]

Hello guys, please help me in understanding what is going on and why. Are 3 days that I am trying desperately to let this work without success.
Please help me!
Thank you so very much.

Issue summary

After complete successfully the caffe installation (No difference using make or cmake), make all, make test and make runtest without any error(all test passed); I launched python in a shell, and called import caffe and I got segmentation fault 11.

I run all in an anaconda environment with python 3.6.3 and all the requirements installed by conda in the environment.

I use brew to install proto boost and boost-python (—with-python3 —without-python -built-from-source) and cv2 and all the other dependence (as in the installation notes).

The installation is only_cpu . The python path is set as the anaconda envs path (cmakecahe).

Everything works fine except the import caffe in python, that gives segmentation fault 11.

Steps to reproduce

Finish the make runtest, run make pycaffe, export caffe/python path, run python shell, run import caffe

If you are having difficulty building Caffe or training a model, please ask the caffe-users mailing list. If you are reporting a build error that seems to be due to a bug in Caffe, please attach your build configuration (either Makefile.config or CMakeCache.txt) and the output of the make (or cmake) command.

Your system configuration

Operating system: osx high Sierra 10.13
Compiler:
CUDA version (if applicable):
CUDNN version (if applicable):
BLAS:
Python or MATLAB version (for pycaffe and matcaffe respectively): python 3.6.3 coming with anaconda

[shuguang101]

I met this problem too, I trying several method to fix the problem, but failed. Who can help me, thank you so very much. The lldb debug information are as flows: (boost version: 1.65.1)

(lldb) target create "python3"
Current executable set to 'python3' (x86_64).
(lldb) r
Process 5016 launched: '/Users/mq/anaconda3/bin/python3' (x86_64)
Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 6 2017, 12:04:38)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import caffe
Process 5016 stopped

• thread dedicated build/ directory #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x25)
  frame #0: 0x00000001058b6a8b libpython3.6m.dylibPyUnicode_InternInPlace + 107 libpython3.6m.dylibPyUnicode_InternInPlace:
  -> 0x1058b6a8b <+107>: movb 0x25(%rax), %r15b
  0x1058b6a8f <+111>: movq 0x1d0dda(%rip), %rax ; _PyThreadState_Current
  0x1058b6a96 <+118>: movb $0x1, 0x25(%rax)
  0x1058b6a9a <+122>: movq %rbx, %rsi
  Target 0: (python3) stopped.
  (lldb) list
  (lldb) bt
• thread dedicated build/ directory #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x25)
  • frame #0: 0x00000001058b6a8b libpython3.6m.dylibPyUnicode_InternInPlace + 107 frame #1: 0x0000000105937c8f libpython3.6m.dylibPyImport_Import + 351
    frame selective search window proposals and better batching of images #2: 0x000000010593909c libpython3.6m.dylibPyImport_ImportModule + 28 frame #3: 0x0000000105371cc9 _caffe.socaffe::init_module__caffe() + 15865
    frame How install in Mac #4: 0x00000001098fd8a8 libboost_python3.dylibboost::python::handle_exception_impl(boost::function0<void>) [inlined] boost::function0<void>::operator()() const at function_template.hpp:759 [opt] frame #5: 0x00000001098fd892 libboost_python3.dylibboost::python::handle_exception_impl(f=) at errors.cpp:25 [opt]
    frame add flatten layer #6: 0x00000001098fe82d libboost_python3.dylibboost::python::detail::init_module(PyModuleDef&, void (*)()) [inlined] bool boost::python::handle_exception<void (*)()>(f=(_caffe.socaffe::init_module__caffe()))()) at errors.hpp:29 [opt]
    frame modify makefile to recompile on changes to header files #7: 0x00000001098fe801 libboost_python3.dylibboost::python::detail::init_module(PyModuleDef&, void (*)()) [inlined] boost::python::detail::(anonymous namespace)::init_module_in_scope(m=0x0000000101e666d8)()) at module.cpp:24 [opt] frame #8: 0x00000001098fe7e1 libboost_python3.dylibboost::python::detail::init_module(moduledef=, init_function=(_caffe.socaffe::init_module__caffe()))()) at module.cpp:43 [opt] frame #9: 0x000000010018fe1b python3_PyImport_LoadDynamicModuleWithSpec + 491
    frame Efficient processing of images, with option to use selective search window proposals #10: 0x000000010018f48c python3_imp_create_dynamic + 188 frame #11: 0x000000010008b5a2 python3PyCFunction_Call + 66
    frame Minimum CUDA arch == compute capability 2.0? #12: 0x0000000100161adf python3_PyEval_EvalFrameDefault + 48639 frame #13: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427
    frame which file is the entry file in src folder? #14: 0x0000000100163c4c python3fast_function + 348 frame #15: 0x000000010016383e python3call_function + 222
    frame Remove Intel MKL dependency #16: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #17: 0x0000000100163baf python3fast_function + 191
    frame change imagenet_val batch size from 200 to intended 50 #18: 0x000000010016383e python3call_function + 222 frame #19: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame Extract the middle features #20: 0x0000000100163baf python3fast_function + 191 frame #21: 0x000000010016383e python3call_function + 222
    frame Where is the "caffe/proto/caffe.pb.h" file? #22: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #23: 0x0000000100163baf python3fast_function + 191
    frame port Caffe to boost + eigen to replace MKL #24: 0x000000010016383e python3call_function + 222 frame #25: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame Testing Fail #26: 0x0000000100163baf python3fast_function + 191 frame #27: 0x000000010016383e python3call_function + 222
    frame Replace MKL with Boost+Eigen3 #28: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #29: 0x00000001001642e2 python3_PyFunction_FastCallDict + 178
    frame Implement adaptive learning rate #30: 0x00000001000096d0 python3_PyObject_FastCallDict + 320 frame #31: 0x0000000100012c2a python3_PyObject_CallMethodIdObjArgs + 490
    frame Implement the model that won the classification task of ImageNet 2013 #32: 0x0000000100189d24 python3PyImport_ImportModuleLevelObject + 1092 frame #33: 0x000000010015f3b4 python3_PyEval_EvalFrameDefault + 38612
    frame setting default power_wrapper batch size to match the imagenet_deploy pr... #34: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427 frame #35: 0x000000010014f329 python3builtin_exec + 345
    frame Write padding aware im2col? #36: 0x000000010008b5a2 python3PyCFunction_Call + 66 frame #37: 0x0000000100161adf python3_PyEval_EvalFrameDefault + 48639
    frame LevelDB core dump while snapshotting (out of files) #38: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427 frame #39: 0x0000000100163c4c python3fast_function + 348
    frame Crash on python close with CPU mode #40: 0x000000010016383e python3call_function + 222 frame #41: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame Building on OSX Mavericks #42: 0x0000000100163baf python3fast_function + 191 frame #43: 0x000000010016383e python3call_function + 222
    frame error when call 'make pycaffe' #44: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #45: 0x0000000100163baf python3fast_function + 191
    frame What is an iteration in the MNIST demo? #46: 0x000000010016383e python3call_function + 222 frame #47: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame Why not using GSL to replace MKL? #48: 0x0000000100163baf python3fast_function + 191 frame #49: 0x000000010016383e python3call_function + 222
    frame make pycaffe error #50: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #51: 0x00000001001642e2 python3_PyFunction_FastCallDict + 178
    frame Add Caffe to the Software links of deeplearning.net #52: 0x00000001000096d0 python3_PyObject_FastCallDict + 320 frame #53: 0x0000000100012c2a python3_PyObject_CallMethodIdObjArgs + 490
    frame Implement Matrix class to abstract algorithms away from data storage details #54: 0x0000000100189d24 python3PyImport_ImportModuleLevelObject + 1092 frame #55: 0x000000010015f3b4 python3_PyEval_EvalFrameDefault + 38612
    frame Make detection module for python wrapper #56: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427 frame #57: 0x000000010014f329 python3builtin_exec + 345
    frame Crash after the iteration 1620. Check failed,cublasSgemm #58: 0x000000010008b5a2 python3PyCFunction_Call + 66 frame #59: 0x0000000100161adf python3_PyEval_EvalFrameDefault + 48639
    frame Sparsity penalties for unsupervised learning #60: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427 frame #61: 0x0000000100163c4c python3fast_function + 348
    frame Simple fixes for program path and CUDA setting #62: 0x000000010016383e python3call_function + 222 frame #63: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame CUDA driver 331.38 incompatibility #64: 0x0000000100163baf python3fast_function + 191 frame #65: 0x000000010016383e python3call_function + 222
    frame Changes and documentation for compiling on OS X 10.8 #66: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #67: 0x0000000100163baf python3fast_function + 191
    frame Clarify MKL compiling and linking process #68: 0x000000010016383e python3call_function + 222 frame #69: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093
    frame Fix for buffer overflow problem with long filenames when setting up the LevelDB #70: 0x0000000100163baf python3fast_function + 191 frame #71: 0x000000010016383e python3call_function + 222
    frame Rename devicequery.cpp to device_query.cpp #72: 0x00000001001614d5 python3_PyEval_EvalFrameDefault + 47093 frame #73: 0x00000001001642e2 python3_PyFunction_FastCallDict + 178
    frame make distribute pycaffe error #74: 0x00000001000096d0 python3_PyObject_FastCallDict + 320 frame #75: 0x0000000100012c2a python3_PyObject_CallMethodIdObjArgs + 490
    frame Added early stopping to the solver #76: 0x0000000100189d24 python3PyImport_ImportModuleLevelObject + 1092 frame #77: 0x000000010015f3b4 python3_PyEval_EvalFrameDefault + 38612
    frame Implement a demo website similar to Decaf Demo #78: 0x0000000100154aab python3_PyEval_EvalCodeWithName + 427 frame #79: 0x00000001001b01dc python3PyRun_InteractiveOneObject + 1132
    frame Add steps to install multi-threaded OpenBLAS on Ubuntu #80: 0x00000001001ad15e python3PyRun_InteractiveLoopFlags + 334 frame #81: 0x00000001001acfeb python3PyRun_AnyFileExFlags + 139
    frame Replace atlas with multithreaded OpenBLAS to speed-up on multi-core CPU for the boost-eigen branch #82: 0x00000001001d3378 python3Py_Main + 4632 frame #83: 0x00000001000016bd python3main + 509
    frame Make gemm fully dependent on eigen #84: 0x00007fff74c68115 libdyld.dylibstart + 1 (lldb) up frame #1: 0x0000000105937c8f libpython3.6m.dylibPyImport_Import + 351
    libpython3.6m.dylibPyImport_Import: 0x105937c8f <+351>: movq -0x38(%rbp), %rax 0x105937c93 <+355>: movq %rax, 0x16cfe6(%rip) ; PyImport_Import.import_str 0x105937c9a <+362>: testq %rax, %rax 0x105937c9d <+365>: je 0x105937d06 ; <+470> (lldb) up frame #2: 0x000000010593909c libpython3.6m.dylibPyImport_ImportModule + 28
    libpython3.6m.dylibPyImport_ImportModule: 0x10593909c <+28>: movq %rax, %r14 0x10593909f <+31>: decq (%rbx) 0x1059390a2 <+34>: jne 0x1059390b3 ; <+51> 0x1059390a4 <+36>: movq 0x8(%rbx), %rax (lldb) up frame #3: 0x0000000105371cc9 _caffe.socaffe::init_module__caffe() + 15865
    _caffe.socaffe::init_module__caffe: 0x105371cc9 <+15865>: movq %rax, %rbx 0x105371ccc <+15868>: testq %rbx, %rbx 0x105371ccf <+15871>: je 0x105371d32 ; <+15970> 0x105371cd1 <+15873>: leaq 0x659c8(%rip), %rsi ; "_ARRAY_API" (lldb) up libboost_python3.dylib was compiled with optimization - stepping may behave oddly; variables may not be available. frame #4: 0x00000001098fd8a8 libboost_python3.dylibboost::python::handle_exception_impl(boost::function0) [inlined] boost::function0::operator()() const at function_template.hpp:759 [opt]
    756 if (this->empty())
    757 boost::throw_exception(bad_function_call());
    758
    -> 759 return get_vtable()->invoker
    760 (this->functor BOOST_FUNCTION_COMMA BOOST_FUNCTION_ARGS);
    761 }
    762
    (lldb) up
    frame OS X makefile changes #5: 0x00000001098fd892 libboost_python3.dylib`boost::python::handle_exception_impl(f=) at errors.cpp:25 [opt]
    22 {
    23 if (detail::exception_handler::chain)
    24 return detail::exception_handler::chain->handle(f);
    -> 25 f();
    26 return false;
    27 }
    28 catch(const boost::python::error_already_set&)
    (lldb)

[c9s]

Same here, this is not just for OS X. It also happens in Linux Conda + Caffe

[CarlosRDomin]

Same here, anyone has found a solution?

I'm on Python 3.7 in a conda environment (Mac OS X 10.13.6)

[c9s]

You can try ipython, it has a dynamic loader that solves the failed dynamic linking

[c9s]

I remember the cause is related to the boost python library

[CarlosRDomin]

Hmmm it's weird. Everything works well on Anaconda's Python 2.7, only get Segmentation Fault: 11 on Anaconda's Python 3 (tried 3.7 and 3.6, with the exact same environment setup as the Python 2.7 that works)

[CarlosRDomin]

FYI, after a lot of tinkering, I was able to compile Caffe in Python 3 without getting the traditional segmentation fault!! 🎉
The key was to use Homebrew + Python 3's venv instead of conda. Full instructions can be found here. I know it's an old thread but hope it can be of any help! :)

[huihuili2019]

@Dinuz same with you . did you solved ?

[shuguang101]

I did’t solved the problem, now I using pytorch.

…

2020年5月8日 00:23，huihuili2019 ***@***.***> 写道： @Dinuz <https://github.com/Dinuz> same with you . did you solved ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6019 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYOHAO3YH7FJXXW3KIFNTDRQLN7FANCNFSM4EBXC5GA>.

[huihuili2019]

I probably found the cause of this problem!
It should be a problem with the python3 library under the conda environment

Under the same thirdparty conditions, change the python version :

• conda create -n python27 python = 2.7
• conda activate python27
• then recompile caffe in current environment
• and don't forget pip install -r requirement.txt in caffe_bvlc/python
• then import caffe successfully

Although the question has been raised for a long time, I still hope to help others.

This problem has troubled me for a long time, I tested the runtime environment in detail, and finally located it may be the reason