Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

使用paddle模型预测时,总是提示进程被KILL掉, #25315

Closed
Hi-ylf opened this issue Jul 1, 2020 · 3 comments
Closed

使用paddle模型预测时,总是提示进程被KILL掉, #25315

Hi-ylf opened this issue Jul 1, 2020 · 3 comments

Comments

@Hi-ylf
Copy link

Hi-ylf commented Jul 1, 2020

  • 版本、环境信息:
      PaddlePaddle版本:1.8.1,python3.6环境,linux
    执行paddle的时候,遇到一个特别奇怪的错误,只运行执行函数A,结果正常,如果在A后再运行B的话,就会报错了。A和B单独执行都正常,如果 我用在调试模式下进行,这整个流程都没问题。所以问题到底是什么呢???

    2020-07-01 19:16:49,783-WARNING: variable file [ ./script/model_files/simnet_bow_pairwise_pretrained_model/model ./script/model_files/simnet_bow_pairwise_pretrained_model/info.yml ] not used
    OMP: Error fixed build issue of double definition of atomicAdd on modern GPUs #15: Initializing libomp.so, but found libiomp5.so already initialized.
    OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
    W0701 19:16:50.744439 15477 init.cc:216] Warning: PaddlePaddle catches a failure signal, it may not work properly
    W0701 19:16:50.744503 15477 init.cc:218] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
    W0701 19:16:50.744510 15477 init.cc:221] The detail failure signal is:

W0701 19:16:50.744521 15477 init.cc:224] *** Aborted at 1593602210 (unix time) try "date -d @1593602210" if you are using GNU date ***
W0701 19:16:50.746780 15477 init.cc:224] PC: @ 0x0 (unknown)
W0701 19:16:50.746992 15477 init.cc:224] *** SIGABRT (@0x3f000003c75) received by PID 15477 (TID 0x7fb653a71740) from PID 15477; stack trace: ***
W0701 19:16:50.748991 15477 init.cc:224] @ 0x7fb6531505e0 (unknown)
W0701 19:16:50.751093 15477 init.cc:224] @ 0x7fb6524a21f7 __GI_raise
W0701 19:16:50.753136 15477 init.cc:224] @ 0x7fb6524a38e8 __GI_abort
W0701 19:16:50.755003 15477 init.cc:224] @ 0x7fb649ad3409 __kmp_abort_process
W0701 19:16:50.756867 15477 init.cc:224] @ 0x7fb649ad1aa6 __kmp_fatal
W0701 19:16:50.758623 15477 init.cc:224] @ 0x7fb649ad693b __kmp_register_library_startup()
W0701 19:16:50.760326 15477 init.cc:224] @ 0x7fb649ad6972 __kmp_do_serial_initialize()
W0701 19:16:50.762022 15477 init.cc:224] @ 0x7fb649ad6fc5 __kmp_do_middle_initialize()
W0701 19:16:50.763746 15477 init.cc:224] @ 0x7fb649ad7210 __kmp_middle_initialize
W0701 19:16:50.765491 15477 init.cc:224] @ 0x7fb649b2ed17 __kmp_api_omp_get_max_threads
W0701 19:16:50.766810 15477 init.cc:224] @ 0x7fb5e4f62004 faiss::knn_L2sqr()
W0701 19:16:50.768106 15477 init.cc:224] @ 0x7fb5e4ebe612 faiss::IndexFlat::search()
W0701 19:16:50.769393 15477 init.cc:224] @ 0x7fb5e4eca059 faiss::IndexIVF::search()
W0701 19:16:50.770663 15477 init.cc:224] @ 0x7fb5e4f08830 faiss::IndexIDMapTemplate<>::search()
W0701 19:16:50.771896 15477 init.cc:224] @ 0x7fb5e5099970 _wrap_IndexIDMap_search
W0701 19:16:50.773838 15477 init.cc:224] @ 0x7fb653420429 _PyCFunction_FastCallDict
W0701 19:16:50.775615 15477 init.cc:224] @ 0x7fb6534a5b8c call_function
W0701 19:16:50.777436 15477 init.cc:224] @ 0x7fb6534a8d40 _PyEval_EvalFrameDefault
W0701 19:16:50.779109 15477 init.cc:224] @ 0x7fb6534a3514 _PyFunction_FastCall
W0701 19:16:50.780776 15477 init.cc:224] @ 0x7fb6534a5c88 call_function
W0701 19:16:50.782538 15477 init.cc:224] @ 0x7fb6534a8d40 _PyEval_EvalFrameDefault
W0701 19:16:50.784210 15477 init.cc:224] @ 0x7fb6534a4100 _PyEval_EvalCodeWithName
W0701 19:16:50.785877 15477 init.cc:224] @ 0x7fb6534a5b2a call_function
W0701 19:16:50.787592 15477 init.cc:224] @ 0x7fb6534a8d40 _PyEval_EvalFrameDefault
W0701 19:16:50.789170 15477 init.cc:224] @ 0x7fb6534a3514 _PyFunction_FastCall
W0701 19:16:50.790745 15477 init.cc:224] @ 0x7fb6534a5c88 call_function
W0701 19:16:50.792412 15477 init.cc:224] @ 0x7fb6534a8d40 _PyEval_EvalFrameDefault
W0701 19:16:50.793987 15477 init.cc:224] @ 0x7fb6534a4100 _PyEval_EvalCodeWithName
W0701 19:16:50.795680 15477 init.cc:224] @ 0x7fb6534a4583 PyEval_EvalCodeEx
W0701 19:16:50.797349 15477 init.cc:224] @ 0x7fb6534a45cb PyEval_EvalCode
W0701 19:16:50.798952 15477 init.cc:224] @ 0x7fb6534d6ee0 PyRun_FileExFlags
W0701 19:16:50.800555 15477 init.cc:224] @ 0x7fb6534d84a3 PyRun_SimpleFileExFlags
Aborted
 

@GaoWei8
Copy link
Contributor

GaoWei8 commented Jul 1, 2020

看起来是OpenMP的链接错误。

@HN123-123
Copy link

您好,我在测试模型时也遇到同样的问题,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
W0818 21:20:17.500154 4207 init.cc:216] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0818 21:20:17.500211 4207 init.cc:218] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0818 21:20:17.500232 4207 init.cc:221] The detail failure signal is:

W0818 21:20:17.500249 4207 init.cc:224] *** Aborted at 1597756817 (unix time) try "date -d @1597756817" if you are using GNU date ***
W0818 21:20:17.501518 4207 init.cc:224] PC: @ 0x0 (unknown)
W0818 21:20:17.501742 4207 init.cc:224] *** SIGSEGV (@0x0) received by PID 4207 (TID 0x7fa560547700) from PID 0; stack trace: ***
W0818 21:20:17.502673 4207 init.cc:224] @ 0x7fa560131390 (unknown)
W0818 21:20:17.511160 4207 init.cc:224] @ 0x7fa5424dda13 paddle::imperative::PrepareDataImpl<>()
W0818 21:20:17.512673 4207 init.cc:224] @ 0x7fa5424e03ed paddle::imperative::PrepareOpImpl<>()
W0818 21:20:17.514470 4207 init.cc:224] @ 0x7fa5424de30e paddle::imperative::PreparedOp::Prepare()
W0818 21:20:17.515631 4207 init.cc:224] @ 0x7fa540772117 paddle::imperative::OpBase::Run()
W0818 21:20:17.516969 4207 init.cc:224] @ 0x7fa540779a5e paddle::imperative::Tracer::TraceOp()
W0818 21:20:17.518510 4207 init.cc:224] @ 0x7fa540779fb8 paddle::imperative::Tracer::TraceOp()
W0818 21:20:17.518959 4207 init.cc:224] @ 0x7fa5405694a2 paddle::pybind::imperative_accuracy()
W0818 21:20:17.519373 4207 init.cc:224] @ 0x7fa540625c0d ZZN8pybind1112cpp_function10initializeIRPFSt5tupleIISt10shared_ptrIN6paddle10imperative7VarBaseEES7_S7_EERKS7_SA_SA_S7_S7_RKNS_4argsEES8_ISA_SA_SA_S7_S7_SD_EINS_4nameENS_5scopeENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENKUlRNS_6detail13function_callEE1_clESX
W0818 21:20:17.519701 4207 init.cc:224] @ 0x7fa540625dfe ZZN8pybind1112cpp_function10initializeIRPFSt5tupleIJSt10shared_ptrIN6paddle10imperative7VarBaseEES7_S7_EERKS7_SA_SA_S7_S7_RKNS_4argsEES8_JSA_SA_SA_S7_S7_SD_EJNS_4nameENS_5scopeENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESX
W0818 21:20:17.520020 4207 init.cc:224] @ 0x7fa5404448c9 pybind11::cpp_function::dispatcher()
W0818 21:20:17.520273 4207 init.cc:224] @ 0x557ac52a0744 _PyMethodDef_RawFastCallKeywords
W0818 21:20:17.520479 4207 init.cc:224] @ 0x557ac52a0861 _PyCFunction_FastCallKeywords
W0818 21:20:17.520674 4207 init.cc:224] @ 0x557ac530c2bd _PyEval_EvalFrameDefault
W0818 21:20:17.520854 4207 init.cc:224] @ 0x557ac5250539 _PyEval_EvalCodeWithName
W0818 21:20:17.521020 4207 init.cc:224] @ 0x557ac529ff57 _PyFunction_FastCallKeywords
W0818 21:20:17.521220 4207 init.cc:224] @ 0x557ac53088cc _PyEval_EvalFrameDefault
W0818 21:20:17.521400 4207 init.cc:224] @ 0x557ac5250539 _PyEval_EvalCodeWithName
W0818 21:20:17.521577 4207 init.cc:224] @ 0x557ac5251635 _PyFunction_FastCallDict
W0818 21:20:17.521751 4207 init.cc:224] @ 0x557ac526fe53 _PyObject_Call_Prepend
W0818 21:20:17.521942 4207 init.cc:224] @ 0x557ac5262dbe PyObject_Call
W0818 21:20:17.522140 4207 init.cc:224] @ 0x557ac5309232 _PyEval_EvalFrameDefault
W0818 21:20:17.522316 4207 init.cc:224] @ 0x557ac5250539 _PyEval_EvalCodeWithName
W0818 21:20:17.522491 4207 init.cc:224] @ 0x557ac5251635 _PyFunction_FastCallDict
W0818 21:20:17.522661 4207 init.cc:224] @ 0x557ac526fe53 _PyObject_Call_Prepend
W0818 21:20:17.522756 4207 init.cc:224] @ 0x557ac52a7a3a slot_tp_call
W0818 21:20:17.522948 4207 init.cc:224] @ 0x557ac52a88fb _PyObject_FastCallKeywords
W0818 21:20:17.523211 4207 init.cc:224] @ 0x557ac530be86 _PyEval_EvalFrameDefault
W0818 21:20:17.523424 4207 init.cc:224] @ 0x557ac5250539 _PyEval_EvalCodeWithName
W0818 21:20:17.523638 4207 init.cc:224] @ 0x557ac5251424 PyEval_EvalCodeEx
W0818 21:20:17.523841 4207 init.cc:224] @ 0x557ac525144c PyEval_EvalCode
W0818 21:20:17.523975 4207 init.cc:224] @ 0x557ac5366b74 run_mod
Segmentation fault (core dumped)
请问这个问题您解决了吗

@paddle-bot-old
Copy link

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复,我们将关闭这个issue/pr。
若问题未解决或有后续问题,请随时重新打开,我们会继续跟进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants