Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

paddle exp计算出core (vsExp),浮点计算溢出? #2024

Closed
sdujq opened this issue May 5, 2017 · 4 comments
Closed

paddle exp计算出core (vsExp),浮点计算溢出? #2024

sdujq opened this issue May 5, 2017 · 4 comments
Assignees

Comments

@sdujq
Copy link

sdujq commented May 5, 2017

#0  0x00007fc79287b0c5 in vsexp_cout_rare () from /home/work/daijunkai/du-da/bin/../lib/libpaddle_capi_shared.so
#1  0x00007fc79287aa30 in mkl_vml_kernel_sExp_E9HAynn () from /home/work/daijunkai/du-da/bin/../lib/libpaddle_capi_shared.so
#2  0x00007fc7927ef110 in vsExp () from /home/work/daijunkai/du-da/bin/../lib/libpaddle_capi_shared.so
#3  0x00007fc792711a93 in paddle::CpuMatrix::softmax (this=<optimized out>, output=...)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/math/Matrix.cpp:3389
#4  0x00007fc79262c11c in paddle::softmaxActivation::forward (this=<optimized out>, act=...)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/gserver/activations/ActivationFunction.cpp:119
#5  0x00007fc792605c86 in paddle::Layer::forwardActivation (this=this@entry=0x7fc58d22f200)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/gserver/layers/Layer.cpp:338
#6  0x00007fc7925e5378 in paddle::FullyConnectedLayer::forward (this=0x7fc58d22f200, passType=<optimized out>)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/gserver/layers/FullyConnectedLayer.cpp:99
#7  0x00007fc792654d7c in paddle::NeuralNetwork::forward (this=0x7fc5a0a53010, inArgs=..., outArgs=0x7fbfe0205188, 
    passType=paddle::enumeration_wrapper::PASS_TEST)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/gserver/gradientmachines/NeuralNetwork.cpp:248
#8  0x00007fc7924d0796 in paddle_gradient_machine_forward (machine=<optimized out>, inArgs=<optimized out>, 
    outArgs=<optimized out>, isTrain=<optimized out>)
    at /home/users/wangshuai21/dev_paddle/Paddle/paddle/capi/gradient_machine.cpp:81
#9  0x00007fc7954f3a4e in PaddleModel::predict (this=0x7fc060691aa0, features=..., modelResult=..., apolloInput=...)
    at src/paddle_model.cpp:200
#10 0x00007fc7954fe4d1 in BaseIntentDetector::detect (this=0x7fc05f6b5f20, features=..., detectorResult=..., apolloInput=...)
    at src/base_intent_detector.cpp:129
#11 0x00007fc7954e623c in IntentProcessor::process (this=0x7fc0cc287080, apolloInput=..., processerRet=..., 
    dict=0x7fc607576d00) at src/intent_processer.cpp:378
#12 0x00007fc7953fd129 in DaService::apollo_process (this=0xf06320) at ps/se/saiya/du-da/src/da_service.cpp:3297
#13 0x00007fc7953fe6e2 in DaService::fill_us_output (this=0xf06320, input=0x7fbff97995e0, output=0x7fc05a9f5180)
    at ps/se/saiya/du-da/src/da_service.cpp:3481
#14 0x00007fc7953fa603 in DaService::analyze (this=0xf06320, input=0x7fbff97995e0, output=0x7fc05a9f5180, 
    write_to_session=0x7fbfe0202120, write_to_user_info=0x7fc00e0fdec0) at ps/se/saiya/du-da/src/da_service.cpp:2884
#15 0x00007fc7953f40f3 in DaService::run (this=0xf06320, conf_name=...) at ps/se/saiya/du-da/src/da_service.cpp:1911
#16 0x00007fc797c769df in AcWorkServer::_parseApplication (this=this@entry=0xae1050, 
    read_data=read_data@entry=0x7fc0d40f7790, static_conf=static_conf@entry=0xa128c0, 
    plugin_manager=plugin_manager@entry=0xa3ca90) at ps/se/nlp-arch/rac/src/AcWorkServer.cpp:130
#17 0x00007fc797c76fb1 in AcWorkServer::run (this=0xae1050) at ps/se/nlp-arch/rac/src/AcWorkServer.cpp:725
#18 0x00000000004b6ae3 in rp::work_thread_run (arg=<optimized out>)
    at ps/se/nlp-arch/service-common/server/src/work_thread_run.cpp:108
#19 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#20 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
@hedaoyuan
Copy link
Contributor

@sdujq 调用栈看起来是已经走到MKL函数mkl_vml_kernel_sExp_E9HAynn里面,出错异常是什么(如果是浮点异常的话会有一个SIGFPE)?我会check一下什么样的数据会导致MKL函数vsExp异常,另外,你那有可以复现的环境吗?

@sdujq
Copy link
Author

sdujq commented May 5, 2017

对有sigfpe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
Core was generated by `/home/work/daijunkai/du-da/bin/./rac -d ../conf/ -f rac.conf'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0 0x00007fc79287b0c5 in vsexp_cout_rare () from /home/work/daijunkai/du-da/bin/../lib/libpaddle_capi_shared.so
[Current thread is 1 (Thread 0x7fc0e4af4960 (LWP 32104))

@hedaoyuan
Copy link
Contributor

浮点异常的原因是,在调用vsExp的参数中出现了过大的值或者非法值,并且在capi的initPaddle里面设置了异常捕获feenableexcept(FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW);
@sdujq 如果要准确定义是什么原因导致的vsExp参数中出现了过大值或非法值,需要能够复现环境,定位具体是什么异常(溢出、除零、非法值等)。如果,期望的是程序稳定,不要core的话,可以去掉initPaddle中的对浮点异常的捕获的语句就行了。

@hedaoyuan
Copy link
Contributor

@sdujq The issue has been closed by PR # 2043. If there are other problems, you can Reopen and comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants