Crash after the iteration 1620. Check failed,cublasSgemm

[lemondan]

I run with the caffe for training my dataset. But after the iteration 1620, the program crushed in the cublasSgemm. The log is listed as following, Can you give some advices for fixing this error?

I0127 14:31:22.608165 19425 solver.cpp:204] Iteration 1580, lr = 0.01
I0127 14:31:22.609833 19425 solver.cpp:66] Iteration 1580, loss = 0.0217456
I0127 14:31:49.345432 19425 solver.cpp:204] Iteration 1600, lr = 0.01
I0127 14:31:49.347100 19425 solver.cpp:66] Iteration 1600, loss = 0.0122987
I0127 14:32:16.079083 19425 solver.cpp:204] Iteration 1620, lr = 0.01
I0127 14:32:16.080762 19425 solver.cpp:66] Iteration 1620, loss = 1.67767
F0127 14:32:39.484519 19425 math_functions.cpp:45] Check failed: (cublasSgemm_v2(Caffe::cublas_handle(), cuTransB, cuTransA, N, M, K, &alpha, B, ldb, A, lda, &beta, C, N)) == CUBLAS_STATUS_SUCCESS (14 vs. 0)
*** Check failure stack trace: ***
@ 0x7fa69de70b7d google::LogMessage::Fail()
@ 0x7fa69de72c7f google::LogMessage::SendToLog()
@ 0x7fa69de7076c google::LogMessage::Flush()
@ 0x7fa69de7351d google::LogMessageFatal::~LogMessageFatal()
@ 0x42ee79 caffe::caffe_gpu_gemm<>()
@ 0x45f7e2 caffe::ConvolutionLayer<>::Backward_gpu()
@ 0x42770b caffe::Net<>::Backward()
@ 0x421278 caffe::Solver<>::Solve()
@ 0x40d055 main

[shelhamer]

Please let us know your OS, GPU, and CUDA version. We have not encountered such errors training on Titans and K20s.

This is a possible duplicate of #39 where the cublasSgemm check fails too.

[lemondan]

My version is Ubuntu 12.04 LTS with GeForce GTX Titan, CUDA 5.5. Thanks a lot.

[shelhamer]

Based on similar issues, this is most likely a GPU configuration problem. Check your driver, bios, fan speed, etc. Good luck.