Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_gradient_based_solver fails #3109

Closed
SimoV8 opened this issue Sep 22, 2015 · 17 comments
Closed

test_gradient_based_solver fails #3109

SimoV8 opened this issue Sep 22, 2015 · 17 comments
Assignees
@ronghanghu
Labels
multi-GPU

Comments

@SimoV8
Copy link

SimoV8 commented Sep 22, 2015

I'm running make runtest on a fresh installation of caffe. Many tests work properly, but some of them fail because of an error in test_gradient_based_solver.cpp, what can I do?

.build_release/test/test_all.testbin 0 --gtest_shuffle 
Cuda number of devices: 4
Setting to use device 0
Current device id: 0
Note: Randomizing tests' orders with a seed of 48617 .
[==========] Running 1677 tests from 242 test cases.
[----------] Global test environment set-up.
[----------] 12 tests from DataTransformTest/1, where TypeParam = double
[ RUN      ] DataTransformTest/1.TestEmptyTransformUniquePixels
[       OK ] DataTransformTest/1.TestEmptyTransformUniquePixels (181 ms)
[ RUN      ] DataTransformTest/1.TestMirrorTest
[       OK ] DataTransformTest/1.TestMirrorTest (0 ms)
[ RUN      ] DataTransformTest/1.TestCropTrain
[       OK ] DataTransformTest/1.TestCropTrain (0 ms)
[ RUN      ] DataTransformTest/1.TestCropMirrorTest
[       OK ] DataTransformTest/1.TestCropMirrorTest (0 ms)
[ RUN      ] DataTransformTest/1.TestMeanFile
[       OK ] DataTransformTest/1.TestMeanFile (1 ms)
[ RUN      ] DataTransformTest/1.TestMeanValue
[       OK ] DataTransformTest/1.TestMeanValue (0 ms)
[ RUN      ] DataTransformTest/1.TestMirrorTrain
[       OK ] DataTransformTest/1.TestMirrorTrain (0 ms)
[ RUN      ] DataTransformTest/1.TestCropTest
[       OK ] DataTransformTest/1.TestCropTest (0 ms)
[ RUN      ] DataTransformTest/1.TestMeanValues
[       OK ] DataTransformTest/1.TestMeanValues (0 ms)
[ RUN      ] DataTransformTest/1.TestCropSize
[       OK ] DataTransformTest/1.TestCropSize (0 ms)
[ RUN      ] DataTransformTest/1.TestCropMirrorTrain
[       OK ] DataTransformTest/1.TestCropMirrorTrain (0 ms)
[ RUN      ] DataTransformTest/1.TestEmptyTransform
[       OK ] DataTransformTest/1.TestEmptyTransform (0 ms)
[----------] 12 tests from DataTransformTest/1 (182 ms total)

[----------] 4 tests from ImageDataLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] ImageDataLayerTest/3.TestReshape
[       OK ] ImageDataLayerTest/3.TestReshape (50 ms)
[ RUN      ] ImageDataLayerTest/3.TestShuffle
[       OK ] ImageDataLayerTest/3.TestShuffle (149 ms)
[ RUN      ] ImageDataLayerTest/3.TestRead
[       OK ] ImageDataLayerTest/3.TestRead (150 ms)
[ RUN      ] ImageDataLayerTest/3.TestResize
[       OK ] ImageDataLayerTest/3.TestResize (117 ms)
[----------] 4 tests from ImageDataLayerTest/3 (466 ms total)

[----------] 4 tests from ContrastiveLossLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] ContrastiveLossLayerTest/2.TestForward
[       OK ] ContrastiveLossLayerTest/2.TestForward (3 ms)
[ RUN      ] ContrastiveLossLayerTest/2.TestGradientLegacy
[       OK ] ContrastiveLossLayerTest/2.TestGradientLegacy (445 ms)
[ RUN      ] ContrastiveLossLayerTest/2.TestForwardLegacy
[       OK ] ContrastiveLossLayerTest/2.TestForwardLegacy (3 ms)
[ RUN      ] ContrastiveLossLayerTest/2.TestGradient
[       OK ] ContrastiveLossLayerTest/2.TestGradient (453 ms)
[----------] 4 tests from ContrastiveLossLayerTest/2 (904 ms total)

[----------] 8 tests from LRNLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] LRNLayerTest/0.TestSetupAcrossChannels
[       OK ] LRNLayerTest/0.TestSetupAcrossChannels (0 ms)
[ RUN      ] LRNLayerTest/0.TestForwardAcrossChannelsLargeRegion
[       OK ] LRNLayerTest/0.TestForwardAcrossChannelsLargeRegion (0 ms)
[ RUN      ] LRNLayerTest/0.TestForwardWithinChannel
[       OK ] LRNLayerTest/0.TestForwardWithinChannel (0 ms)
[ RUN      ] LRNLayerTest/0.TestForwardAcrossChannels
[       OK ] LRNLayerTest/0.TestForwardAcrossChannels (0 ms)
[ RUN      ] LRNLayerTest/0.TestGradientAcrossChannelsLargeRegion
[       OK ] LRNLayerTest/0.TestGradientAcrossChannelsLargeRegion (863 ms)
[ RUN      ] LRNLayerTest/0.TestGradientAcrossChannels
[       OK ] LRNLayerTest/0.TestGradientAcrossChannels (746 ms)
[ RUN      ] LRNLayerTest/0.TestGradientWithinChannel
[       OK ] LRNLayerTest/0.TestGradientWithinChannel (628 ms)
[ RUN      ] LRNLayerTest/0.TestSetupWithinChannel
[       OK ] LRNLayerTest/0.TestSetupWithinChannel (0 ms)
[----------] 8 tests from LRNLayerTest/0 (2237 ms total)

[----------] 8 tests from RMSPropSolverTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithEverything
[       OK ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithEverything (38 ms)
[ RUN      ] RMSPropSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] RMSPropSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum (2 ms)
[ RUN      ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithWeightDecay
[       OK ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithWeightDecay (7 ms)
[ RUN      ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithEverythingShare
[       OK ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithEverythingShare (37 ms)
[ RUN      ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithRmsDecay
[       OK ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithRmsDecay (34 ms)
[ RUN      ] RMSPropSolverTest/1.TestSnapshot
[       OK ] RMSPropSolverTest/1.TestSnapshot (11 ms)
[ RUN      ] RMSPropSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] RMSPropSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare (3 ms)
[ RUN      ] RMSPropSolverTest/1.TestSnapshotShare
[       OK ] RMSPropSolverTest/1.TestSnapshotShare (16 ms)
[----------] 8 tests from RMSPropSolverTest/1 (148 ms total)

[----------] 6 tests from CuDNNConvolutionLayerTest/1, where TypeParam = double
[ RUN      ] CuDNNConvolutionLayerTest/1.TestSimpleConvolutionGroupCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestSimpleConvolutionGroupCuDNN (122 ms)
[ RUN      ] CuDNNConvolutionLayerTest/1.TestSobelConvolutionCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestSobelConvolutionCuDNN (5 ms)
[ RUN      ] CuDNNConvolutionLayerTest/1.TestGradientCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestGradientCuDNN (1429 ms)
[ RUN      ] CuDNNConvolutionLayerTest/1.TestSimpleConvolutionCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestSimpleConvolutionCuDNN (3 ms)
[ RUN      ] CuDNNConvolutionLayerTest/1.TestSetupCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestSetupCuDNN (5 ms)
[ RUN      ] CuDNNConvolutionLayerTest/1.TestGradientGroupCuDNN
[       OK ] CuDNNConvolutionLayerTest/1.TestGradientGroupCuDNN (512 ms)
[----------] 6 tests from CuDNNConvolutionLayerTest/1 (2076 ms total)

[----------] 8 tests from AdamSolverTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithWeightDecay
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithWeightDecay (7 ms)
[ RUN      ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare (2 ms)
[ RUN      ] AdamSolverTest/0.TestSnapshot
[       OK ] AdamSolverTest/0.TestSnapshot (12 ms)
[ RUN      ] AdamSolverTest/0.TestSnapshotShare
[       OK ] AdamSolverTest/0.TestSnapshotShare (15 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverythingShare
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverythingShare (35 ms)
[ RUN      ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum (2 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverything
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverything (32 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdate
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdate (7 ms)
[----------] 8 tests from AdamSolverTest/0 (112 ms total)

[----------] 9 tests from AdaGradSolverTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] AdaGradSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] AdaGradSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare (2 ms)
[ RUN      ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithEverything
[       OK ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithEverything (34 ms)
[ RUN      ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithWeightDecay
[       OK ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithWeightDecay (6 ms)
[ RUN      ] AdaGradSolverTest/1.TestSnapshotShare
[       OK ] AdaGradSolverTest/1.TestSnapshotShare (16 ms)
[ RUN      ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdate
[       OK ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdate (7 ms)
[ RUN      ] AdaGradSolverTest/1.TestSnapshot
[       OK ] AdaGradSolverTest/1.TestSnapshot (11 ms)
[ RUN      ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithEverythingShare
[       OK ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateWithEverythingShare (37 ms)
[ RUN      ] AdaGradSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] AdaGradSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum (2 ms)
[ RUN      ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateLROneHundredth
[       OK ] AdaGradSolverTest/1.TestAdaGradLeastSquaresUpdateLROneHundredth (6 ms)
[----------] 9 tests from AdaGradSolverTest/1 (121 ms total)

[----------] 1 test from MultinomialLogisticLossLayerTest/1, where TypeParam = double
[ RUN      ] MultinomialLogisticLossLayerTest/1.TestGradientCPU
[       OK ] MultinomialLogisticLossLayerTest/1.TestGradientCPU (1 ms)
[----------] 1 test from MultinomialLogisticLossLayerTest/1 (1 ms total)

[----------] 11 tests from AdaDeltaSolverTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithMomentumMultiIter
[       OK ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithMomentumMultiIter (32 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithWeightDecay
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithWeightDecay (6 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithHalfMomentum
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithHalfMomentum (13 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestSnapshot
[       OK ] AdaDeltaSolverTest/0.TestSnapshot (11 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithEverythingShare
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithEverythingShare (36 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithMomentum
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithMomentum (12 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare (3 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] AdaDeltaSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum (2 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdate
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdate (6 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithEverything
[       OK ] AdaDeltaSolverTest/0.TestAdaDeltaLeastSquaresUpdateWithEverything (32 ms)
[ RUN      ] AdaDeltaSolverTest/0.TestSnapshotShare
[       OK ] AdaDeltaSolverTest/0.TestSnapshotShare (16 ms)
[----------] 11 tests from AdaDeltaSolverTest/0 (169 ms total)

[----------] 6 tests from FlattenLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] FlattenLayerTest/1.TestSetupWithAxis
[       OK ] FlattenLayerTest/1.TestSetupWithAxis (0 ms)
[ RUN      ] FlattenLayerTest/1.TestGradient
[       OK ] FlattenLayerTest/1.TestGradient (2 ms)
[ RUN      ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis
[       OK ] FlattenLayerTest/1.TestSetupWithStartAndEndAxis (0 ms)
[ RUN      ] FlattenLayerTest/1.TestSetup
[       OK ] FlattenLayerTest/1.TestSetup (0 ms)
[ RUN      ] FlattenLayerTest/1.TestForward
[       OK ] FlattenLayerTest/1.TestForward (0 ms)
[ RUN      ] FlattenLayerTest/1.TestSetupWithEndAxis
[       OK ] FlattenLayerTest/1.TestSetupWithEndAxis (0 ms)
[----------] 6 tests from FlattenLayerTest/1 (2 ms total)

[----------] 22 tests from NetTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] NetTest/0.TestBottomNeedBackwardEuclideanForce
[       OK ] NetTest/0.TestBottomNeedBackwardEuclideanForce (1 ms)
[ RUN      ] NetTest/0.TestParamPropagateDown
[       OK ] NetTest/0.TestParamPropagateDown (2 ms)
[ RUN      ] NetTest/0.TestUnsharedWeightsDiffNet
[       OK ] NetTest/0.TestUnsharedWeightsDiffNet (0 ms)
[ RUN      ] NetTest/0.TestBottomNeedBackwardTricky
[       OK ] NetTest/0.TestBottomNeedBackwardTricky (2 ms)
[ RUN      ] NetTest/0.TestBottomNeedBackward
[       OK ] NetTest/0.TestBottomNeedBackward (1 ms)
[ RUN      ] NetTest/0.TestComboLossWeight
[       OK ] NetTest/0.TestComboLossWeight (3 ms)
[ RUN      ] NetTest/0.TestHasBlob
[       OK ] NetTest/0.TestHasBlob (1 ms)
[ RUN      ] NetTest/0.TestGetBlob
[       OK ] NetTest/0.TestGetBlob (2 ms)
[ RUN      ] NetTest/0.TestHasLayer
[       OK ] NetTest/0.TestHasLayer (1 ms)
[ RUN      ] NetTest/0.TestSkipPropagateDown
[       OK ] NetTest/0.TestSkipPropagateDown (1 ms)
[ RUN      ] NetTest/0.TestBackwardWithAccuracyLayer
[       OK ] NetTest/0.TestBackwardWithAccuracyLayer (2 ms)
[ RUN      ] NetTest/0.TestSharedWeightsUpdate
[       OK ] NetTest/0.TestSharedWeightsUpdate (1 ms)
[ RUN      ] NetTest/0.TestLossWeight
[       OK ] NetTest/0.TestLossWeight (3 ms)
[ RUN      ] NetTest/0.TestSharedWeightsResume
[       OK ] NetTest/0.TestSharedWeightsResume (1 ms)
[ RUN      ] NetTest/0.TestUnsharedWeightsDataNet
[       OK ] NetTest/0.TestUnsharedWeightsDataNet (1 ms)
[ RUN      ] NetTest/0.TestLossWeightMidNet
[       OK ] NetTest/0.TestLossWeightMidNet (3 ms)
[ RUN      ] NetTest/0.TestFromTo
[       OK ] NetTest/0.TestFromTo (2 ms)
[ RUN      ] NetTest/0.TestGetLayerByName
[       OK ] NetTest/0.TestGetLayerByName (1 ms)
[ RUN      ] NetTest/0.TestSharedWeightsDataNet
[       OK ] NetTest/0.TestSharedWeightsDataNet (1 ms)
[ RUN      ] NetTest/0.TestBottomNeedBackwardForce
[       OK ] NetTest/0.TestBottomNeedBackwardForce (1 ms)
[ RUN      ] NetTest/0.TestReshape
[       OK ] NetTest/0.TestReshape (2 ms)
[ RUN      ] NetTest/0.TestSharedWeightsDiffNet
[       OK ] NetTest/0.TestSharedWeightsDiffNet (1 ms)
[----------] 22 tests from NetTest/0 (33 ms total)

[----------] 3 tests from XavierFillerTest/0, where TypeParam = float
[ RUN      ] XavierFillerTest/0.TestFillAverage
[       OK ] XavierFillerTest/0.TestFillAverage (0 ms)
[ RUN      ] XavierFillerTest/0.TestFillFanOut
[       OK ] XavierFillerTest/0.TestFillFanOut (1 ms)
[ RUN      ] XavierFillerTest/0.TestFillFanIn
[       OK ] XavierFillerTest/0.TestFillFanIn (0 ms)
[----------] 3 tests from XavierFillerTest/0 (1 ms total)

[----------] 6 tests from Im2colLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] Im2colLayerTest/0.TestSetup
[       OK ] Im2colLayerTest/0.TestSetup (0 ms)
[ RUN      ] Im2colLayerTest/0.TestForward
[       OK ] Im2colLayerTest/0.TestForward (0 ms)
[ RUN      ] Im2colLayerTest/0.TestRect
[       OK ] Im2colLayerTest/0.TestRect (0 ms)
[ RUN      ] Im2colLayerTest/0.TestGradientForceND
[       OK ] Im2colLayerTest/0.TestGradientForceND (440 ms)
[ RUN      ] Im2colLayerTest/0.TestGradient
[       OK ] Im2colLayerTest/0.TestGradient (324 ms)
[ RUN      ] Im2colLayerTest/0.TestRectGradient
[       OK ] Im2colLayerTest/0.TestRectGradient (288 ms)
[----------] 6 tests from Im2colLayerTest/0 (1052 ms total)

[----------] 1 test from LayerFactoryTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] LayerFactoryTest/1.TestCreateLayer
[       OK ] LayerFactoryTest/1.TestCreateLayer (5 ms)
[----------] 1 test from LayerFactoryTest/1 (5 ms total)

[----------] 3 tests from ThresholdLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] ThresholdLayerTest/0.Test2
[       OK ] ThresholdLayerTest/0.Test2 (0 ms)
[ RUN      ] ThresholdLayerTest/0.Test
[       OK ] ThresholdLayerTest/0.Test (0 ms)
[ RUN      ] ThresholdLayerTest/0.TestSetup
[       OK ] ThresholdLayerTest/0.TestSetup (0 ms)
[----------] 3 tests from ThresholdLayerTest/0 (0 ms total)

[----------] 6 tests from Im2colLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] Im2colLayerTest/2.TestRect
[       OK ] Im2colLayerTest/2.TestRect (1 ms)
[ RUN      ] Im2colLayerTest/2.TestRectGradient
[       OK ] Im2colLayerTest/2.TestRectGradient (3712 ms)
[ RUN      ] Im2colLayerTest/2.TestSetup
[       OK ] Im2colLayerTest/2.TestSetup (1 ms)
[ RUN      ] Im2colLayerTest/2.TestForward
[       OK ] Im2colLayerTest/2.TestForward (1 ms)
[ RUN      ] Im2colLayerTest/2.TestGradient
[       OK ] Im2colLayerTest/2.TestGradient (3846 ms)
[ RUN      ] Im2colLayerTest/2.TestGradientForceND
[       OK ] Im2colLayerTest/2.TestGradientForceND (6430 ms)
[----------] 6 tests from Im2colLayerTest/2 (13991 ms total)

[----------] 1 test from HDF5DataLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] HDF5DataLayerTest/2.TestRead
[       OK ] HDF5DataLayerTest/2.TestRead (8 ms)
[----------] 1 test from HDF5DataLayerTest/2 (8 ms total)

[----------] 5 tests from BenchmarkTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] BenchmarkTest/0.TestTimerStart
[       OK ] BenchmarkTest/0.TestTimerStart (0 ms)
[ RUN      ] BenchmarkTest/0.TestTimerMilliSeconds
[       OK ] BenchmarkTest/0.TestTimerMilliSeconds (301 ms)
[ RUN      ] BenchmarkTest/0.TestTimerConstructor
[       OK ] BenchmarkTest/0.TestTimerConstructor (0 ms)
[ RUN      ] BenchmarkTest/0.TestTimerSeconds
[       OK ] BenchmarkTest/0.TestTimerSeconds (300 ms)
[ RUN      ] BenchmarkTest/0.TestTimerStop
[       OK ] BenchmarkTest/0.TestTimerStop (0 ms)
[----------] 5 tests from BenchmarkTest/0 (601 ms total)

[----------] 6 tests from FlattenLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis
[       OK ] FlattenLayerTest/3.TestSetupWithStartAndEndAxis (2 ms)
[ RUN      ] FlattenLayerTest/3.TestGradient
[       OK ] FlattenLayerTest/3.TestGradient (10 ms)
[ RUN      ] FlattenLayerTest/3.TestSetupWithAxis
[       OK ] FlattenLayerTest/3.TestSetupWithAxis (2 ms)
[ RUN      ] FlattenLayerTest/3.TestSetupWithEndAxis
[       OK ] FlattenLayerTest/3.TestSetupWithEndAxis (2 ms)
[ RUN      ] FlattenLayerTest/3.TestSetup
[       OK ] FlattenLayerTest/3.TestSetup (2 ms)
[ RUN      ] FlattenLayerTest/3.TestForward
[       OK ] FlattenLayerTest/3.TestForward (2 ms)
[----------] 6 tests from FlattenLayerTest/3 (20 ms total)

[----------] 5 tests from DeconvolutionLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] DeconvolutionLayerTest/1.TestGradient3D
[       OK ] DeconvolutionLayerTest/1.TestGradient3D (157 ms)
[ RUN      ] DeconvolutionLayerTest/1.TestGradient
[       OK ] DeconvolutionLayerTest/1.TestGradient (631 ms)
[ RUN      ] DeconvolutionLayerTest/1.TestNDAgainst2D
[       OK ] DeconvolutionLayerTest/1.TestNDAgainst2D (973 ms)
[ RUN      ] DeconvolutionLayerTest/1.TestSetup
[       OK ] DeconvolutionLayerTest/1.TestSetup (0 ms)
[ RUN      ] DeconvolutionLayerTest/1.TestSimpleDeconvolution
[       OK ] DeconvolutionLayerTest/1.TestSimpleDeconvolution (0 ms)
[----------] 5 tests from DeconvolutionLayerTest/1 (1761 ms total)

[----------] 5 tests from MemoryDataLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] MemoryDataLayerTest/1.AddMatVectorDefaultTransform
[       OK ] MemoryDataLayerTest/1.AddMatVectorDefaultTransform (3 ms)
[ RUN      ] MemoryDataLayerTest/1.TestSetup
[       OK ] MemoryDataLayerTest/1.TestSetup (2 ms)
[ RUN      ] MemoryDataLayerTest/1.TestSetBatchSize
[       OK ] MemoryDataLayerTest/1.TestSetBatchSize (3 ms)
[ RUN      ] MemoryDataLayerTest/1.AddDatumVectorDefaultTransform
[       OK ] MemoryDataLayerTest/1.AddDatumVectorDefaultTransform (2 ms)
[ RUN      ] MemoryDataLayerTest/1.TestForward
[       OK ] MemoryDataLayerTest/1.TestForward (19 ms)
[----------] 5 tests from MemoryDataLayerTest/1 (29 ms total)

[----------] 4 tests from ContrastiveLossLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] ContrastiveLossLayerTest/3.TestForwardLegacy
[       OK ] ContrastiveLossLayerTest/3.TestForwardLegacy (5 ms)
[ RUN      ] ContrastiveLossLayerTest/3.TestGradientLegacy
[       OK ] ContrastiveLossLayerTest/3.TestGradientLegacy (491 ms)
[ RUN      ] ContrastiveLossLayerTest/3.TestGradient
[       OK ] ContrastiveLossLayerTest/3.TestGradient (496 ms)
[ RUN      ] ContrastiveLossLayerTest/3.TestForward
[       OK ] ContrastiveLossLayerTest/3.TestForward (4 ms)
[----------] 4 tests from ContrastiveLossLayerTest/3 (996 ms total)

[----------] 3 tests from DummyDataLayerTest/0, where TypeParam = float
[ RUN      ] DummyDataLayerTest/0.TestOneTopConstant
[       OK ] DummyDataLayerTest/0.TestOneTopConstant (0 ms)
[ RUN      ] DummyDataLayerTest/0.TestThreeTopConstantGaussianConstant
[       OK ] DummyDataLayerTest/0.TestThreeTopConstantGaussianConstant (0 ms)
[ RUN      ] DummyDataLayerTest/0.TestTwoTopConstant
[       OK ] DummyDataLayerTest/0.TestTwoTopConstant (0 ms)
[----------] 3 tests from DummyDataLayerTest/0 (0 ms total)

[----------] 6 tests from ArgMaxLayerTest/1, where TypeParam = double
[ RUN      ] ArgMaxLayerTest/1.TestSetupMaxVal
[       OK ] ArgMaxLayerTest/1.TestSetupMaxVal (0 ms)
[ RUN      ] ArgMaxLayerTest/1.TestCPUMaxVal
[       OK ] ArgMaxLayerTest/1.TestCPUMaxVal (0 ms)
[ RUN      ] ArgMaxLayerTest/1.TestSetup
[       OK ] ArgMaxLayerTest/1.TestSetup (0 ms)
[ RUN      ] ArgMaxLayerTest/1.TestCPUMaxValTopK
[       OK ] ArgMaxLayerTest/1.TestCPUMaxValTopK (0 ms)
[ RUN      ] ArgMaxLayerTest/1.TestCPUTopK
[       OK ] ArgMaxLayerTest/1.TestCPUTopK (0 ms)
[ RUN      ] ArgMaxLayerTest/1.TestCPU
[       OK ] ArgMaxLayerTest/1.TestCPU (0 ms)
[----------] 6 tests from ArgMaxLayerTest/1 (0 ms total)

[----------] 10 tests from ConcatLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] ConcatLayerTest/3.TestForwardTrivial
[       OK ] ConcatLayerTest/3.TestForwardTrivial (2 ms)
[ RUN      ] ConcatLayerTest/3.TestGradientChannels
[       OK ] ConcatLayerTest/3.TestGradientChannels (46 ms)
[ RUN      ] ConcatLayerTest/3.TestSetupChannels
[       OK ] ConcatLayerTest/3.TestSetupChannels (2 ms)
[ RUN      ] ConcatLayerTest/3.TestForwardChannels
[       OK ] ConcatLayerTest/3.TestForwardChannels (2 ms)
[ RUN      ] ConcatLayerTest/3.TestSetupNum
[       OK ] ConcatLayerTest/3.TestSetupNum (2 ms)
[ RUN      ] ConcatLayerTest/3.TestGradientNum
[       OK ] ConcatLayerTest/3.TestGradientNum (63 ms)
[ RUN      ] ConcatLayerTest/3.TestGradientTrivial
[       OK ] ConcatLayerTest/3.TestGradientTrivial (9 ms)
[ RUN      ] ConcatLayerTest/3.TestSetupChannelsNegativeIndexing
[       OK ] ConcatLayerTest/3.TestSetupChannelsNegativeIndexing (2 ms)
[ RUN      ] ConcatLayerTest/3.TestGradientChannelsBottomOneOnly
[       OK ] ConcatLayerTest/3.TestGradientChannelsBottomOneOnly (30 ms)
[ RUN      ] ConcatLayerTest/3.TestForwardNum
[       OK ] ConcatLayerTest/3.TestForwardNum (4 ms)
[----------] 10 tests from ConcatLayerTest/3 (162 ms total)

[----------] 3 tests from TanHLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] TanHLayerTest/2.TestTanHGradient
[       OK ] TanHLayerTest/2.TestTanHGradient (19 ms)
[ RUN      ] TanHLayerTest/2.TestTanHOverflow
[       OK ] TanHLayerTest/2.TestTanHOverflow (1 ms)
[ RUN      ] TanHLayerTest/2.TestTanH
[       OK ] TanHLayerTest/2.TestTanH (1 ms)
[----------] 3 tests from TanHLayerTest/2 (21 ms total)

[----------] 26 tests from IOTest
[ RUN      ] IOTest.TestReadImageToDatumReference
[       OK ] IOTest.TestReadImageToDatumReference (10 ms)
[ RUN      ] IOTest.TestReadImageToDatumContent
[       OK ] IOTest.TestReadImageToDatumContent (8 ms)
[ RUN      ] IOTest.TestReadImageToDatum
[       OK ] IOTest.TestReadImageToDatum (4 ms)
[ RUN      ] IOTest.TestReadImageToDatumGray
[       OK ] IOTest.TestReadImageToDatumGray (3 ms)
[ RUN      ] IOTest.TestReadImageToCVMatResizedSquare
[       OK ] IOTest.TestReadImageToCVMatResizedSquare (3 ms)
[ RUN      ] IOTest.TestReadImageToCVMatResizedGray
[       OK ] IOTest.TestReadImageToCVMatResizedGray (3 ms)
[ RUN      ] IOTest.TestReadImageToCVMatResized
[       OK ] IOTest.TestReadImageToCVMatResized (3 ms)
[ RUN      ] IOTest.TestDecodeDatumToCVMatContentNative
[       OK ] IOTest.TestDecodeDatumToCVMatContentNative (11 ms)
[ RUN      ] IOTest.TestDecodeDatumNativeGray
[       OK ] IOTest.TestDecodeDatumNativeGray (4 ms)
[ RUN      ] IOTest.TestDecodeDatumNative
[       OK ] IOTest.TestDecodeDatumNative (10 ms)
[ RUN      ] IOTest.TestReadImageToDatumResized
[       OK ] IOTest.TestReadImageToDatumResized (3 ms)
[ RUN      ] IOTest.TestDecodeDatumToCVMat
[       OK ] IOTest.TestDecodeDatumToCVMat (5 ms)
[ RUN      ] IOTest.TestCVMatToDatumContent
[       OK ] IOTest.TestCVMatToDatumContent (9 ms)
[ RUN      ] IOTest.TestDecodeDatum
[       OK ] IOTest.TestDecodeDatum (10 ms)
[ RUN      ] IOTest.TestReadImageToCVMat
[       OK ] IOTest.TestReadImageToCVMat (3 ms)
[ RUN      ] IOTest.TestReadFileToDatum
[       OK ] IOTest.TestReadFileToDatum (0 ms)
[ RUN      ] IOTest.TestCVMatToDatumReference
[       OK ] IOTest.TestCVMatToDatumReference (9 ms)
[ RUN      ] IOTest.TestReadImageToDatumResizedSquare
[       OK ] IOTest.TestReadImageToDatumResizedSquare (4 ms)
[ RUN      ] IOTest.TestDecodeDatumToCVMatContent
[       OK ] IOTest.TestDecodeDatumToCVMatContent (10 ms)
[ RUN      ] IOTest.TestDecodeDatumToCVMatNative
[       OK ] IOTest.TestDecodeDatumToCVMatNative (4 ms)
[ RUN      ] IOTest.TestReadImageToDatumContentGray
[       OK ] IOTest.TestReadImageToDatumContentGray (5 ms)
[ RUN      ] IOTest.TestReadImageToDatumResizedGray
[       OK ] IOTest.TestReadImageToDatumResizedGray (2 ms)
[ RUN      ] IOTest.TestReadImageToDatumReferenceResized
[       OK ] IOTest.TestReadImageToDatumReferenceResized (7 ms)
[ RUN      ] IOTest.TestCVMatToDatum
[       OK ] IOTest.TestCVMatToDatum (4 ms)
[ RUN      ] IOTest.TestReadImageToCVMatGray
[       OK ] IOTest.TestReadImageToCVMatGray (2 ms)
[ RUN      ] IOTest.TestDecodeDatumToCVMatNativeGray
[       OK ] IOTest.TestDecodeDatumToCVMatNativeGray (2 ms)
[----------] 26 tests from IOTest (138 ms total)

[----------] 12 tests from SGDSolverTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdate
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdate (6 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateLROneHundredth
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateLROneHundredth (6 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithWeightDecay
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithWeightDecay (13 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithMomentum
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithMomentum (13 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingShare
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingShare (36 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingAccum (2 ms)
[ RUN      ] SGDSolverTest/1.TestSnapshot
[       OK ] SGDSolverTest/1.TestSnapshot (10 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverythingAccumShare (2 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithMomentumMultiIter
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithMomentumMultiIter (33 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithWeightDecayMultiIter
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithWeightDecayMultiIter (33 ms)
[ RUN      ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverything
[       OK ] SGDSolverTest/1.TestLeastSquaresUpdateWithEverything (33 ms)
[ RUN      ] SGDSolverTest/1.TestSnapshotShare
[       OK ] SGDSolverTest/1.TestSnapshotShare (14 ms)
[----------] 12 tests from SGDSolverTest/1 (202 ms total)

[----------] 1 test from HDF5OutputLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] HDF5OutputLayerTest/1.TestForward
[       OK ] HDF5OutputLayerTest/1.TestForward (1 ms)
[----------] 1 test from HDF5OutputLayerTest/1 (2 ms total)

[----------] 4 tests from InnerProductLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] InnerProductLayerTest/3.TestSetUp
[       OK ] InnerProductLayerTest/3.TestSetUp (2 ms)
[ RUN      ] InnerProductLayerTest/3.TestForwardNoBatch
[       OK ] InnerProductLayerTest/3.TestForwardNoBatch (3 ms)
[ RUN      ] InnerProductLayerTest/3.TestGradient
[       OK ] InnerProductLayerTest/3.TestGradient (2000 ms)
[ RUN      ] InnerProductLayerTest/3.TestForward
[       OK ] InnerProductLayerTest/3.TestForward (2 ms)
[----------] 4 tests from InnerProductLayerTest/3 (2007 ms total)

[----------] 8 tests from SliceLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] SliceLayerTest/1.TestSliceAcrossChannels
[       OK ] SliceLayerTest/1.TestSliceAcrossChannels (1 ms)
[ RUN      ] SliceLayerTest/1.TestSetupChannels
[       OK ] SliceLayerTest/1.TestSetupChannels (0 ms)
[ RUN      ] SliceLayerTest/1.TestTrivialSlice
[       OK ] SliceLayerTest/1.TestTrivialSlice (0 ms)
[ RUN      ] SliceLayerTest/1.TestGradientTrivial
[       OK ] SliceLayerTest/1.TestGradientTrivial (8 ms)
[ RUN      ] SliceLayerTest/1.TestGradientAcrossNum
[       OK ] SliceLayerTest/1.TestGradientAcrossNum (34 ms)
[ RUN      ] SliceLayerTest/1.TestSliceAcrossNum
[       OK ] SliceLayerTest/1.TestSliceAcrossNum (0 ms)
[ RUN      ] SliceLayerTest/1.TestSetupNum
[       OK ] SliceLayerTest/1.TestSetupNum (0 ms)
[ RUN      ] SliceLayerTest/1.TestGradientAcrossChannels
[       OK ] SliceLayerTest/1.TestGradientAcrossChannels (39 ms)
[----------] 8 tests from SliceLayerTest/1 (82 ms total)

[----------] 8 tests from RMSPropSolverTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] RMSPropSolverTest/2.TestRMSPropLeastSquaresUpdateWithEverythingShare
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.0681678056716919,
solver_updated_weight evaluates to 0.97872507572174072, and
error_margin evaluates to 0.0097872503101825714.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.68911933898925781,
solver_updated_weight evaluates to 0.59967660903930664, and
error_margin evaluates to 0.0059967660345137119.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.1574332714080811,
solver_updated_weight evaluates to 1.2468760013580322, and
error_margin evaluates to 0.011574332602322102.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442610740661621, which exceeds error_margin, where
expected_updated_weight evaluates to -1.7331176996231079,
solver_updated_weight evaluates to -1.6436750888824463, and
error_margin evaluates to 0.016436750069260597.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442655444145203, which exceeds error_margin, where
expected_updated_weight evaluates to 0.16190646588802338,
solver_updated_weight evaluates to 0.25134912133216858, and
error_margin evaluates to 0.0016190647147595882.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -2.1003293991088867,
solver_updated_weight evaluates to -2.1897721290588379, and
error_margin evaluates to 0.021003294736146927.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.5554871559143066,
solver_updated_weight evaluates to 1.6449298858642578, and
error_margin evaluates to 0.015554871410131454.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to 0.37893420457839966,
solver_updated_weight evaluates to 0.46837687492370605, and
error_margin evaluates to 0.0037893420085310936.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to 0.73583024740219116,
solver_updated_weight evaluates to 0.82527291774749756, and
error_margin evaluates to 0.0073583023622632027.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.61082571744918823,
solver_updated_weight evaluates to 0.52138298749923706, and
error_margin evaluates to 0.0052138296887278557.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.08944268524646759, which exceeds error_margin, where
expected_updated_weight evaluates to -0.19101883471012115,
solver_updated_weight evaluates to -0.28046151995658875, and
error_margin evaluates to 0.0019101883517578244.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to 1.0228723287582397,
solver_updated_weight evaluates to 0.93342965841293335, and
error_margin evaluates to 0.0093342969194054604.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.1336876153945923,
solver_updated_weight evaluates to 1.2231303453445435, and
error_margin evaluates to 0.011336876079440117.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.63193082809448242,
solver_updated_weight evaluates to -0.72137355804443359, and
error_margin evaluates to 0.0063193081878125668.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442715048789978, which exceeds error_margin, where
expected_updated_weight evaluates to -0.32952800393104553,
solver_updated_weight evaluates to -0.24008528888225555, and
error_margin evaluates to 0.0024008529726415873.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.0735756158828735,
solver_updated_weight evaluates to -1.1630183458328247, and
error_margin evaluates to 0.010735755786299706.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.43916666507720947,
solver_updated_weight evaluates to 0.52860939502716064, and
error_margin evaluates to 0.0043916665017604828.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.89592903852462769,
solver_updated_weight evaluates to 0.80648630857467651, and
error_margin evaluates to 0.0080648632720112801.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442707598209381, which exceeds error_margin, where
expected_updated_weight evaluates to 0.020399667322635651,
solver_updated_weight evaluates to 0.10984237492084503, and
error_margin evaluates to 0.00020399667846504599.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.4179819822311401,
solver_updated_weight evaluates to -1.328539252281189, and
error_margin evaluates to 0.013285392895340919.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 2.3643043041229248,
solver_updated_weight evaluates to 2.453747034072876, and
error_margin evaluates to 0.023643042892217636.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.0964062213897705,
solver_updated_weight evaluates to -1.0069634914398193, and
error_margin evaluates to 0.010069634765386581.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.32539069652557373,
solver_updated_weight evaluates to 0.4148334264755249, and
error_margin evaluates to 0.0032539069652557373.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.4484039843082428,
solver_updated_weight evaluates to 0.53784668445587158, and
error_margin evaluates to 0.004484039731323719.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.41075518727302551,
solver_updated_weight evaluates to -0.32131245732307434, and
error_margin evaluates to 0.0032131245825439692.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.1062955856323242,
solver_updated_weight evaluates to 1.016852855682373, and
error_margin evaluates to 0.010168528184294701.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.0858433246612549,
solver_updated_weight evaluates to -1.1752860546112061, and
error_margin evaluates to 0.010858433321118355.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.99519288539886475,
solver_updated_weight evaluates to -1.0846356153488159, and
error_margin evaluates to 0.0099519286304712296.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.28243523836135864,
solver_updated_weight evaluates to 0.37187793850898743, and
error_margin evaluates to 0.0028243523556739092.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.39283764362335205,
solver_updated_weight evaluates to -0.30339491367340088, and
error_margin evaluates to 0.0030339490622282028.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.90309077501296997,
solver_updated_weight evaluates to 0.99253350496292114, and
error_margin evaluates to 0.0090309074148535728.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.91239470243453979,
solver_updated_weight evaluates to -0.82295197248458862, and
error_margin evaluates to 0.008229520171880722.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.64705961942672729,
solver_updated_weight evaluates to 0.55761688947677612, and
error_margin evaluates to 0.0055761691182851791.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.08944268524646759, which exceeds error_margin, where
expected_updated_weight evaluates to 0.32463529706001282,
solver_updated_weight evaluates to 0.23519261181354523, and
error_margin evaluates to 0.0023519261740148067.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.46214234828948975,
solver_updated_weight evaluates to 0.37269961833953857, and
error_margin evaluates to 0.003726996248587966.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.59986329078674316,
solver_updated_weight evaluates to 0.68930602073669434, and
error_margin evaluates to 0.0059986328706145287.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.38134607672691345,
solver_updated_weight evaluates to 0.29190334677696228, and
error_margin evaluates to 0.0029190334025770426.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.41132861375808716,
solver_updated_weight evaluates to -0.50077134370803833, and
error_margin evaluates to 0.004113286267966032.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.71234363317489624,
solver_updated_weight evaluates to -0.80178636312484741, and
error_margin evaluates to 0.0071234363131225109.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.50054287910461426,
solver_updated_weight evaluates to 0.41110014915466309, and
error_margin evaluates to 0.004111001268029213.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.91526716947555542,
solver_updated_weight evaluates to -0.82582443952560425, and
error_margin evaluates to 0.0082582440227270126.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.95072376728057861,
solver_updated_weight evaluates to 0.86128103733062744, and
error_margin evaluates to 0.0086128106340765953.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442703872919083, which exceeds error_margin, where
expected_updated_weight evaluates to -0.093630053102970123,
solver_updated_weight evaluates to -0.0041873492300510406, and
error_margin evaluates to 4.1873492591548711e-05.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.57892680168151855,
solver_updated_weight evaluates to 0.66836953163146973, and
error_margin evaluates to 0.0057892678305506706.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.7397282123565674,
solver_updated_weight evaluates to -1.6502854824066162, and
error_margin evaluates to 0.016502855345606804.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.6257833242416382,
solver_updated_weight evaluates to 1.7152260541915894, and
error_margin evaluates to 0.016257833689451218.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.26197928190231323,
solver_updated_weight evaluates to 0.3514220118522644, and
error_margin evaluates to 0.0026197927072644234.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.0557692050933838,
solver_updated_weight evaluates to 1.145211935043335, and
error_margin evaluates to 0.010557692497968674.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442703872919083, which exceeds error_margin, where
expected_updated_weight evaluates to 0.11992640793323517,
solver_updated_weight evaluates to 0.030483704060316086, and
error_margin evaluates to 0.00030483704176731408.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 2.2293267250061035,
solver_updated_weight evaluates to 2.3187694549560547, and
error_margin evaluates to 0.022293267771601677.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.85786294937133789,
solver_updated_weight evaluates to 0.76842021942138672, and
error_margin evaluates to 0.0076842023991048336.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.38772383332252502,
solver_updated_weight evaluates to 0.29828113317489624, and
error_margin evaluates to 0.0029828113038092852.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.13145491480827332,
solver_updated_weight evaluates to -0.2208976149559021, and
error_margin evaluates to 0.0013145491247996688.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442707598209381, which exceeds error_margin, where
expected_updated_weight evaluates to -0.0095283761620521545,
solver_updated_weight evaluates to 0.079914331436157227, and
error_margin evaluates to 9.5283758128061891e-05.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442715048789978, which exceeds error_margin, where
expected_updated_weight evaluates to -0.28776848316192627,
solver_updated_weight evaluates to -0.19832576811313629, and
error_margin evaluates to 0.0019832577090710402.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.043729186058044434,
solver_updated_weight evaluates to 0.045713514089584351, and
error_margin evaluates to 0.00043729186290875077.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442707598209381, which exceeds error_margin, where
expected_updated_weight evaluates to -0.039601121097803116,
solver_updated_weight evaluates to 0.049841586500406265, and
error_margin evaluates to 0.00039601122261956334.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.5211186408996582,
solver_updated_weight evaluates to -1.6105613708496094, and
error_margin evaluates to 0.015211186371743679.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.63648557662963867,
solver_updated_weight evaluates to -0.72592830657958984, and
error_margin evaluates to 0.0063648559153079987.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.81732118129730225,
solver_updated_weight evaluates to 0.72787845134735107, and
error_margin evaluates to 0.0072787846438586712.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.56730997562408447,
solver_updated_weight evaluates to 0.65675270557403564, and
error_margin evaluates to 0.0056730997748672962.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442715048789978, which exceeds error_margin, where
expected_updated_weight evaluates to 0.29200208187103271,
solver_updated_weight evaluates to 0.20255936682224274, and
error_margin evaluates to 0.0020255937706679106.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.1177157163619995,
solver_updated_weight evaluates to -1.0282729864120483, and
error_margin evaluates to 0.010282729752361774.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -2.050178050994873,
solver_updated_weight evaluates to -1.9607353210449219, and
error_margin evaluates to 0.019607353955507278.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.57181090116500854,
solver_updated_weight evaluates to 0.66125363111495972, and
error_margin evaluates to 0.0057181091979146004.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to -0.42313146591186523,
solver_updated_weight evaluates to -0.51257413625717163, and
error_margin evaluates to 0.0042313146404922009.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.39570507407188416,
solver_updated_weight evaluates to 0.48514777421951294, and
error_margin evaluates to 0.0039570508524775505.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -2.0060889720916748,
solver_updated_weight evaluates to -1.9166462421417236, and
error_margin evaluates to 0.019166462123394012.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442715048789978, which exceeds error_margin, where
expected_updated_weight evaluates to 0.2001865953207016,
solver_updated_weight evaluates to 0.28962931036949158, and
error_margin evaluates to 0.0020018659997731447.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.19667679071426392,
solver_updated_weight evaluates to 0.2861194908618927, and
error_margin evaluates to 0.0019667679443955421.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to 0.51950317621231079,
solver_updated_weight evaluates to 0.43006050586700439, and
error_margin evaluates to 0.0043006050400435925.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.6118415594100952,
solver_updated_weight evaluates to 1.7012842893600464, and
error_margin evaluates to 0.016118414700031281.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.1507506370544434,
solver_updated_weight evaluates to 1.2401933670043945, and
error_margin evaluates to 0.011507506482303143.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to -0.47385537624359131,
solver_updated_weight evaluates to -0.56329804658889771, and
error_margin evaluates to 0.0047385538928210735.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.17971956729888916,
solver_updated_weight evaluates to -0.26916226744651794, and
error_margin evaluates to 0.0017971956403926015.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442715048789978, which exceeds error_margin, where
expected_updated_weight evaluates to 0.2427658885717392,
solver_updated_weight evaluates to 0.33220860362052917, and
error_margin evaluates to 0.0024276589974761009.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442703872919083, which exceeds error_margin, where
expected_updated_weight evaluates to -0.0065182410180568695,
solver_updated_weight evaluates to 0.082924462854862213, and
error_margin evaluates to 6.5182408434338868e-05.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.56556856632232666,
solver_updated_weight evaluates to 0.65501129627227783, and
error_margin evaluates to 0.0056556854397058487.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.22528168559074402,
solver_updated_weight evaluates to -0.13583898544311523, and
error_margin evaluates to 0.0013583898544311523.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442703872919083, which exceeds error_margin, where
expected_updated_weight evaluates to -0.023510020226240158,
solver_updated_weight evaluates to -0.11295272409915924, and
error_margin evaluates to 0.00023510020400863141.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.54767173528671265,
solver_updated_weight evaluates to -0.63711446523666382, and
error_margin evaluates to 0.005476717371493578.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442705735564232, which exceeds error_margin, where
expected_updated_weight evaluates to -0.028965046629309654,
solver_updated_weight evaluates to 0.060477659106254578, and
error_margin evaluates to 0.00028965045930817723.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.075612306594848633,
solver_updated_weight evaluates to -0.16505500674247742, and
error_margin evaluates to 0.0007561230449937284.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.71186208724975586,
solver_updated_weight evaluates to 0.62241935729980469, and
error_margin evaluates to 0.0062241936102509499.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 0.57383918762207031,
solver_updated_weight evaluates to 0.66328191757202148, and
error_margin evaluates to 0.0057383920066058636.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.2244682312011719,
solver_updated_weight evaluates to 1.313910961151123, and
error_margin evaluates to 0.012244682759046555.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.38451087474823,
solver_updated_weight evaluates to -1.2950681447982788, and
error_margin evaluates to 0.012950681149959564.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.66358798742294312,
solver_updated_weight evaluates to -0.57414525747299194, and
error_margin evaluates to 0.0057414527982473373.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -1.272305965423584,
solver_updated_weight evaluates to -1.1828632354736328, and
error_margin evaluates to 0.011828632093966007.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.17693130671977997,
solver_updated_weight evaluates to 0.087488606572151184, and
error_margin evaluates to 0.00087488605640828609.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to -0.35261321067810059,
solver_updated_weight evaluates to -0.44205594062805176, and
error_margin evaluates to 0.0035261320881545544.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.1726519912481308,
solver_updated_weight evaluates to -0.083209291100502014, and
error_margin evaluates to 0.00083209289005026221.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 2.455979585647583,
solver_updated_weight evaluates to 2.3665368556976318, and
error_margin evaluates to 0.023665368556976318.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to 0.13342031836509705,
solver_updated_weight evaluates to 0.043977618217468262, and
error_margin evaluates to 0.00043977619498036802.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442670345306396, which exceeds error_margin, where
expected_updated_weight evaluates to -0.7673683762550354,
solver_updated_weight evaluates to -0.677925705909729, and
error_margin evaluates to 0.006779257208108902.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.0000710487365723,
solver_updated_weight evaluates to 1.0895137786865234, and
error_margin evaluates to 0.010000710375607014.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.08944271132349968, which exceeds error_margin, where
expected_updated_weight evaluates to -0.076422721147537231,
solver_updated_weight evaluates to 0.013019990175962448, and
error_margin evaluates to 0.00013019990001339465.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442700147628784, which exceeds error_margin, where
expected_updated_weight evaluates to -0.34078282117843628,
solver_updated_weight evaluates to -0.2513401210308075, and
error_margin evaluates to 0.002513401210308075.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442707598209381, which exceeds error_margin, where
expected_updated_weight evaluates to 0.02112220972776413,
solver_updated_weight evaluates to -0.068320497870445251, and
error_margin evaluates to 0.0002112220972776413.
src/caffe/test/test_gradient_based_solver.cpp:392: Failure
The difference between expected_updated_bias and solver_updated_bias is 0.089442729949951172, which exceeds error_margin, where
expected_updated_bias evaluates to 0.40764647722244263,
solver_updated_bias evaluates to 0.4970892071723938, and
error_margin evaluates to 0.0040764645673334599.
src/caffe/test/test_gradient_based_solver.cpp:384: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.089442729949951172, which exceeds error_margin, where
expected_updated_weight evaluates to 1.0681678056716919,
solver_updated_weight evaluates to 0.97872507572174072, and
error_margin evaluates to 0.0097872503101825714.
...
@vuptran
Copy link

vuptran commented Oct 6, 2015

I'm also getting this error using the latest caffe master. Can someone look into this?

@jeffdonahue
Copy link
Contributor

I have one machine on which some of the TestGradientBasedSolver GPU tests fail, whether using a Tesla K40 or GTX 980. I git bisected back to the Multi-GPU merge (#2903) -- specifically commit e5575cf fails, while the previous one (d2f0457) passes. (All other tests pass, before and after the merge.) I'm not sure what to make of it -- I checked that the driver and CUDA versions exactly match those on a known-good configuration which passes all tests. It could be that there is still a race condition lurking somewhere...

While it's not overly comforting, I will say that it didn't seem to make a difference in actual training behavior -- using an RNG seed, I got the same results before and after the multi-GPU merge on that machine.

@ronghanghu
Copy link
Member

I will look into this problem some time in this week.

@ronghanghu ronghanghu added the RH label Oct 6, 2015
@ronghanghu ronghanghu self-assigned this Oct 6, 2015
@ronghanghu ronghanghu removed the RH label Oct 6, 2015
@ronghanghu
Copy link
Member

I tried to look into this today.

Although I haven't been able to reproduce the error in this PR, I got some other errors in TestGradientBasedSolver on my laptop (Ubuntu 14.04) with CPU-only build.

    [----------] Global test environment tear-down
    [==========] 906 tests from 133 test cases ran. (72964 ms total)
    [  PASSED  ] 899 tests.
    [  FAILED  ] 7 tests, listed below:
    [  FAILED  ] SGDSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] AdaGradSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] NesterovSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] AdaDeltaSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] AdamSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] RMSPropSolverTest/0.TestSnapshot, where TypeParam = caffe::CPUDevice<float>
    [  FAILED  ] RMSPropSolverTest/0.TestSnapshotShare, where TypeParam = caffe::CPUDevice<float>

This seems to trace back to #2836

The difference between actual and expected values are very small, but they are not the same. Part of the output log:

    src/caffe/test/test_gradient_based_solver.cpp:566: Failure
    Value of: history[i]->cpu_data()[j]
      Actual: 0.0555761
    Expected: history_copies[i]->cpu_data()[j]
    Which is: 0.055576
    history blob 2 data differed at dim 288
    src/caffe/test/test_gradient_based_solver.cpp:566: Failure
    Value of: history[i]->cpu_data()[j]
      Actual: 0.105941
    Expected: history_copies[i]->cpu_data()[j]
    Which is: 0.105942
    history blob 2 data differed at dim 290
    src/caffe/test/test_gradient_based_solver.cpp:566: Failure
    Value of: history[i]->cpu_data()[j]
      Actual: 0.215496
    Expected: history_copies[i]->cpu_data()[j]
    Which is: 0.215496
    history blob 2 data differed at dim 291
    src/caffe/test/test_gradient_based_solver.cpp:566: Failure
    Value of: history[i]->cpu_data()[j]
      Actual: 0.42607
    Expected: history_copies[i]->cpu_data()[j]
    Which is: 0.42607
    history blob 2 data differed at dim 293

Update: I found the cause of my issue mentioned above (not the original issue in this PR): Intel MKL's float point operations (such as matrix multiplication) are non-deterministic by default on my laptop.

After export MKL_CBWR=AUTO, the results are deterministic and those snapshot test failures disappear.

Reference on deterministic property of Intel MKL:

@ronghanghu
Copy link
Member

@SimoV8 @jeffdonahue If you are running your tests on a machine with multiple GPUs, could you try setting export CUDA_VISIBLE_DEVICES=0 (only run single GPU test on GPU 0) and run the tests again, to identify whether this is a multi-gpu issue?

This was referenced Oct 8, 2015
Closed
Open
@vuptran
Copy link

vuptran commented Oct 8, 2015

I have a multi-GPU machine, and following the above suggestion by @ronghanghu I was able to successfully pass all tests.

@SimoV8
Copy link
Author

SimoV8 commented Oct 8, 2015

Yes, I'm using a multi-GPU machine too and setting export CUDA_VISIBLE_DEVICES=0 I was able to pass all the tests.

@ronghanghu
Copy link
Member

Based on the feed backs I suspect this is most likely a multiple GPU issue, and does not affect single GPU training.

Probably the GPU's failed to communicate due to some hardware configuration issues (such as PCI topology), or maybe we can have some bugs with communication (for example, the next iteration starts before synchronization is over, or some race condition?).

@mpekalski mpekalski mentioned this issue Dec 25, 2015
Open
@sliterok
Copy link

This is not multiple gpu problem.

On OpenCL caffe version this problem also appends in single gpu

@doonny
Copy link

doonny commented Jan 21, 2016

@sliterok Hi sliterok， I got the same issue on OpenCL caffe, I am using W9100 GPU. Have you fixed this issue yet ?

@sliterok
Copy link

@doonny, nope, still waiting for fix from AMD. Subscribe to OpenCL Caffe issue and wait as me.

@Kindaichi
Copy link

+1, export CUDA_VISIBLE_DEVICES=0 work for me

@yechaochen
Copy link

My machine is single GPU and I meet some problem when I used make runtest command some process named TestNDAgainst2D is failed.Can someone tell me how to fix it.

@Deepak- Deepak- mentioned this issue May 28, 2016
Closed
@monaen
Copy link

monaen commented Jun 19, 2016

+1 export MKL_CBWR=AUTO works for me

@cervantes-loves-ai
Copy link

i've this error how to solve it? thanks in advance..
[----------] 5 tests from ImageDataLayerTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] ImageDataLayerTest/1.TestReshape
[ OK ] ImageDataLayerTest/1.TestReshape (46 ms)
[ RUN ] ImageDataLayerTest/1.TestShuffle
[ OK ] ImageDataLayerTest/1.TestShuffle (106 ms)
[ RUN ] ImageDataLayerTest/1.TestRead
[ OK ] ImageDataLayerTest/1.TestRead (103 ms)
[ RUN ] ImageDataLayerTest/1.TestResize
[ OK ] ImageDataLayerTest/1.TestResize (122 ms)
[ RUN ] ImageDataLayerTest/1.TestSpace
[ OK ] ImageDataLayerTest/1.TestSpace (63 ms)
[----------] 5 tests from ImageDataLayerTest/1 (440 ms total)

[----------] Global test environment tear-down
[==========] 1096 tests from 150 test cases ran. (68513 ms total)
[ PASSED ] 1095 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] NeuronLayerTest/0.TestPReLUForward, where TypeParam = caffe::CPUDevice

1 FAILED TEST
make[3]: *** [src/caffe/test/CMakeFiles/runtest] Error 1
make[2]: *** [src/caffe/test/CMakeFiles/runtest.dir/all] Error 2
make[1]: *** [src/caffe/test/CMakeFiles/runtest.dir/rule] Error 2
make: *** [runtest] Error 2

@scotwilli
Copy link

Thanks to the post. I too got errors for "make runtest" with four gpus. Initially I thought the error might be due to latest caffe repository. So I tried to use earlier caffe versions. But still I got the same errors. Finally after going through this post, I used this command export CUDA_VISIBLE_DEVICES=0, which resolved all the errors.

I want to know why make runtest fails with multiple gpus.
Regards
sk06

@thomhert thomhert mentioned this issue Oct 19, 2016
Closed
@jwgu jwgu mentioned this issue Dec 7, 2016
Closed
@shelhamer
Copy link
Member

This was caused by the multi-GPU edition of the tests fixed with the parallelism switch in #4563.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ronghanghu ronghanghu

Labels
multi-GPU
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests