Fix SP unstable output, synapse decay #1382

breznak · 2018-01-11T03:46:38Z

This PR adds tests for identified issue where c++ SP starts to misbehave under certain conditions and returns (random) different values for the same inputs.
Identified at least 2 cases

unstable set of params test
- fix for unstable

Fixes: #1380
Obsoletes: #1381

EDIT:
separate into several atomic PRs

… 0 boosting

the unstable combination of constructor parameters can cause SP output to be different on the same input, which is clearly wrong!

breznak

Please review, this provides test-cases for 2 identified problems with SP.

breznak · 2018-01-11T14:22:42Z

src/nupic/algorithms/SpatialPooler.cpp

@@ -488,6 +488,18 @@ const vector<Real>& SpatialPooler::getBoostedOverlaps() const
  return boostedOverlaps_;
 }

+/** helper method that checks SP output is stable for given configuration */
+bool checkUnstableParams_(SpatialPooler &sp) {


this is the "stability check" that I suggest to run on each SP initialization. So far it has already identified many misconfigurations in our unit-tests.

breznak · 2018-01-11T14:24:44Z

src/nupic/algorithms/SpatialPooler.cpp

+/** helper method that checks SP output is stable for given configuration */
+bool checkUnstableParams_(SpatialPooler &sp) {
+  vector<UInt> input = {1,1,0,0,1,0}; //FIXME auto to size of SP's input
+  if(sp.getInputDimensions().size()>1) return true; //FIXME skip nD as I cannot set auto input correctly


TODO: I need some "automatic" input. And it must fit to the dimensions specified for the SP's input.
In most cases it's 1D and hard-coded value is sufficient, but for nD I'd need something as sp.getDemoInput(); or is there any getter that has same dimensions as the input?

breznak · 2018-01-11T14:26:09Z

src/nupic/algorithms/SpatialPooler.cpp

+  vector<UInt> out2(sp.getNumColumns(), 0); 
+  sp.compute(input.data(), true, out1.data());
+  sp.compute(input.data(), true, out2.data());
+  //TODO should we add SP.reset() and call it here? 


TODO: now, the single run through the values is "stored" in SP's memory. Should we implement something as SP::reset() (that just calls initialize with the same params) to avoid that?

breznak · 2018-01-11T14:26:42Z

src/nupic/algorithms/SpatialPooler.cpp

@@ -593,6 +605,9 @@ void SpatialPooler::initialize(vector<UInt> inputDimensions,
    printParameters();
    std::cout << "CPP SP seed                 = " << seed << std::endl;
  }
+
+  //check for reasonable params
+  NTA_CHECK(checkUnstableParams_(*this)); //TODO the assert runs only at debug builds, make mandatory?


TODO: make this mandatory even for Release builds?

Currently the check crashes? on some instances, while already identifies some problematic instances in our unit-tests.

.. crash fixed in #1391

breznak · 2018-01-11T14:27:14Z

src/test/unit/algorithms/SpatialPoolerTest.cpp

+  different output for the same input. 
+  XXX sensitive - marks (empirically!) discovered set of parameter 
+  that produce the err behavior; changing any of the XXX params 
+  may cause the SP to behave "normally"


Test-case for unstable combination of init params

breznak · 2018-01-11T14:28:29Z

src/test/unit/algorithms/SpatialPoolerTest.cpp

+  {
+  /** this test exposes wrong behavior, where SP created via constructor
+  behaves differently to a SP via setters (setXXX()), both with the same
+  params. 


Test case: different behavior when setters vs constructor is used for initialization

breznak · 2018-01-11T14:37:03Z

My patches cause some segfault, can you help me debug that please?

gdb run on SP unittests

Program received signal SIGSEGV, Segmentation fault.
0x00000000009341df in nupic::algorithms::spatial_pooler::SpatialPooler::isWinner_ (this=0x7fffffffca48, score=0,
winners=std::vector of length 0, capacity 0, numWinners=0) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:1246
1246 if (score >= winners[numWinners-1].second)

bt

(gdb) bt
#0  0x00000000009341df in nupic::algorithms::spatial_pooler::SpatialPooler::isWinner_ (this=0x7fffffffca48, score=0,
    winners=std::vector of length 0, capacity 0, numWinners=0) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:1246
#1  0x00000000009339c7 in nupic::algorithms::spatial_pooler::SpatialPooler::inhibitColumnsGlobal_ (this=0x7fffffffca48,
    overlaps=std::vector of length 1, capacity 1 = {...}, density=0.5, activeColumns=std::vector of length 0, capacity 0)
    at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:1280
#2  0x0000000000930841 in nupic::algorithms::spatial_pooler::SpatialPooler::inhibitColumns_ (this=0x7fffffffca48,
    overlaps=std::vector of length 1, capacity 1 = {...}, activeColumns=std::vector of length 0, capacity 0)
    at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:1225
#3  0x0000000000930363 in nupic::algorithms::spatial_pooler::SpatialPooler::compute (this=0x7fffffffca48, inputArray=0xf083d0, learn=true,
    activeArray=0xee4160) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:629
#4  0x000000000092de9e in checkUnstableParams_ (sp=...) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:497
#5  0x000000000092f245 in nupic::algorithms::spatial_pooler::SpatialPooler::initialize (this=0x7fffffffca48,
    inputDimensions=std::vector of length 1, capacity 1 = {...}, columnDimensions=std::vector of length 1, capacity 1 = {...}, potentialRadius=16,
    potentialPct=0.5, globalInhibition=true, localAreaDensity=-1, numActiveColumnsPerInhArea=10, stimulusThreshold=0, synPermInactiveDec=0.00800000038,
    synPermActiveInc=0.0500000007, synPermConnected=0.100000001, minPctOverlapDutyCycles=0.00100000005, dutyCyclePeriod=1000, boostStrength=0, seed=1,
    spVerbosity=0, wrapAround=true) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:610
#6  0x000000000092c20f in nupic::algorithms::spatial_pooler::SpatialPooler::SpatialPooler (this=0x7fffffffca48,
    inputDimensions=std::vector of length 1, capacity 1 = {...}, columnDimensions=std::vector of length 1, capacity 1 = {...}, potentialRadius=16,
    potentialPct=0.5, globalInhibition=true, localAreaDensity=-1, numActiveColumnsPerInhArea=10, stimulusThreshold=0, synPermInactiveDec=0.00800000038,
    synPermActiveInc=0.0500000007, synPermConnected=0.100000001, minPctOverlapDutyCycles=0.00100000005, dutyCyclePeriod=1000, boostStrength=0, seed=1,
    spVerbosity=0, wrapAround=true) at /home/mmm/devel/HTM/nupic.core/src/nupic/algorithms/SpatialPooler.cpp:142
#7  0x00000000006f6319 in (anonymous namespace)::SpatialPoolerTest_testMapColumn_Test::TestBody (this=0xee4490)
    at /home/mmm/devel/HTM/nupic.core/src/test/unit/algorithms/SpatialPoolerTest.cpp:1863
#8  0x00000000008c384a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0xee4490,
    method=&virtual testing::Test::TestBody(), location=0xb18d6d "the test body")
    at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3562
#9  0x00000000008b4407 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0xee4490,
    method=&virtual testing::Test::TestBody(), location=0xb18d6d "the test body")
    at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3598
#10 0x000000000089ce05 in testing::Test::Run (this=0xee4490) at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3634
#11 0x000000000089dac8 in testing::TestInfo::Run (this=0xec78d0) at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3810
#12 0x000000000089e187 in testing::TestCase::Run (this=0xec5310) at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3928
#13 0x00000000008a5941 in testing::internal::UnitTestImpl::RunAllTests (this=0xec0db0)
    at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:5799
#14 0x00000000008c622a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xec0db0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x8a5660 <testing::internal::UnitTestImpl::RunAllTests()>,
    location=0xb19471 "auxiliary test code (environments or event listeners)")
    at /home/mmm/devel/HTM/nupic.core/external/common/src/gtest/gtest-all.cpp:3562
#15 0x00000000008b6457 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xec0db0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x8a5660 <testing::internal::UnitTestImpl::RunAllTests()>,

breznak · 2018-01-11T14:44:33Z

@scottpurdy @mrcslws @lscheinkman can you please have a look on the failing tests? (it's correct that the tests fail, that's the PR detecting hidden problems)

@rhyolight A disadvantage with the circleCI is that I cannot link to specific lines of the build anymore (?)

on each initialization, SP is tested to produce stable output.

SP start

as it's required

breznak · 2018-01-12T13:43:43Z

Update: the crash is fixed in #1391 and different params in #1387 , these 2 PRs need to be merged before work on this.

TL;DR: Problem: cannot reproduce the diverging output as @scottpurdy reported in #1380

breznak · 2018-01-12T14:08:56Z

Closing as cannot reproduce. Waiting for feedback in the issue on how to reproduce.

lscheinkman and others added 2 commits January 9, 2018 23:52

Issue numenta#1380: Add SP test checking results with no learning and…

eba72d0

… 0 boosting

working SP test for unstable params

30f9227

the unstable combination of constructor parameters can cause SP output to be different on the same input, which is clearly wrong!

breznak mentioned this pull request Jan 11, 2018

Issue #1380: Add SP test checking results with no learning and 0 boos… #1381

Closed

breznak commented Jan 11, 2018

View reviewed changes

breznak mentioned this pull request Jan 12, 2018

Issue #1380: Validate python parameters passed to SP compute method #1385

Merged

SP: add check for unstable params in constructor

c01a95b

on each initialization, SP is tested to produce stable output.

breznak force-pushed the fix_sp_synapsedecay branch from 2c0bdc0 to c01a95b Compare January 12, 2018 03:18

breznak requested a review from mrcslws January 12, 2018 03:20

breznak added 2 commits January 12, 2018 11:59

SP: unstable compute test: add burn in to make stable

b34d1cf

SP start

SP unstable output check: use no learning

9d4c4a2

as it's required

breznak closed this Jan 12, 2018

breznak deleted the fix_sp_synapsedecay branch January 12, 2018 14:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix SP unstable output, synapse decay #1382

Fix SP unstable output, synapse decay #1382

breznak commented Jan 11, 2018 •

edited

Loading

breznak left a comment

breznak Jan 11, 2018

breznak Jan 11, 2018

breznak Jan 11, 2018

breznak Jan 11, 2018

breznak Jan 12, 2018

breznak Jan 12, 2018

breznak Jan 11, 2018

breznak Jan 11, 2018

breznak commented Jan 11, 2018 •

edited

Loading

breznak commented Jan 11, 2018

breznak commented Jan 12, 2018

breznak commented Jan 12, 2018

Fix SP unstable output, synapse decay #1382

Fix SP unstable output, synapse decay #1382

Conversation

breznak commented Jan 11, 2018 • edited Loading

breznak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak commented Jan 11, 2018 • edited Loading

breznak commented Jan 11, 2018

breznak commented Jan 12, 2018

breznak commented Jan 12, 2018

breznak commented Jan 11, 2018 •

edited

Loading

breznak commented Jan 11, 2018 •

edited

Loading