Dropout WIP

[breznak]

WIP dropout implementation

• dropout works
• move to connections?
• use for SP, TM
• doc

EDIT:
Motivation: I believe this change can be considered biological (noise on signal during transfer) and also acording to deep learning (more robust representations). It should be supported by measurable of SDR quality #155

[breznak]

is it OK to rename this to shuffle() and have addNoise for the new fn? @ctrl-z-9000-times

[ctrl-z-9000-times]

I looked at your new addNoise function and I think it will have issues with keeping the sparsity at a reasonable level. I think that the sparsity of an SDR after this method is called on it will always tend towards 50%.

[breznak]

that would be indeed wrong. What I intended:

• have SDR of current input
  • flip 0.01% bits
• have a new SDR
  • flip 0.01%bits

So the sparsity would remain the same (actially grow, because we have much more off bits, so flipping on would be more probable). But it should remain the x% (2%) + 0.001%

[breznak]

I'm trying to write an effective implementation, but this has problem with p << size. Should we bother with such cases? return/assert?

[breznak]

proof of concept dropout applied to input (as noise) and output (as killCells).

• I'd prefer this be applied in Connections (in adaptSegment?)
• where to apply?
  • ideally all of: SP, TM. & synapse, segment, cell, column
  • but that would be computationally infeasible, so..?

[breznak]

Deterministic test are still expected to fail, until we decide on values and update the exact outputs.

[ctrl-z-9000-times]

Maybe I don't understand this change, but it seems this will make the HTM perform worse. While it's interesting that the HTM keeps working even when some of its components are disabled, I don't think this belongs in the mainline. Maybe instead you could make an example/demonstration of these fault-tolerance properties (like numenta did in their SP paper).

[breznak]

but it seems this will make the HTM perform worse.

it's commonly used in deeplearning where it improves a lot. To be exact, dropout helps to prevent overfitting.

While HTM is already more robust to that (sparse SDR for output, stimulus threshold on input segments) I want to see if this helps and how much.
For now, on MNIST this gave a little (1-2%) better results, that is similar as impact of boosting.

I am looking for biological confirmation and datasets to proof if this works better. (It does slowdown a bit but that is an implementation detail).

• I'd say it's biologically possible that the signal gets corrupted while transfered over the dendrite.
• maybe this would help in wrong parameter configuration, for a well tuned system dropout indeed had low effect.

HTM keeps working even when some of its components are disabled

umm..no components are disabled permanently, this temporarily flips the bit, adding noise to the input.

[breznak]

Hotgym example internally uses dropout input.addNoise(), when testing, disable the explicit dropout there and use one from this PR (also test results w/o dropout)