Parameter optimization improve documentation

[breznak]

• which label to use for "parameter optimization"? "optimization" is taken by "CPU performance optimizations", NuPIC used "swarming" which is more specific than necessary, but already used in the community to mean "parameter optimization" (also shorter)
• improve docs how to actually run the optimization
• make opt. run in CI as a default example (some short swarm)
• there's a "bug" in the way we describe and run the swarming now! As we're faking. We are testing the results on the test set, but that is a mete-parameter and our behavior leads to overfitting. We should use
  • cross-validation
  • train/eval/test split (where test is out-of-sample and never touched)
• kill optimization when plateauing? (changes start to be too insignificant)
• implement other methods: EA/GA (genetic algorithms), simulated-annealing, but rather I'd see a 3rd party framework, a proper interface, and this project moved to a separate htm-community repo.

[breznak]

FYI @ctrl-z-9000-times

[breznak]

Related:
#536 Smarter (SP) params

#477 3rd party parameter-optimization framework

[breznak]

I'm currently playing with optimization, it works nice but it does have its space for improvement.
What do you think of

• make it in a separate repo to make it usable by whole community?
• what was the problem with NNI Hyper-Parameter Optimization Framework - WIP #477 ? I'll try to update it, would you think NNI is still superior to the current custom solution?

[ctrl-z-9000-times]

make it in a separate repo to make it usable by whole community?

I'd rather keep it here since the experiments here use it.

make opt. run in CI as a default example (some short swarm)

I think it would take too long to run anything interesting.

there's a "bug" in the way we describe and run the swarming now! As we're faking. We are testing the results on the test set, but that is a mete-parameter and our behavior leads to overfitting. We should use

• cross-validation
• train/eval/test split (where test is out-of-sample and never touched)

The parameter optimization framework does not have this bug, the program being optimized has this bug. The optimization framework gives the experiment it's parameters, and accepts in return a score. It assumes that the experiment is splitting the dataset into train/eval/test.

implement other methods

Totally doable. The framework has an interface for adding new methods.

what was the problem with NNI #477 ? I'll try to update it, would you think NNI is still superior to the current custom solution?

I found NNI to be complicated. Its tailored for large & long running deep learning networks. We can use it if you can figure out how to make the HTM-based experiments interface with it.

[Thanh-Binh]

Hi all,
currently we have python version for parameter optimizations.
I want to use it for optimizing my framework in C++ but I do not know

1. how to call my framework from python for running it multiple times?
2. or how can we rewrite optimization framework of @ctrl-z-9000-times in pure C++?
   Any idea and hint? Thanks

[breznak]

I want to use it for optimizing my framework in C++ but I do not know

the current way is to optimize parameters in python (all code has a py wrapper/equivalent). Find the best params, then apply them to your c++ only code.

If you have a custom c++ code, you're out of luck and need to do one of:

• write a py wrapper for such code to be used by the param opt framework.
• call c++ program from py (and fix all the requirements of the framework: main method,...)
• rewrite the framework to c++ (but I think that's useless, you'd still need the point above then)
• if you have some interesting code that'd fit this repo, you can try publishing it and having it included, then we can help with the py integration

[Thanh-Binh]

@breznak writing a python wrapper for such codes, which have to be optimized, is not an intelligent way. I think, the optimization procedure, independent on which programming languages it is written, should work like that:

1. given a set of parameters
2. set those parameters into your framework, which will be optimized
3. run your framework with those parameters and assign the results as a score
4. if score is smaller than a threshold, then stop optimization process
   5.else change those parameters somehows, then goto step 1

I think, we can do it well in c++.
What do you think?

[breznak]

writing a python wrapper for such codes, which have to be optimized, is not an intelligent way.

I think python is a pretty good choice for the optimization framework. This repo does C++ & Python, and we have a Py wrapper for all of the c++ code; py is a good prototyping language; param optimization is not a main goal of this repo, so the code is "just" a helper/side project; and the code is rather lean&clean written in py and is flexible to change. And most of all, in the parameter optimization, we don't (primarily) care about speed, but results (which can then be used for a c++ implementation).

So the problem is really just for 3rd party code that is written in c++ only.

optimization procedure, independent on which programming languages

you're right that a generic method for setting up classes and theirs params and evaluating would be nice. Your description is correct but on a very high-level. The devil is in the details:

• step 2: how would you do it?
  • we're discussing with @dkeeney a new way for Network to set params from a single JSON file config
  • I've long wanted the Algorithm classes to have an alternative JSON constructor
• step 5: is where the logic is, and it might be complicated to write in c++. The current uses PSO, GridSearch, and Manual methods.

I think, we can do it well in c++.

I'm not sure it's worth it given the existing solutions, but I'll be happy to review PRs or advise in the process.

[Thanh-Binh]

@breznak thank for your explain! I am sure that python is good for automation. But I really do not understand why we need it for optimizing because writing python wrapper is unnecessary double works for c++ user!

[breznak]

because writing python wrapper is unnecessary double works for c++ user!

it would be the same the other way around. If we had optimization in c++, someone would have to write a wrapper/bindings for python.

The explanation why is simple, someone (David) wrote it in Py, and it was imho the better language to choose for a prototyping task.