Dev networkmorphism (#413)

* Quick fix nnictl config logic (#289) * fix nnictl bug * fix install.sh * add desc for Dockerfile.build.base * update document for Dockerfile * update * refactor port detect * update * refactor NNICTLDOC.md * add document for pai and nnictl * add default value for port * add exception handling in trial_keeper.py * fix port bug * fix resume * fix nnictl resume and fix nnictl stop * fix document * update * refactor nnictl * update * update doc * update * update nnictl * fix comment * revert dockerfile * update * update * update * fix nnictl error hit * fix comments * fix bash-completion * fix paramiko install * quick fix resume logic * update * quick fix nnictl * PR merge to 0.3 (#297) * refactor doc * update with Mao's suggestions * Set theme jekyll-theme-dinky * update doc * fix links * fix links * fix links * merge * fix links and doc errors * merge * merge * merge * merge * Update README.md (#288) added License badge * merge * updated the "Contribute" part (merged Gems' wiki in, updated ReadMe) * fix link * fix doc mistakes and broken links. (#271) * refactor doc * update with Mao's suggestions * Set theme jekyll-theme-dinky * updated the "Contribute" part (merged Gems' wiki in, updated ReadMe) * fix link * Update README.md * Fix misspelling in examples/trials/ga_squad/README.md * revise the installation cmd to v0.2 * revise to install v0.2 * remove enas readme (#292) * Fix datastore performance issue (#301) * Fix nnictl in v0.3 (#299) Fix old version of config file fix sklearn requirements Fix resume log logic * add basic tuner and trial for network morphism * Complete basic receive_trial_result() and generate_parameters(). Use onnx as the intermediate representation ( But it cannot convert to pytorch model ) * add tensorflow cifar10 for network morphism * add unit test for tuner and its function * use temporary torch_model * fix request bug and program can communicate nni * add basic pickle support for graph and train successful in pytorch * Update unittest for networkmorphism_tuner * Network Morphism add multi-gpu trial training support * Format code with black tool * change intermediate representation from pickle file to json we defined * successfully pass the unittest for test_graph_json_transform * add README for network morphism and it works fine in both Pytorch and Keras. * separate the original Readme.md in network-morphism into two parts (tuner and trial) * change the openpai image path * beautify the file structure of network_morphism and add a fashion_mnist keras example * pretty the source and add some docstring for funtion in order to pass the pylint. * remove unused module import and add some docstring * add some details for the application scenario Network Morphism Tuner * follow the advice and modify the doc file * add the config file for each task in the examples trial of network morphism * change default python interpreter from python to python3
microsoft · Dec 13, 2018 · e31839c · e31839c
1 parent 0425766
commit e31839c
Show file tree

Hide file tree

Showing 38 changed files with 5,155 additions and 148 deletions.
diff --git a/.gitignore b/.gitignore
@@ -66,4 +66,4 @@ typings/
 # Python cache files
 __pycache__
 build
-*.egg-info
+*.egg-info
diff --git a/docs/HowToChooseTuner.md b/docs/HowToChooseTuner.md
@@ -10,6 +10,7 @@ For now, NNI has supported the following tuner algorithms. Note that NNI install
  - [Batch Tuner](#Batch)
  - [Grid Search](#Grid)
  - [Hyperband](#Hyperband)
+ - [Network Morphism](#NetworkMorphism)
 
  ## Supported tuner algorithms
 
@@ -22,11 +23,11 @@ We will introduce some basic knowledge about the tuner algorithms, suggested sce
 The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.
 The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. 
 This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization][1].
-
+    
 _Suggested scenario_: TPE, as a black-box optimization, can be used in various scenarios, and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: TPE
@@ -43,7 +44,7 @@ In [Random Search for Hyper-Parameter Optimization][2] show that Random Search m
 _Suggested scenario_: Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Random
@@ -57,7 +58,7 @@ This simple annealing algorithm begins by sampling from the prior, but tends ove
 _Suggested scenario_: Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Anneal
@@ -74,7 +75,7 @@ Naive Evolution comes from [Large-Scale Evolution of Image Classifiers][3]. It r
 _Suggested scenario_: Its requirement of computation resource is relatively high. Specifically, it requires large inital population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: Evolution
@@ -93,7 +94,7 @@ Note that SMAC on nni only supports a subset of the types in [search space spec]
 _Suggested scenario_: Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: SMAC
@@ -110,14 +111,14 @@ Batch tuner allows users to simply provide several configurations (i.e., choices
 _Suggested sceanrio_: If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: BatchTuner
 ```
 
 Note that the search space that BatchTuner supported like:
-```
+```json
 {
     "combine_params":
     {
@@ -133,7 +134,6 @@ Note that the search space that BatchTuner supported like:
 ```
 The search space file including the high-level key `combine_params`. The type of params in search space must be `choice` and the `values` including all the combined-params value.
 
-
 <a name="Grid"></a>
 **Grid Search**
 
@@ -143,7 +143,7 @@ Note that the only acceptable types of search space are `choice`, `quniform`, `q
 _Suggested scenario_: It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   tuner:
     builtinTunerName: GridSearch
@@ -152,12 +152,12 @@ _Usage_:
 <a name="Hyperband"></a>
 **Hyperband**
 
-[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be refered to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md)
+[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be referred to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md).
 
 _Suggested scenario_: It is suggested when you have limited computation resource but have relatively large search space. It performs good in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent.
 
 _Usage_:
-```
+```yaml
   # config.yaml
   advisor:
     builtinAdvisorName: Hyperband
@@ -171,6 +171,31 @@ _Usage_:
       eta: 3
 ```
 
+<a name="NetworkMorphism"></a>
+**Network Morphism**
+
+[Network Morphism](7) provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md). 
+
+_Suggested scenario_: It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](../examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the cv domain.
+
+_Usage_:
+```yaml
+  # config.yaml
+  tuner:
+    builtinTunerName: NetworkMorphism
+    classArgs:
+      #choice: maximize, minimize
+      optimize_mode: maximize
+      #for now, this tuner only supports cv domain
+      task: cv
+      #input image width
+      input_width: 32
+      #input image channel
+      input_channel: 3
+      #number of classes
+      n_output_node: 10
+```
+
 
 # How to use Assessor that NNI supports?
 
@@ -184,12 +209,12 @@ For now, NNI has supported the following assessor algorithms.
 <a name="Medianstop"></a>
 **Medianstop**
 
-Medianstop is a simple early stopping rule mentioned in the [paper][7]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
+Medianstop is a simple early stopping rule mentioned in the [paper][8]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
 
 _Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
 
 _Usage_:
-```
+```yaml
   assessor:
     builtinAssessorName: Medianstop
     classArgs:
@@ -201,10 +226,11 @@ _Usage_:
       start_step: 5
 ```
 
-  [1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
-  [2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
-  [3]: https://arxiv.org/pdf/1703.01041.pdf
-  [4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
-  [5]: https://github.com/automl/SMAC3
-  [6]: https://arxiv.org/pdf/1603.06560.pdf
-  [7]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
+[1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
+[2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
+[3]: https://arxiv.org/pdf/1703.01041.pdf
+[4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
+[5]: https://github.com/automl/SMAC3
+[6]: https://arxiv.org/pdf/1603.06560.pdf
+[7]: https://arxiv.org/abs/1806.10282
+[8]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf