merge master

README.md

@@ -11,10 +11,12 @@
 [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen)
 [![Version](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases) [![Join the chat at https://gitter.im/Microsoft/nni](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
+[简体中文](zh_CN/README.md)
+
 NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments. 
 The tool dispatches and runs trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in different environments like local machine, remote servers and cloud.
 
-### **NNI [v0.5](https://github.com/Microsoft/nni/releases) has been released!**
+### **NNI [v0.5.1](https://github.com/Microsoft/nni/releases) has been released!**
 <p align="center">
   <a href="#nni-v05-has-been-released"><img src="https://microsoft.github.io/nni/docs/img/overview.svg" /></a>
 </p>
@@ -49,33 +51,33 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search
         </ul>
       </td>
       <td>
-        <a href="docs/HowToChooseTuner.md">Tuner</a>
+        <a href="docs/Builtin_Tuner.md">Tuner</a>
         <ul>
-          <li><a href="docs/HowToChooseTuner.md#TPE">TPE</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Random">Random Search</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Anneal">Anneal</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Evolution">Naive Evolution</a></li>
-          <li><a href="docs/HowToChooseTuner.md#SMAC">SMAC</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Batch">Batch</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Grid">Grid Search</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Hyperband">Hyperband</a></li>
-          <li><a href="docs/HowToChooseTuner.md#NetworkMorphism">Network Morphism</a></li>
+          <li><a href="docs/Builtin_Tuner.md#TPE">TPE</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Random">Random Search</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Anneal">Anneal</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Evolution">Naive Evolution</a></li>
+          <li><a href="docs/Builtin_Tuner.md#SMAC">SMAC</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Batch">Batch</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Grid">Grid Search</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Hyperband">Hyperband</a></li>
+          <li><a href="docs/Builtin_Tuner.md#NetworkMorphism">Network Morphism</a></li>
           <li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li>
-          <li><a href="docs/HowToChooseTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
+          <li><a href="docs/Builtin_Tuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
         </ul> 
-          <a href="docs/HowToChooseTuner.md#assessor">Assessor</a> 
+          <a href="docs/Builtin_Tuner.md#assessor">Assessor</a> 
         <ul>
-          <li><a href="docs/HowToChooseTuner.md#Medianstop">Median Stop</a></li>
-          <li><a href="docs/HowToChooseTuner.md#Curvefitting">Curve Fitting</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Medianstop">Median Stop</a></li>
+          <li><a href="docs/Builtin_Tuner.md#Curvefitting">Curve Fitting</a></li>
         </ul>
       </td>
       <td>
       <ul>
         <li><a href="docs/tutorial_1_CR_exp_local_api.md">Local Machine</a></li>
-        <li><a href="docs/tutorial_2_RemoteMachineMode.md">Remote Servers</a></li>
+        <li><a href="docs/RemoteMachineMode.md">Remote Servers</a></li>
         <li><a href="docs/PAIMode.md">OpenPAI</a></li>
         <li><a href="docs/KubeflowMode.md">Kubeflow</a></li>
-        <li><a href="docs/KubeflowMode.md">FrameworkController on K8S (AKS etc.)</a></li>
+        <li><a href="docs/FrameworkControllerMode.md">FrameworkController on K8S (AKS etc.)</a></li>
       </ul>
       </td>
     </tr>
@@ -112,7 +114,7 @@ Note:
 * We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) in our current stage. 
 * Run the following commands in an environment that has `python >= 3.5`, `git` and `wget`.
 ```bash	
-    git clone -b v0.5 https://github.com/Microsoft/nni.git
+    git clone -b v0.5.1 https://github.com/Microsoft/nni.git
     cd nni	
     source install.sh	
 ```
@@ -124,7 +126,7 @@ For the system requirements of NNI, please refer to [Install NNI](docs/Installat
 The following example is an experiment built on TensorFlow. Make sure you have **TensorFlow installed** before running it.	
 * Download the examples via clone the source code.	
 ```bash	
-    git clone -b v0.5 https://github.com/Microsoft/nni.git
+    git clone -b v0.5.1 https://github.com/Microsoft/nni.git
 ```
 * Run the mnist example.
 ```bash
@@ -168,24 +170,25 @@ You can use these commands to get more information about the experiment
 
 ## **Documentation**
 * [NNI overview](docs/Overview.md)
-* [Quick start](docs/GetStarted.md)
+* [Quick start](docs/QuickStart.md)
 
 ## **How to**
 * [Install NNI](docs/Installation.md)
 * [Use command line tool nnictl](docs/NNICTLDOC.md)
 * [Use NNIBoard](docs/WebUI.md)
 * [How to define search space](docs/SearchSpaceSpec.md)
-* [How to define a trial](docs/howto_1_WriteTrial.md)
-* [How to choose tuner/search-algorithm](docs/HowToChooseTuner.md)
+* [How to define a trial](docs/Trials.md)
+* [How to choose tuner/search-algorithm](docs/Builtin_Tuner.md)
 * [Config an experiment](docs/ExperimentConfig.md)
-* [How to use annotation](docs/howto_1_WriteTrial.md#nni-python-annotation)
+* [How to use annotation](docs/Trials.md#nni-python-annotation)
 ## **Tutorials**
 * [Run an experiment on local (with multiple GPUs)?](docs/tutorial_1_CR_exp_local_api.md)
-* [Run an experiment on multiple machines?](docs/tutorial_2_RemoteMachineMode.md)
+* [Run an experiment on multiple machines?](docs/RemoteMachineMode.md)
 * [Run an experiment on OpenPAI?](docs/PAIMode.md)
 * [Run an experiment on Kubeflow?](docs/KubeflowMode.md)
-* [Try different tuners and assessors](docs/tutorial_3_tryTunersAndAssessors.md)
-* [Implement a customized tuner](docs/howto_2_CustomizedTuner.md)
+* [Try different tuners](docs/tuners.rst)
+* [Try different assessors](docs/assessors.rst)
+* [Implement a customized tuner](docs/Customize_Tuner.md)
 * [Implement a customized assessor](examples/assessors/README.md)
 * [Use Genetic Algorithm to find good model architectures for Reading Comprehension task](examples/trials/ga_squad/README.md)
 

azure-pipelines.yml

@@ -33,7 +33,7 @@ jobs:
     displayName: 'Built-in tuners / assessors tests'
   - script: |
       cd test
-      PATH=$HOME/.local/bin:$PATH python3 config_test.py --ts local
+      PATH=$HOME/.local/bin:$PATH python3 config_test.py --ts local --local_gpu
     displayName: 'Examples and advanced features tests on local machine'
   - script: |
       cd test

deployment/docker/README.md

@@ -1,18 +1,18 @@
 Dockerfile
 ===
 ## 1.Description
-This is the Dockerfile of nni project. It includes serveral popular deep learning frameworks and NNI. It is tested on `Ubuntu 16.04 LTS`:
+This is the Dockerfile of NNI project. It includes serveral popular deep learning frameworks and NNI. It is tested on `Ubuntu 16.04 LTS`:
 
 ```
 CUDA 9.0, CuDNN 7.0
 numpy 1.14.3,scipy 1.1.0
-TensorFlow 1.10.0
+TensorFlow-gpu 1.10.0
 Keras 2.1.6
 PyTorch 0.4.1
 scikit-learn 0.20.0
 pandas 0.23.4
 lightgbm 2.2.2
-NNI v0.5
+NNI v0.5.1
 ```
 You can take this Dockerfile as a reference for your own customized Dockerfile. 
 
@@ -26,6 +26,8 @@ __Run the docker image__
 ```
     docker run -it nni/nni
 ```
+Note that if you want to use tensorflow, please uninstall tensorflow-gpu and install tensorflow in this docker container. Or modify `Dockerfile` to install tensorflow (without gpu) and build docker image.
+
 * If use GPU in docker container, make sure you have installed [NVIDIA Container Runtime](https://github.com/NVIDIA/nvidia-docker), then run the following command
 ```
     nvidia-docker run -it nni/nni

docs/.gitignore

@@ -0,0 +1,3 @@
+_build
+_static
+_templates

docs/AdvancedNAS.md

@@ -8,6 +8,7 @@ Currently we recommend sharing weights through NFS (Network File System), which
 
 ### Weight Sharing through NFS file
 With the NFS setup (see below), trial code can share model weight through loading & saving files. Here we recommend that user feed the tuner with the storage path:
+
 ```yaml
 tuner:
   codeDir: path/to/customer_tuner
@@ -17,9 +18,10 @@ tuner:
     ...
     save_dir_root: /nfs/storage/path/
 ```
+
 And let tuner decide where to save & load weights and feed the paths to trials through `nni.get_next_parameters()`:
 
-![weight_sharing_design](./img/weight_sharing.png)
+<img src="https://user-images.githubusercontent.com/23273522/51817667-93ebf080-2306-11e9-8395-b18b322062bc.png" alt="drawing" width="700"/>
 
  For example, in tensorflow:
 ```python
@@ -80,10 +82,10 @@ The feature of weight sharing enables trials from different machines, in which m
 ```
 
 ## Examples
-For details, please refer to this [simple weight sharing example](../test/async_sharing_test). We also provided a [practice example](../examples/trials/weight_sharing/ga_squad) for reading comprehension, based on previous [ga_squad](../examples/trials/ga_squad) example.
+For details, please refer to this [simple weight sharing example](https://github.com/Microsoft/nni/tree/master/test/async_sharing_test). We also provided a [practice example](https://github.com/Microsoft/nni/tree/master/examples/trials/weight_sharing/ga_squad) for reading comprehension, based on previous [ga_squad](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad) example.
 
 [1]: https://arxiv.org/abs/1802.03268
 [2]: https://arxiv.org/abs/1707.07012
 [3]: https://arxiv.org/abs/1806.09055
 [4]: https://arxiv.org/abs/1806.10282
-[5]: https://arxiv.org/abs/1703.01041 
+[5]: https://arxiv.org/abs/1703.01041 

docs/AnnotationSpec.md

@@ -1,58 +1,70 @@
 # NNI Annotation 
 
-For good user experience and reduce user effort, we need to design a good annotation grammar.
 
-If users use NNI system, they only need to:
+## Overview
 
- 1. Use nni.get_next_parameter() to retrieve hyper parameters from Tuner, before using other annotation, use following annotation at the begining of trial code:
-    '''@nni.get_next_parameter()'''
+To improve user experience and reduce user effort, we design an annotation grammar. Using NNI annotation, users can adapt their code to NNI just by adding some standalone annotating strings, which does not affect the execution of the original code. 
 
- 2. Annotation variable in code as:
+Below is an example:
 
-    '''@nni.variable(nni.choice(2,3,5,7),name=self.conv_size)'''
+```python
+'''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+learning_rate = 0.1
+```
+The meaning of this example is that NNI will choose one of several values (0.1, 0.01, 0.001) to assign to the learning_rate variable. Specifically, this first line is an NNI annotation, which is a single string. Following is an assignment statement. What nni does here is to replace the right value of this assignment statement according to the information provided by the annotation line.
 
- 3. Annotation intermediate in code as:
 
-    '''@nni.report_intermediate_result(test_acc)'''
+In this way, users could either run the python code directly or launch NNI to tune hyper-parameter in this code, without changing any codes.
 
- 4. Annotation output in code as:
+## Types of Annotation:
 
-    '''@nni.report_final_result(test_acc)'''
+In NNI, there are mainly four types of annotation:
 
- 5. Annotation `function_choice` in code as:
 
-    '''@nni.function_choice(max_pool(h_conv1, self.pool_size),avg_pool(h_conv1, self.pool_size),name=max_pool)'''
+### 1. Annotate variables
 
-In this way, they can easily implement automatic tuning on NNI. 
+   `'''@nni.variable(sampling_algo, name)'''`
 
-For `@nni.variable`, `nni.choice` is the type of search space and there are 10 types to express your search space as follows:
+`@nni.variable` is used in NNI to annotate a variable.
 
- 1. `@nni.variable(nni.choice(option1,option2,...,optionN),name=variable)`  
-    Which means the variable value is one of the options, which should be a list The elements of options can themselves be stochastic expressions
+**Arguments**
 
- 2. `@nni.variable(nni.randint(upper),name=variable)`  
-    Which means the variable value is a random integer in the range [0, upper).
+- **sampling_algo**: Sampling algorithm that specifies a search space. User should replace it with a built-in NNI sampling function whose name consists of an `nni.` identification and a search space type specified in [SearchSpaceSpec](SearchSpaceSpec.md) such as `choice` or `uniform`. 
+- **name**: The name of the variable that the selected value will be assigned to. Note that this argument should be the same as the left value of the following assignment statement.
 
- 3. `@nni.variable(nni.uniform(low, high),name=variable)`  
-    Which means the variable value is a value uniformly between low and high.
+An example here is:
 
- 4. `@nni.variable(nni.quniform(low, high, q),name=variable)`  
-    Which means the variable value is a value like round(uniform(low, high) / q) * q
+```python
+'''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+learning_rate = 0.1
+```
 
- 5. `@nni.variable(nni.loguniform(low, high),name=variable)`  
-    Which means the variable value is a value drawn according to exp(uniform(low, high)) so that the logarithm of the return value is uniformly distributed.
+### 2. Annotate functions
 
- 6. `@nni.variable(nni.qloguniform(low, high, q),name=variable)`  
-    Which means the variable value is a value like round(exp(uniform(low, high)) / q) * q
+   `'''@nni.function_choice(*functions, name)'''`
 
- 7. `@nni.variable(nni.normal(label, mu, sigma),name=variable)`  
-    Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma.
+`@nni.function_choice` is used to choose one from several functions.
 
- 8. `@nni.variable(nni.qnormal(label, mu, sigma, q),name=variable)`  
-    Which means the variable value is a value like round(normal(mu, sigma) / q) * q
+**Arguments**
 
- 9. `@nni.variable(nni.lognormal(label, mu, sigma),name=variable)`  
-    Which means the variable value is a value drawn according to exp(normal(mu, sigma))
+- **\*functions**: Several functions that are waiting to be selected from. Note that it should be a complete function call with arguments. Such as `max_pool(hidden_layer, pool_size)`.
+- **name**: The name of the function that will be replaced in the following assignment statement.
 
-10. `@nni.variable(nni.qlognormal(label, mu, sigma, q),name=variable)`  
-    Which means the variable value is a value like round(exp(normal(mu, sigma)) / q) * q
+An example here is:
+
+```python
+"""@nni.function_choice(max_pool(hidden_layer, pool_size), avg_pool(hidden_layer, pool_size), name=max_pool)"""
+h_pooling = max_pool(hidden_layer, pool_size)
+```
+
+### 3. Annotate intermediate result
+
+   `'''@nni.report_intermediate_result(metrics)'''`
+
+`@nni.report_intermediate_result` is used to report intermediate result, whose usage is the same as `nni.report_intermediate_result` in [Trials.md](Trials.md)
+
+### 4. Annotate final result
+
+   `'''@nni.report_final_result(metrics)'''`
+
+`@nni.report_final_result` is used to report the final result of the current trial, whose usage is the same as `nni.report_final_result` in [Trials.md](Trials.md)

docs/Builtin_Assessors.md

@@ -0,0 +1,72 @@
+# Built-in Assessors
+
+NNI provides state-of-the-art tuning algorithm in our builtin-assessors and makes them easy to use. Below is the brief overview of NNI current builtin Assessors:
+
+|Assessor|Brief Introduction of Algorithm|
+|---|---|
+|**Medianstop** [(Usage)](#MedianStop)|Medianstop is a simple early stopping rule mentioned in the [paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf). It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.|
+|[Curvefitting](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/curvefitting_assessor/README.md) [(Usage)](#Curvefitting)|Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve|
+
+## Usage of Builtin Assessors
+
+Use builtin assessors provided by NNI SDK requires to declare the  **builtinAssessorName** and **classArgs** in `config.yml` file. In this part, we will introduce the detailed usage about the suggested scenarios, classArg requirements, and example for each assessor.
+
+Note: Please follow the format when you write your `config.yml` file.
+
+<a name="MedianStop"></a>
+
+![](https://placehold.it/15/1589F0/000000?text=+) `Median Stop Assessor`
+
+> Builtin Assessor Name: **Medianstop**
+
+**Suggested scenario**
+
+It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
+
+**Requirement of classArg**
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', assessor will **stop** the trial with smaller expectation. If 'minimize', assessor will **stop** the trial with larger expectation.
+* **start_step** (*int, optional, default = 0*) - A trial is determined to be stopped or not, only after receiving start_step number of reported intermediate results.
+
+**Usage example:**
+
+```yaml
+# config.yml
+assessor:
+    builtinAssessorName: Medianstop
+    classArgs:
+      optimize_mode: maximize
+      start_step: 5
+```
+
+<br>
+
+<a name="Curvefitting"></a>
+
+![](https://placehold.it/15/1589F0/000000?text=+) `Curve Fitting Assessor`
+
+> Builtin Assessor Name: **Curvefitting**
+
+**Suggested scenario**
+
+It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance.
+
+**Requirement of classArg**
+
+* **epoch_num** (*int, **required***) - The total number of epoch. We need to know the number of epoch to determine which point we need to predict.
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', assessor will **stop** the trial with smaller expectation. If 'minimize', assessor will **stop** the trial with larger expectation.
+* **start_step** (*int, optional, default = 6*) - A trial is determined to be stopped or not, we start to predict only after receiving start_step number of reported intermediate results.
+* **threshold** (*float, optional, default = 0.95*) - The threshold that we decide to early stop the worse performance curve. For example: if threshold = 0.95, optimize_mode = maximize, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
+
+**Usage example:**
+
+```yaml
+# config.yml
+assessor:
+    builtinAssessorName: Curvefitting
+    classArgs:
+      epoch_num: 20
+      optimize_mode: maximize
+      start_step: 6
+      threshold: 0.95
+```