Skip to content

Commit

Permalink
Merge pull request #90 from microsoft/master
Browse files Browse the repository at this point in the history 
pull code
  • Loading branch information
@chicm-ms
chicm-ms authored Jun 15, 2020
2 parents 7f4cdcd + 813ab0f commit 574db2c
Show file tree
Hide file tree
Showing 82 changed files with 2,237 additions and 1,622 deletions.
46 changes: 18 additions & 28 deletions .github/ISSUE_TEMPLATE/bug-report.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,25 @@ about: Report an issue or question while using nni instance (deployment).

---

<!-- Please use this template while reporting an issue and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->
**Environment**:
- NNI version:
- NNI mode (local|remote|pai):
- Client OS:
- Server OS (for remote mode only):
- Python version:
- PyTorch/TensorFlow version:
- Is conda/virtualenv/venv used?:
- Is running in Docker?:

**Log message**:
- nnimanager.log:
- dispatcher.log:
- nnictl stdout and stderr:

<!-- Where can you find the log files: [log](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/HowToDebug.md#experiment-root-director), [stdout/stderr](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/Nnictl.md#nnictl%20log%20stdout) -->

**Short summary about the issue/question**:

**Brief what process you are following**:

<!--deployment related issues
Please fill this for deployment related issues:
- Operating type: Initial deployment / upgrading / operating etc.
- Brief what deployment process you are following -->

**How to reproduce it**:

<!--Fill the following information if your issue need diagnostic support from the team, as minimally and precisely as possible!-->

**nni Environment**:
- nni version:
- nni mode(local|pai|remote):
- OS:
- python version:
- is conda or virtualenv used?:
- is running in docker?:

**need to update document(yes/no)**:

**Anything else we need to know**:
**What issue meet, what's expected?**:

**Log message**:
- [nnimanager.log and dispatcher.log](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/HowToDebug.md#experiment-root-directory) :
**How to reproduce it?**:

- [nnictl stdout and stderr](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/Nnictl.md#nnictl%20log%20stdout) :
**Additional information**:
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ For detail system requirements of NNI, please refer to [here](https://nni.readth
Note:

* If there is any privilege issue, add `--user` to install NNI in the user directory.
* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install NNI on Windows.
* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](docs/en_US/Tutorial/InstallationWin.md).
* If there is any error like `Segmentation fault`, please refer to [FAQ](docs/en_US/Tutorial/FAQ.md). For FAQ on Windows, please refer to [NNI on Windows](docs/en_US/Tutorial/InstallationWin.md#faq).

### **Verify installation**
Expand Down Expand Up @@ -341,7 +341,7 @@ With authors' permission, we listed a set of NNI usage examples and relevant art
Join IM discussion groups:
|Gitter||WeChat|
|----|----|----|
|<img src="https://user-images.githubusercontent.com/39592018/80665738-e0574a80-8acc-11ea-91bc-0836dc4cbf89.png" width="180"/>| OR |<img src="https://user-images.githubusercontent.com/39592018/83108240-113d9600-a0f2-11ea-91f8-8754af11a0ee.png" width="180"/>|
|![image](https://user-images.githubusercontent.com/39592018/80665738-e0574a80-8acc-11ea-91bc-0836dc4cbf89.png)| OR |![image](https://github.com/scarlett2018/nniutil/raw/master/wechat.png)|


## Related Projects
Expand Down
11 changes: 6 additions & 5 deletions docs/en_US/Assessor/BuiltinAssessor.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,15 @@ assessor:

It's applicable in a wide range of performance curves, thus, it can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance. [Detailed Description](./CurvefittingAssessor.md)

**Note**, according to the original paper, only incremental functions are supported. Therefore this assessor can only be used to maximize optimization metrics. For example, it can be used for accuracy, but not for loss.


**classArgs requirements:**

* **epoch_num** (*int, **required***) - The total number of epochs. We need to know the number of epochs to determine which points we need to predict.
* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', assessor will **stop** the trial with smaller expectation. If 'minimize', assessor will **stop** the trial with larger expectation.
* **start_step** (*int, optional, default = 6*) - A trial is determined to be stopped or not only after receiving start_step number of reported intermediate results.
* **threshold** (*float, optional, default = 0.95*) - The threshold that we use to decide to early stop the worst performance curve. For example: if threshold = 0.95, optimize_mode = maximize, and the best performance in the history is 0.9, then we will stop the trial who's predicted value is lower than 0.95 * 0.9 = 0.855.
* **gap** (*int, optional, default = 1*) - The gap interval between Assesor judgements. For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermediate results.
* **threshold** (*float, optional, default = 0.95*) - The threshold that we use to decide to early stop the worst performance curve. For example: if threshold = 0.95, and the best performance in the history is 0.9, then we will stop the trial who's predicted value is lower than 0.95 * 0.9 = 0.855.
* **gap** (*int, optional, default = 1*) - The gap interval between Assessor judgements. For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermediate results.

**Usage example:**

Expand All @@ -71,8 +73,7 @@ assessor:
builtinAssessorName: Curvefitting
classArgs:
epoch_num: 20
optimize_mode: maximize
start_step: 6
threshold: 0.95
gap: 1
```
```
66 changes: 34 additions & 32 deletions docs/en_US/Assessor/CurvefittingAssessor.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
Curve Fitting Assessor on NNI
===
# Curve Fitting Assessor on NNI

## Introduction

## 1. Introduction
The Curve Fitting Assessor is an LPA (learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of the final epoch's performance is worse than the best final performance in the trial history.

In this algorithm, we use 12 curves to fit the learning curve. The set of parametric curve models are chosen from this [reference paper][1]. The learning curves' shape coincides with our prior knowledge about the form of learning curves: They are typically increasing, saturating functions.

![](../../img/curvefitting_learning_curve.PNG)
![learning_curve](../../img/curvefitting_learning_curve.PNG)

We combine all learning curve models into a single, more powerful model. This combined model is given by a weighted linear combination:

![](../../img/curvefitting_f_comb.gif)
![f_comb](../../img/curvefitting_f_comb.gif)

with the new combined parameter vector

![](../../img/curvefitting_expression_xi.gif)
![expression_xi](../../img/curvefitting_expression_xi.gif)

Assuming additive Gaussian noise and the noise parameter being initialized to its maximum likelihood estimate.

Expand All @@ -30,44 +30,46 @@ Concretely, this algorithm goes through three stages of learning, predicting, an

The figure below is the result of our algorithm on MNIST trial history data, where the green point represents the data obtained by Assessor, the blue point represents the future but unknown data, and the red line is the Curve predicted by the Curve fitting assessor.

![](../../img/curvefitting_example.PNG)
![examples](../../img/curvefitting_example.PNG)

## Usage

## 2. Usage
To use Curve Fitting Assessor, you should add the following spec in your experiment's YAML config file:

```
```yaml
assessor:
builtinAssessorName: Curvefitting
classArgs:
# (required)The total number of epoch.
# We need to know the number of epoch to determine which point we need to predict.
epoch_num: 20
# (optional) choice: maximize, minimize
* The default value of optimize_mode is maximize
optimize_mode: maximize
# (optional) In order to save our computing resource, we start to predict when we have more than only after receiving start_step number of reported intermediate results.
* The default value of start_step is 6.
start_step: 6
# (optional) The threshold that we decide to early stop the worse performance curve.
# For example: if threshold = 0.95, optimize_mode = maximize, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
* The default value of threshold is 0.95.
# Kindly reminds that if you choose minimize mode, please adjust the value of threshold >= 1.0 (e.g threshold=1.1)
threshold: 0.95
# (optional) The gap interval between Assesor judgements.
# For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermedian result.
* The default value of gap is 1.
gap: 1
builtinAssessorName: Curvefitting
classArgs:
# (required)The total number of epoch.
# We need to know the number of epoch to determine which point we need to predict.
epoch_num: 20
# (optional) In order to save our computing resource, we start to predict when we have more than only after receiving start_step number of reported intermediate results.
# The default value of start_step is 6.
start_step: 6
# (optional) The threshold that we decide to early stop the worse performance curve.
# For example: if threshold = 0.95, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
# The default value of threshold is 0.95.
threshold: 0.95
# (optional) The gap interval between Assesor judgements.
# For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermedian result.
# The default value of gap is 1.
gap: 1
```
## 3. File Structure
## Limitation
According to the original paper, only incremental functions are supported. Therefore this assessor can only be used to maximize optimization metrics. For example, it can be used for accuracy, but not for loss.
## File Structure
The assessor has a lot of different files, functions, and classes. Here we briefly describe a few of them.
* `curvefunctions.py` includes all the function expressions and default parameters.
* `modelfactory.py` includes learning and predicting; the corresponding calculation part is also implemented here.
* `curvefitting_assessor.py` is the assessor which receives the trial history and assess whether to early stop the trial.

## 4. TODO
* Further improve the accuracy of the prediction and test it on more models.
## TODO

* Further improve the accuracy of the prediction and test it on more models.

[1]: http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf
Loading

0 comments on commit 574db2c

Please sign in to comment.