diff --git a/README.md b/README.md index 9ee9c9a542..9b3edee47b 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ [![Bugs](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug) [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen) [![Version](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases) [![Join the chat at https://gitter.im/Microsoft/nni](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=latest)](https://nni.readthedocs.io/en/latest/?badge=latest) [简体中文](README_zh_CN.md) diff --git a/README_zh_CN.md b/README_zh_CN.md index cb8aa70fb2..dd6b3b0d52 100644 --- a/README_zh_CN.md +++ b/README_zh_CN.md @@ -186,7 +186,7 @@ NNI (Neural Network Intelligence) 是自动机器学习(AutoML)的工具包 * [使用命令行工具 nnictl](docs/zh_CN/NNICTLDOC.md) * [使用 NNIBoard](docs/zh_CN/WebUI.md) * [如何定义搜索空间](docs/zh_CN/SearchSpaceSpec.md) -* [如何定义一次 Trial](docs/zh_CN/Trials.md) +* [如何编写 Trial 代码](docs/zh_CN/Trials.md) * [如何选择 Tuner、搜索算法](docs/zh_CN/Builtin_Tuner.md) * [配置 Experiment](docs/zh_CN/ExperimentConfig.md) * [如何使用 Annotation](docs/zh_CN/Trials.md#nni-python-annotation) diff --git a/deployment/deployment-pipelines.yml b/deployment/deployment-pipelines.yml index 727749342e..4a6abfa679 100644 --- a/deployment/deployment-pipelines.yml +++ b/deployment/deployment-pipelines.yml @@ -26,7 +26,7 @@ jobs: echo 'building release package...' make build fi - + condition: eq( variables['upload_package'], 'true' ) displayName: 'build nni bdsit_wheel' - script: | cd deployment/pypi @@ -38,21 +38,36 @@ jobs: echo 'uploading release package to pypi...' python3 -m twine upload -u $(pypi_user) -p $(pypi_pwd) dist/* fi + condition: eq( variables['upload_package'], 'true' ) displayName: 'upload nni package to pypi/testpypi' - script: | - export IMG_TAG=`git describe --tags --abbrev=0` cd deployment/docker - docker login -u $(docker_hub_user) -p $(docker_hub_pwd) if [ $(build_type) = 'prerelease' ] then + docker login -u $(docker_hub_dev_user) -p $(docker_hub_dev_pwd) + export IMG_NAME=$(dev_docker_img) + export IMG_TAG=`git describe --tags --abbrev=0`.`date +%y%m%d%H%M` echo 'updating docker file for testpyi...' - sed -ie 's/RUN python3 -m pip --no-cache-dir install nni/RUN python3 -m pip install --user --no-cache-dir --index-url https:\/\/test.pypi.org\/simple\/ --extra-index-url https:\/\/pypi.org\/simple nni/' Dockerfile + sed -ie 's/RUN python3 -m pip --no-cache-dir install nni/RUN python3 -m pip install --user --no-cache-dir --index-url https:\/\/test.pypi.org\/simple --extra-index-url https:\/\/pypi.org\/simple nni/' Dockerfile + else + docker login -u $(docker_hub_user) -p $(docker_hub_pwd) + export IMG_NAME=msranni/nni + export IMG_TAG=`git describe --tags --abbrev=0` fi - echo $IMG_TAG - docker build -f Dockerfile -t msranni/nni:$IMG_TAG . - docker push msranni/nni:$IMG_TAG + echo $IMG_NAME:$IMG_TAG + cat Dockerfile + docker build -f Dockerfile -t $IMG_NAME:$IMG_TAG . + docker push $IMG_NAME:$IMG_TAG + + if [ $(update_latest_tag) = 'true' ] + then + docker tag $IMG_NAME:$IMG_TAG $IMG_NAME:latest + docker push $IMG_NAME:latest + fi + + condition: eq( variables['build_docker_img'], 'true' ) displayName: 'build and upload nni docker image' - job: 'Build_upload_nni_macos' @@ -83,7 +98,7 @@ jobs: echo 'building release package...' PATH=$HOME/Library/Python/3.7/bin:$PATH make build fi - + condition: eq( variables['upload_package'], 'true' ) displayName: 'build nni bdsit_wheel' - script: | cd deployment/pypi @@ -95,4 +110,5 @@ jobs: echo 'uploading release package to pypi...' python3 -m twine upload -u $(pypi_user) -p $(pypi_pwd) dist/* fi + condition: eq( variables['upload_package'], 'true' ) displayName: 'upload nni package to pypi/testpypi' diff --git a/docs/en_US/RELEASE.md b/docs/en_US/RELEASE.md index 3af83d4ee5..63ea82ba55 100644 --- a/docs/en_US/RELEASE.md +++ b/docs/en_US/RELEASE.md @@ -2,8 +2,8 @@ ## Release 0.5.1 - 1/31/2018 ### Improvements -* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md) configurable -* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md), making it easier for debugging +* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md) configurable +* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md), making it easier for debugging ### Documentation * Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/ @@ -20,9 +20,9 @@ #### New tuner and assessor supports -* Support [Metis tuner](./Builtin_Tuner.md#MetisTuner) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning. +* Support [Metis tuner](metisTuner.md) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning. * Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS. -* Support [Curve fitting assessor](./Builtin_Tuner.md#Curvefitting) for early stop policy using learning curve extrapolation. +* Support [Curve fitting assessor](curvefittingAssessor.md) for early stop policy using learning curve extrapolation. * Advanced Support of [Weight Sharing](./AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS. #### Training Service Enhancement @@ -45,7 +45,7 @@ #### New tuner supports -* Support [network morphism](./Builtin_Tuner.md#NetworkMorphism) as a new tuner +* Support [network morphism](networkmorphismTuner.md) as a new tuner #### Training Service improvements @@ -79,8 +79,8 @@ * [Kubeflow Training service](./KubeflowMode.md) * Support tf-operator * [Distributed trial example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow -* [Grid search tuner](Builtin_Tuner.md#GridSearch) -* [Hyperband tuner](Builtin_Tuner.md#Hyperband) +* [Grid search tuner](gridsearchTuner.md) +* [Hyperband tuner](hyperbandAdvisor.md) * Support launch NNI experiment on MAC * WebUI * UI support for hyperband tuner @@ -163,7 +163,7 @@ * Support [OpenPAI](https://github.com/Microsoft/pai) Training Platform (See [here](./PAIMode.md) for instructions about how to submit NNI job in pai mode) * Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster * NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking -* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](Builtin_Tuner.md) for instructions about how to use SMAC tuner) +* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](smacTuner.md) for instructions about how to use SMAC tuner) * [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3) * Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment * Others diff --git a/docs/en_US/Tutorials.rst b/docs/en_US/Tutorials.rst index 48f6a8054b..c2794fc5c1 100644 --- a/docs/en_US/Tutorials.rst +++ b/docs/en_US/Tutorials.rst @@ -3,6 +3,8 @@ Tutorials ###################### .. toctree:: + :maxdepth: 2 + Installation Write Trial Tuners diff --git a/docs/en_US/curvefittingAssessor.md b/docs/en_US/curvefittingAssessor.md index 2be3bc2bcb..e054d15b15 100644 --- a/docs/en_US/curvefittingAssessor.md +++ b/docs/en_US/curvefittingAssessor.md @@ -6,18 +6,16 @@ Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It s In this algorithm, we use 12 curves to fit the learning curve, the large set of parametric curve models are chosen from [reference paper][1]. The learning curves' shape coincides with our prior knowlwdge about the form of learning curves: They are typically increasing, saturating functions. -

-drawing -

+![](../img/curvefitting_learning_curve.PNG) We combine all learning curve models into a single, more powerful model. This combined model is given by a weighted linear combination: -

-drawing -

+ +![](../img/curvefitting_f_comb.gif) + where the new combined parameter vector -

-drawing -

+ +![](../img/curvefitting_expression_xi.gif) + Assuming additive a Gaussian noise and the noise parameter is initialized to its maximum likelihood estimate. We determine the maximum probability value of the new combined parameter vector by learing the historical data. Use such value to predict the future trial performance, and stop the inadequate experiments to save computing resource. @@ -32,9 +30,7 @@ Concretely,this algorithm goes through three stages of learning, predicting and The figure below is the result of our algorithm on MNIST trial history data, where the green point represents the data obtained by Assessor, the blue point represents the future but unknown data, and the red line is the Curve predicted by the Curve fitting assessor. -

-drawing -

+![](../img/curvefitting_example.PNG) ## 2. Usage To use Curve Fitting Assessor, you should add the following spec in your experiment's YAML config file: diff --git a/src/sdk/pynni/nni/curvefitting_assessor/example_of_curve_fitting.PNG b/docs/img/curvefitting_example.PNG similarity index 100% rename from src/sdk/pynni/nni/curvefitting_assessor/example_of_curve_fitting.PNG rename to docs/img/curvefitting_example.PNG diff --git a/src/sdk/pynni/nni/curvefitting_assessor/expression_xi.gif b/docs/img/curvefitting_expression_xi.gif similarity index 100% rename from src/sdk/pynni/nni/curvefitting_assessor/expression_xi.gif rename to docs/img/curvefitting_expression_xi.gif diff --git a/src/sdk/pynni/nni/curvefitting_assessor/f_comb.gif b/docs/img/curvefitting_f_comb.gif similarity index 100% rename from src/sdk/pynni/nni/curvefitting_assessor/f_comb.gif rename to docs/img/curvefitting_f_comb.gif diff --git a/src/sdk/pynni/nni/curvefitting_assessor/learning_curve.PNG b/docs/img/curvefitting_learning_curve.PNG similarity index 100% rename from src/sdk/pynni/nni/curvefitting_assessor/learning_curve.PNG rename to docs/img/curvefitting_learning_curve.PNG diff --git a/docs/img/example_of_curve_fitting.PNG b/docs/img/example_of_curve_fitting.PNG new file mode 100644 index 0000000000..f405411eb6 Binary files /dev/null and b/docs/img/example_of_curve_fitting.PNG differ diff --git a/docs/img/expression_xi.gif b/docs/img/expression_xi.gif new file mode 100644 index 0000000000..ff8d217c96 Binary files /dev/null and b/docs/img/expression_xi.gif differ diff --git a/docs/img/f_comb.gif b/docs/img/f_comb.gif new file mode 100644 index 0000000000..f2da808855 Binary files /dev/null and b/docs/img/f_comb.gif differ diff --git a/docs/img/learning_curve.PNG b/docs/img/learning_curve.PNG new file mode 100644 index 0000000000..247e1e3968 Binary files /dev/null and b/docs/img/learning_curve.PNG differ diff --git a/docs/requirements.txt b/docs/requirements.txt index 2079b87621..48a401bd5c 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -8,4 +8,5 @@ hyperopt json_tricks numpy scipy -coverage \ No newline at end of file +coverage +sklearn \ No newline at end of file diff --git a/docs/zh_CN/Builtin_Assessors.md b/docs/zh_CN/Builtin_Assessors.md index 3ffc3e6f92..99e8aff23d 100644 --- a/docs/zh_CN/Builtin_Assessors.md +++ b/docs/zh_CN/Builtin_Assessors.md @@ -64,6 +64,7 @@ assessor: * **optimize_mode** (*maximize 或 minimize, 可选, 默认值为 maximize*) - 如果为 'maximize', Assessor 会在结果小于期望值时**终止** Trial。 如果为 'minimize',Assessor 会在结果大于期望值时**终止** Trial。 * **start_step** (*int, 可选, 默认值为 6*) - 只有收到 start_step 个中间结果后,才开始判断是否一个 Trial 应该被终止。 * **threshold** (*float, 可选, 默认值为 0.95*) - 用来确定提前终止较差结果的阈值。 例如,如果 threshold = 0.95, optimize_mode = maximize,最好的历史结果是 0.9,那么会在 Trial 的预测值低于 0.95 * 0.9 = 0.855 时停止。 +* **gap** (*int, 可选, 默认值为 1*) - Assessor 两次评估之间的间隔次数。 例如:如果 gap = 2, start_step = 6,就会评估第 6, 8, 10, 12... 个中间结果。 **使用样例:** @@ -76,4 +77,5 @@ assessor: optimize_mode: maximize start_step: 6 threshold: 0.95 + gap: 1 ``` \ No newline at end of file diff --git a/docs/zh_CN/curvefittingAssessor.md b/docs/zh_CN/curvefittingAssessor.md index dcfe29680f..65a1d1ef9a 100644 --- a/docs/zh_CN/curvefittingAssessor.md +++ b/docs/zh_CN/curvefittingAssessor.md @@ -6,21 +6,15 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing,即学 此算法中,使用了 12 条曲线来拟合学习曲线,从[参考论文](http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf)中选择了大量的参数曲线模型。 学习曲线的形状与先验知识是一致的:都是典型的递增的、饱和的函数。 -

-drawing -

+![](../img/curvefitting_learning_curve.PNG) 所有学习曲线模型被合并到了单个,更强大的模型中。 合并的模型通过加权线性混合: -

-drawing -

+![](../img/curvefitting_f_comb.gif) 合并后的参数向量 -

-drawing -

+![](../img/curvefitting_expression_xi.gif) 假设增加一个高斯噪声,且噪声参数初始化为最大似然估计。 @@ -36,9 +30,7 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing,即学 下图显示了此算法在 MNIST Trial 历史数据上结果。其中绿点表示 Assessor 获得的数据,蓝点表示将来,但未知的数据,红色线条是 Curve fitting Assessor 的预测曲线。 -

-drawing -

+![](../img/curvefitting_example.PNG) ## 2. 用法 @@ -61,6 +53,10 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing,即学 * 阈值的默认值是 0.95。 # 注意:如果选择了 minimize 模式,要让 threshold >= 1.0 (如 threshold=1.1) threshold: 0.95 + # (可选) gap 是两次评估之间的间隔次数。 + # 例如:如果 gap = 2, start_step = 6,就会评估第 6, 8, 10, 12... 个中间结果。 + * gap 的默认值是 1。 + gap: 1 ## 3. 文件结构