Merge pull request #166 from Microsoft/master

merge master

README.md

@@ -17,7 +17,7 @@
 NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments. 
 The tool dispatches and runs trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in different environments like local machine, remote servers and cloud.
 
-### **NNI [v0.6](https://github.com/Microsoft/nni/releases) has been released!**
+### **NNI [v0.7](https://github.com/Microsoft/nni/releases) has been released!**
 <p align="center">
   <a href="#nni-v05-has-been-released"><img src="docs/img/overview.svg" /></a>
 </p>
@@ -100,41 +100,65 @@ Targeting at openness and advancing state-of-art technology, [Microsoft Research
 We encourage researchers and students leverage these projects to accelerate the AI development and research.
 
 ## **Install & Verify**
+If you choose NNI Windows local mode and you use powershell to run script for the first time, you need to **run powershell as administrator** with this command first:
+```bash
+    Set-ExecutionPolicy -ExecutionPolicy Unrestricted
+```
 
 **Install through pip** 	
-* We support Linux and MacOS in current stage, Ubuntu 16.04 or higher, along with MacOS 10.14.1 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.	
+* We support Linux, MacOS and Windows(local mode) in current stage, Ubuntu 16.04 or higher, MacOS 10.14.1 along with Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
 
+Linux and MacOS
 ```bash
     python3 -m pip install --upgrade nni
 ```
+Windows
+```bash
+    python -m pip install --upgrade nni
+```
 Note:
 
 * `--user` can be added if you want to install NNI in your home directory, which does not require any special privileges.
+* Currently NNI on Windows only support local mode. Anaconda is highly recommanded to install NNI on Windows. 
 * If there is any error like `Segmentation fault`, please refer to [FAQ](docs/en_US/FAQ.md)
 
 **Install through source code**
-* We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) in our current stage. 
+* We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) and Windows local mode (10.1809) in our current stage. 
+
+Linux and MacOS
 * Run the following commands in an environment that has `python >= 3.5`, `git` and `wget`.
 ```bash	
-    git clone -b v0.6 https://github.com/Microsoft/nni.git
+    git clone -b v0.7 https://github.com/Microsoft/nni.git
     cd nni	
     source install.sh	
 ```
-
-For the system requirements of NNI, please refer to [Install NNI](docs/en_US/Installation.md)
+Windows
+* Run the following commands in an environment that has `python >=3.5`, `git` and `powershell`    
+```bash
+  git clone -b v0.7 https://github.com/Microsoft/nni.git
+  cd nni
+  powershell ./install.ps1
+```
+For the system requirements of NNI, please refer to [Install NNI](docs/en_US/Installation.md)  
+For NNI Windows local mode, please refer to [NNI Windows local mode](docs/en_US/WindowsLocalMode.md) 
 
 **Verify install**	
 
 The following example is an experiment built on TensorFlow. Make sure you have **TensorFlow installed** before running it.	
 * Download the examples via clone the source code.	
 ```bash	
-    git clone -b v0.6 https://github.com/Microsoft/nni.git
+    git clone -b v0.7 https://github.com/Microsoft/nni.git
 ```
+Linux and MacOS
 * Run the mnist example.
 ```bash
     nnictl create --config nni/examples/trials/mnist/config.yml
 ```
-
+Windows
+* Run the mnist example.
+```bash
+    nnictl create --config nni/examples/trials/mnist/config_windows.yml
+```
 * Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.
 
 ```

deployment/deployment-pipelines.yml

@@ -172,7 +172,7 @@ jobs:
     condition: eq( variables['upload_package'], 'true')
     displayName: 'upload nni package to pypi/testpypi'
 
-- job: 'Build_upload_nni_windows'
+- job: 'Build_upload_nni_win32'
   dependsOn: version_number_validation
   condition: succeeded()
   pool:
@@ -182,35 +182,78 @@ jobs:
       Python36:
         PYTHON_VERSION: '3.6'
   steps:
-  - script: |
-      python -m pip install --upgrade pip setuptools --user
-      python -m pip install twine --user
+  - powershell: |
+      python -m pip install --upgrade pip setuptools
+      python -m pip install twine
     displayName: 'Install twine'
 
-  - script: |
-      cd deployment/pypi
-      if [ $(build_type) = 'prerelease' ]
-      then
+  - powershell: |
+      cd deployment\pypi
+      if($env:BUILD_TYPE -eq 'prerelease'){
         # NNI build scripts (powershell) uses branch tag as package version number
-        git tag $(build_version)
-        echo 'building prerelease package...'
-        powershell.exe ./install.ps1 -version_ts $True
-      else
-        echo 'building release package...'
-        powershell.exe ./install.ps1
-      fi
+        git tag $env:BUILD_VERSION
+        Write-Host 'building prerelease package...'
+        .\install.ps1 -version_os 32 -version_ts $True
+      }
+      else{
+        Write-Host 'building release package...'
+        .\install.ps1 -version_os 32 -version_ts $False
+      }
     condition: eq( variables['upload_package'], 'true')
     displayName: 'build nni bdsit_wheel'
 
-  - script: |
-      cd deployment/pypi
-      if [ $(build_type) = 'prerelease' ]
-      then
-        echo 'uploading prerelease package to testpypi...'
-        python -m twine upload -u $(testpypi_user) -p $(testpypi_pwd) --repository-url https://test.pypi.org/legacy/ dist/*
-      else
-        echo 'uploading release package to pypi...'
-        python -m twine upload -u $(pypi_user) -p $(pypi_pwd) dist/*
-      fi
+  - powershell: |
+      cd deployment\pypi
+      if($env:BUILD_TYPE -eq 'prerelease'){
+        Write-Host 'uploading prerelease package to testpypi...'
+        python -m twine upload -u $env:TESTPYPI_USER -p $env:TESTPYPI_PWD --repository-url https://test.pypi.org/legacy/ dist/*
+      }
+      else{
+        Write-Host 'uploading release package to pypi...'
+        python -m twine upload -u $env:PYPI_USER -p $env:PYPI_PWD dist/*
+      }
+    condition: eq( variables['upload_package'], 'true')
+    displayName: 'upload nni package to pypi/testpypi'
+
+- job: 'Build_upload_nni_win_amd64'
+  dependsOn: version_number_validation
+  condition: succeeded()
+  pool:
+    vmImage: 'vs2017-win2016'
+  strategy:
+    matrix:
+      Python36:
+        PYTHON_VERSION: '3.6'
+  steps:
+  - powershell: |
+      python -m pip install --upgrade pip setuptools
+      python -m pip install twine
+    displayName: 'Install twine'
+
+  - powershell: |
+      cd deployment\pypi
+      if($env:BUILD_TYPE -eq 'prerelease'){
+        # NNI build scripts (powershell) uses branch tag as package version number
+        git tag $env:BUILD_VERSION
+        Write-Host 'building prerelease package...'
+        .\install.ps1 -version_os 64 -version_ts $True
+      }
+      else{
+        Write-Host 'building release package...'
+        .\install.ps1 -version_os 64 -version_ts $False
+      }
+    condition: eq( variables['upload_package'], 'true')
+    displayName: 'build nni bdsit_wheel'
+
+  - powershell: |
+      cd deployment\pypi
+      if($env:BUILD_TYPE -eq 'prerelease'){
+        Write-Host 'uploading prerelease package to testpypi...'
+        python -m twine upload -u $env:TESTPYPI_USER -p $env:TESTPYPI_PWD --repository-url https://test.pypi.org/legacy/ dist/*
+      }
+      else{
+        Write-Host 'uploading release package to pypi...'
+        python -m twine upload -u $env:PYPI_USER -p $env:PYPI_PWD dist/*
+      }
     condition: eq( variables['upload_package'], 'true')
     displayName: 'upload nni package to pypi/testpypi'

deployment/docker/README.md

@@ -12,7 +12,7 @@ PyTorch 0.4.1
 scikit-learn 0.20.0
 pandas 0.23.4
 lightgbm 2.2.2
-NNI v0.6
+NNI v0.7
 ```
 You can take this Dockerfile as a reference for your own customized Dockerfile. 
 

deployment/pypi/README.md

@@ -47,15 +47,14 @@ This is the PyPI build and upload tool for NNI project.
   powershell
   Python >= 3.5
   Pip
-  Node.js
   Yarn
-  tar
   ```
 
 * __How to build__
-
+
+  parameter `version_os` is used to build for Windows 64-bit or 32-bit. 
   ```bash
-  powershell ./install.ps1
+  powershell ./install.ps1 -version_os [64/32]
   ```
 
 * __How to upload__
@@ -70,4 +69,4 @@ This is the PyPI build and upload tool for NNI project.
   ```bash
   powershell ./upload.ps1 -test $False
   ```
-  You may need to input the account and password of https://pypi.org during this process.
+  You may need to input the account and password of https://pypi.org during this process.

deployment/pypi/clean.ps1

@@ -4,4 +4,4 @@ Remove-Item $CWD\build -Recurse -Force
 Remove-Item $CWD\dist -Recurse -Force
 Remove-Item $CWD\nni -Recurse -Force
 Remove-Item $CWD\nni.egg-info -Recurse -Force
-Remove-Item $CWD\node-$OS_SPEC-x64 -Recurse -Force
+Remove-Item $CWD\node-$OS_SPEC -Recurse -Force

deployment/pypi/install.ps1

@@ -1,9 +1,17 @@
-param([bool]$version_ts=$false)
+param([int]$version_os, [bool]$version_ts=$false)
+[System.Net.ServicePointManager]::DefaultConnectionLimit = 100
 [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
 $CWD = $PWD
 
 $OS_SPEC = "windows"
-$WHEEL_SPEC = "win_amd64"
+if($version_os -eq 64){
+    $OS_VERSION = 'win64'
+    $WHEEL_SPEC = 'win_amd64'
+}
+else{
+    $OS_VERSION = 'win32'
+    $WHEEL_SPEC = 'win32'
+}
 
 $TIME_STAMP = date -u "+%y%m%d%H%M"
 $NNI_VERSION_VALUE = git describe --tags --abbrev=0
@@ -17,18 +25,21 @@ if($version_ts){
 
 $NNI_VERSION_TEMPLATE = "999.0.0-developing"
 
-python -m pip install --user --upgrade setuptools wheel
+python -m pip install --upgrade setuptools wheel
 
-$nodeUrl = "https://aka.ms/nni/nodejs-download/win64"
-$NNI_NODE_ZIP = "$CWD\node-$OS_SPEC-x64.zip"
-$NNI_NODE_FOLDER = "$CWD\node-$OS_SPEC-x64"
+$nodeUrl = "https://aka.ms/nni/nodejs-download/" + $OS_VERSION
+$NNI_NODE_ZIP = "$CWD\node-$OS_SPEC.zip"
+$NNI_NODE_FOLDER = "$CWD\node-$OS_SPEC"
+$unzipNodeDir = "node-v*"
 (New-Object Net.WebClient).DownloadFile($nodeUrl, $NNI_NODE_ZIP)
 if(Test-Path $NNI_NODE_FOLDER){
     Remove-Item $NNI_NODE_FOLDER -Recurse -Force
 }
-New-Item $NNI_NODE_FOLDER -ItemType Directory
-cmd /c tar -xf $NNI_NODE_ZIP -C $NNI_NODE_FOLDER --strip-components 1
+Expand-Archive $NNI_NODE_ZIP -DestinationPath $CWD
+$unzipNodeDir = Get-ChildItem "$CWD\$unzipNodeDir"
+Rename-Item $unzipNodeDir $NNI_NODE_FOLDER
 
+$env:PATH = $NNI_NODE_FOLDER+';'+$env:PATH
 cd $CWD\..\..\src\nni_manager
 yarn
 yarn build
@@ -46,4 +57,4 @@ cd $CWD\nni
 yarn --prod
 cd $CWD
 (Get-Content setup.py).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content setup.py
-python setup.py bdist_wheel -p $WHEEL_SPEC
+python setup.py bdist_wheel -p $WHEEL_SPEC

deployment/pypi/setup.py

@@ -34,7 +34,7 @@
 
 data_files = [('bin', ['node-{}-x64/bin/node'.format(os_type.lower())])]
 if os_type == 'Windows':
-    data_files = [('.\Scripts', ['node-{}-x64/node.exe'.format(os_type.lower())])]
+    data_files = [('.\Scripts', ['node-{}/node.exe'.format(os_type.lower())])]
 
 for (dirpath, dirnames, filenames) in walk('./nni'):
     files = [path.normpath(path.join(dirpath, filename)) for filename in filenames]

docs/en_US/Builtin_Tuner.md

@@ -132,6 +132,8 @@ tuner:
 
 > Builtin Tuner Name: **SMAC**
 
+**Please note that SMAC doesn't support running on windows currently. The specific reason can be referred to this [github issue](https://github.com/automl/SMAC3/issues/483).**
+
 **Installation**
 
 SMAC need to be installed by following command before first use.

docs/en_US/Installation.md

@@ -1,6 +1,6 @@
 # Installation of NNI
 
-Currently we support installation on Linux, Mac and Windows.
+Currently we support installation on Linux, Mac and Windows(local mode).
 
 ## **Installation on Linux & Mac**
 
@@ -15,7 +15,7 @@ Currently we support installation on Linux, Mac and Windows.
 
   Prerequisite: `python >=3.5`, `git`, `wget`
   ```bash
-  git clone -b v0.6 https://github.com/Microsoft/nni.git
+  git clone -b v0.7 https://github.com/Microsoft/nni.git
   cd nni
   ./install.sh
   ```
@@ -24,25 +24,25 @@ Currently we support installation on Linux, Mac and Windows.
 
   You can also install NNI in a docker image. Please follow the instructions [here](https://github.com/Microsoft/nni/tree/master/deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`.
 
-## **Installation on Windows**
-
+## **Installation on Windows** 
+  When you use powershell to run script for the first time, you need **run powershell as administrator** with this command:
+  ```bash
+  Set-ExecutionPolicy -ExecutionPolicy Unrestricted
+  ```
+  Anaconda is highly recommanded.
 * __Install NNI through pip__
 
-  Prerequisite: `python >= 3.5`
+  Prerequisite: `python(64-bit) >= 3.5`
   ```bash
   python -m pip install --upgrade nni
   ```
 
 * __Install NNI through source code__
 
   Prerequisite: `python >=3.5`, `git`, `powershell`    
-  When you use powershell to run script for the first time, you need run powershell as Administrator with this command:
-  ```bash
-  Set-ExecutionPolicy -ExecutionPolicy Unrestricted
-  ```
-  Then you can install nni as administrator or current user as follows:
+  you can install nni as administrator or current user as follows:
   ```bash
-  git clone https://github.com/Microsoft/nni.git
+  git clone -b v0.7 https://github.com/Microsoft/nni.git
   cd nni
   powershell ./install.ps1
   ```
@@ -73,7 +73,7 @@ Below are the minimum system requirements for NNI on macOS. Due to potential pro
 |**Internet**|Boardband internet connection|
 |**Resolution**|1024 x 768 minimum display resolution|
 
-Below are the minimum system requirements for NNI on Windows. Due to potential programming changes, the minimum system requirements for NNI may change over time.
+Below are the minimum system requirements for NNI on Windows, Windows 10.1809 is well tested and recommend. Due to potential programming changes, the minimum system requirements for NNI may change over time.
 
 ||Minimum Requirements|Recommended Specifications|
 |---|---|---|
@@ -96,4 +96,4 @@ Below are the minimum system requirements for NNI on Windows. Due to potential p
 * [How to run an experiment on multiple machines?](RemoteMachineMode.md)
 * [How to run an experiment on OpenPAI?](PAIMode.md)
 * [How to run an experiment on Kubernetes through Kubeflow?](KubeflowMode.md)
-* [How to run an experiment on Kubernetes through FrameworkController?](FrameworkControllerMode.md)
+* [How to run an experiment on Kubernetes through FrameworkController?](FrameworkControllerMode.md)