NAS documentation initation

docs/en_US/NAS/Overview.md

@@ -1,66 +1,62 @@
-# NNI Programming Interface for Neural Architecture Search (NAS)
-
-*This is an experimental feature, programming APIs are almost done, NAS trainers are under intensive development. ([NAS annotation](../AdvancedFeature/GeneralNasInterfaces.md) will become deprecated in future)*
-
-Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging. However, it takes great efforts to implement those algorithms, and it is hard to reuse code base of one algorithm for implementing another.
-
-To facilitate NAS innovations (e.g., design/implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexible programming interface is crucial.
-
-## Programming interface
-
-A new programming interface for designing and searching for a model is often demanded in two scenarios.
-
-    1. When designing a neural network, the designer may have multiple choices for a layer, sub-model, or connection, and not sure which one or a combination performs the best. It would be appealing to have an easy way to express the candidate layers/sub-models they want to try. 
-    2. For the researchers who are working on automatic NAS, they want to have an unified way to express the search space of neural architectures. And making unchanged trial code adapted to different searching algorithms.
-
-For expressing neural architecture search space, we provide two APIs:
-
-```python
-# choose one ``op`` from ``ops``, for pytorch this is a module.
-# ops: for pytorch ``ops`` is a list of modules, for tensorflow it is a list of keras layers. An example in pytroch:
-# ops = [PoolBN('max', channels, 3, stride, 1, affine=False),
-#        PoolBN('avg', channels, 3, stride, 1, affine=False),
-#        FactorizedReduce(channels, channels, affine=False),
-#        SepConv(channels, channels, 3, stride, 1, affine=False),
-#        DilConv(channels, channels, 3, stride, 2, 2, affine=False)]
-# key: the name of this ``LayerChoice`` instance
-nni.nas.LayerChoice(ops, key)
-# choose ``n_selected`` from ``n_candidates`` inputs.
-# n_candidates: the number of candidate inputs
-# n_selected: the number of chosen inputs
-# reduction: reduction operation for the chosen inputs
-# key: the name of this ``InputChoice`` instance
-nni.nas.InputChoice(n_candidates, n_selected, reduction, key)
+# Neural Architecture Search (NAS) on NNI
+
+Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging.
+
+However, it takes great efforts to implement NAS algorithms, and it is hard to reuse code base of existing algorithms in new one. To facilitate NAS innovations (e.g., design and implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexible programming interface is crucial.
+
+With this motivation, our ambition is to provide a unified architecture in NNI, to accelerate innovations on NAS, and apply state-of-art algorithms on real world problems faster.
+
+## Supported algorithms
+
+NNI supports below NAS algorithms now, and being adding more. User can reproduce an algorithm, or use it on owned dataset. we also encourage user to implement other algorithms with [NNI API](#use-nni-api), to benefit more people.
+
+Note, these algorithms run standalone without nnictl, and supports PyTorch only.
+
+### DARTS
+
+The main contribution of [DARTS: Differentiable Architecture Search][3] on algorithm is to introduce a novel algorithm for differentiable network architecture search on bilevel optimization.
+
+#### Usage
+
+```bash
+### In case NNI code is not cloned.
+git clone https://github.com/Microsoft/nni.git
+
+cd examples/nas/darts
+python search.py
 ```
 
-After writing your model with search space embedded in the model using the above two APIs, the next step is finding the best model from the search space. Similar to optimizers of deep learning models, the procedure of finding the best model from search space can be viewed as a type of optimizing process, we call it `NAS trainer`. There have been several NAS trainers, for example, `DartsTrainer` which uses SGD to train architecture weights and model weights iteratively, `ENASTrainer` which uses a controller to train the model. New and more efficient NAS trainers keep emerging in research community.
-
-NNI provides some popular NAS trainers, to use a NAS trainer, users could initialize a trainer after the model is defined:
-
-```python
-# create a DartsTrainer
-trainer = DartsTrainer(model,
-                       loss=criterion,
-                       metrics=lambda output, target: accuracy(output, target, topk=(1,)),
-                       model_optim=optim,
-                       lr_scheduler=lr_scheduler,
-                       num_epochs=50,
-                       dataset_train=dataset_train,
-                       dataset_valid=dataset_valid,
-                       batch_size=args.batch_size,
-                       log_frequency=args.log_frequency)
-# finding the best model from search space
-trainer.train()
-# export the best found model
-trainer.export_model()
+### P-DARTS
+
+[Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://arxiv.org/abs/1904.12760) bases on DARTS(#DARTS). It main contribution on algorithm is to introduce an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure.
+
+#### Usage
+
+```bash
+### In case NNI code is not cloned.
+git clone https://github.com/Microsoft/nni.git
+
+cd examples/nas/pdarts
+python main.py
 ```
 
-Different trainers could have different input arguments depending on their algorithms. After training, users could export the best one of the found models through `trainer.export_model()`.
+## Use NNI API
+
+NOTE, we are trying to support various NAS algorithms with unified programming interface, and it's in very experimental stage. It means the current programing interface may be updated significantly.
+
+*previous [NAS annotation](../AdvancedFeature/GeneralNasInterfaces.md) interface will be deprecated soon.*
+
+### Programming interface
+
+The programming interface of designing and searching a model is often demanded in two scenarios.
+
+1. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs  best. So it needs an easy way to express the candidate layers or sub-models.
+2. When applying NAS on a neural network, it needs an unified way to express the search space of architectures, so that it doesn't need to update trial code for different searching algorithms.
 
-[Here](https://github.com/microsoft/nni/blob/dev-nas-refactor/examples/nas/darts/main.py) is a trial example using DartsTrainer.
+NNI proposed API is [here](https://github.com/microsoft/nni/tree/dev-nas-refactor/src/sdk/pynni/nni/nas/pytorch). And [here](https://github.com/microsoft/nni/tree/dev-nas-refactor/examples/nas/darts) is an example of NAS implementation, which bases on NNI proposed interface.
 
 [1]: https://arxiv.org/abs/1802.03268
 [2]: https://arxiv.org/abs/1707.07012
 [3]: https://arxiv.org/abs/1806.09055
 [4]: https://arxiv.org/abs/1806.10282
-[5]: https://arxiv.org/abs/1703.01041
+[5]: https://arxiv.org/abs/1703.01041