v2.0 Release Plan

[QuanluZhang]

This is the draft plan for the upcoming iteration in Oct ~ Nov, it's a 4 weeks iteration.

Release Plan for v2.0

• Release manager: @liuzhe-lz
• Feature freeze date: 11.20 11.27 11.30
• Code freeze and demo date: 12.4 12.8 12.11 12.14 12.16
• Release date: 12.21 12.28
  • alpha 2: 12.28
  • alpha 3: 12.30 1.4
  • rc: 1.4 1.6
  • release: 1.11

---

Let's try to adopt the legend of annotations:

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
💪	stretch goal for this iteration
🔴	missing issue / pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team
⬛	a large work item, larger than one iteration

Epic

• Improved NAS framework (Retiarii) on NNI, 40 days

  • graph converter for PyTorch code, 10 days，NAS 2.0 - graph converter for PyTorch code #2938, dev owner: @QuanluZhang
  • execution engine (with integration), 5 days, NAS 2.0 - execution engine (with integration) #2939, dev owner: @ultmaster
  • mutation, 6 days, NAS 2.0 - mutation #2941, dev owner: @liuzhe-lz
  • User interface, 5 days @liuzhe-lz ETA: 11/18 11/20 (PR experiment management backend #3081 merge +1d) PR Create experiment from Python code #3111
    • weight sharing support @QuanluZhang
  • cross-graph optimizations, 10 days, NAS 2.0 - cross-graph optimizations #2940, dev owner: @hzhua ETA to be decided: PR [Retiarii] cross-graph optimization: input deduplication #3105
  • integration (first end-to-end), 5 days, NAS 2.0 - Integration #2942, dev owner: @QuanluZhang ETA: 11/13 11/16 after user interface (review: @liuzhe-lz @ultmaster )
  • search space (P0 Quanlu 1, P0 Yuge 1, P1 others best efforts) support and test, 10 days, NAS 2.0 - search space support and test #2943, dev owners: @ultmaster /@QuanluZhang ETA: P0 11/20; P1: 11/27
  • integration tests, 5 days, NAS 2.0 - Integration Test #2944, dev owner: @liuzhe-lz ETA: (depends on NAS 2.0 - search space support and test #2943's P0 API) 11/27 11/2 (after feature freeze)
  • Weight sharing, 10 days @QuanluZhang TODO: before 11/17 Quanlu drive the possible implementation plan; after 11/17, Yuge take over design and implementation. ETA: 11/27
  • P2 - (blocked by NAS search space structure) support NAS Once-for-all alog on NNI, 5 days, @colorjam NAS 2.0 - NAS algo Once for all #2955

• Support heterogenous training service @SparkSnail PR Support heterogeneous environment service #3097 (review: @squirrelsc @liuzhe-lz )

  • Support remote
  • Support azureml
  • Support local
  • Support PAI
  • doc 11.25
  • refactor code for review 11.26
  • IT pipeline (on going)

• Release materials @scarlett2018 /@liuzhe-lz (after feature freeze 11/27, target end date 12/11)

  • release process, 1-2 week @liuzhe-lz
  • doc, article, and demo 1 week @scarlett2018
  • Release press (demo new features) @scarlett2018
    • Upload video

Features

• Reorganize NNI python package, package hierarchy, source code directory hierarchy - dev owner: @liuzhe-lz
  • P0 - able to install and run, 3 days, ETA: 10/15
  • add UT pipeline test, 7 days, ETA: 10/26 PR Refactor code hierarchy part 3: Unit test #3037 review: @chicm-ms @SparkSnail
  • IT pipelines PR Integration test #3088 review: @SparkSnail @chicm-ms
• P2 - Add one NAS NLP benchmark, 7 days - dev owner: @98may (@ultmaster offers help), NAS NLP benchmark #2954 ETA: 11/17 11/20 11/27
• P0 WebUI, 9 days
  • backend support of experiment management, @J-shang PR experiment management backend #3081 review: @SparkSnail @liuzhe-lz
  • add experiment management page 6 days, @Lijiaoa (demo done) PR Multiple Experiments Management UX #3127
  • log UI refactor 3 days (@Lijiaoa ) 11/6 PR fix bug of advisor and assesor written backwards #3070 ready for review
  • New Overview Table & Chart UI Improvement  #2937 overview refactor @yangou1988 PR update overview layout #3046 (review: @ultmaster @corlorjam )
    • Last update 11/25 PR [webui] update the layout of overview page #3123
• P1 - model compression / graph_utils supports pytorch 1.6 AttributeError: module 'torch.onnx' has no attribute 'set_training' in pytorch1.6 #2877 can not generate graph using TorchModuleGraph in pytorch 1.6 #2879 @chicm-ms 11/6 PR graphutils supports torch17 #3076 (review @QuanluZhang @zheng-ningxin )
  • Fix test case @ultmaster
• P2 - add fpgm to agp /other pruning schedule (add more benchmark results) Add various pruner algorithm for different pruning schedule #2853 , 2 day, @chicm-ms ETA: 11/17 11/19 PR Pruning schedule supports fpgm #3110
• P2 - refactor experiment config file, @liuzhe-lz draft PR New experiment config #3138
  • 2 days design & discussion (issue Planning of updating experiment configuration format #3087 )
  • 3 days implementation ETA: 11/27
• P2 - Tuning algorithm's meta data are located in three different places, plan to unify them, design, depends, @chicm-ms , 5 days PR Refactor builtin/customized tuner installation #3134

Tasks

• Port doc to rst and fix links, 2 day evaluation, @ultmaster

• Unify the names of trail job id; @J-shang 9/16 , pr unify trial job id name #2896 unify trial job id name #3053 (review: @chicm-ms @QuanluZhang )

• use NNI's flops counter (refactor the counter first) @colorjam PR: Refactor flops counter #3048 reivew: @chicm-ms @QuanluZhang @ultmaster

• Bug fix of experiment resume, @J-shang 1 day (evaluated by quanlu)

• check experiment duration under the tuner_no_more_trial state (Issue execDuration does not incrase when TUNER_NO_MORE_TRIAL #2758 ), @J-shang 1 day PR: quick fix exp-duration pause timing under no_more_trial status #3043 @liuzhe-lz @QuanluZhang

• AML scheduling (Issue new scheduler for Azure ML training service #3014) @SparkSnail 1 day ~PR Support single machine scheduler policy in AML mode #3034 review: @QuanluZhang @chicm-ms ~

• HyperBand refactor @J-shang 3 days PR: Hyperband Refector #3040 review: @QuanluZhang @colorjam

• Add algorithm import shortcut @liuzhe-lz 3 days draft PR Add algorithms importing shortcut #3139

• AdaptDL integration test @SparkSnail

• Quantization export @linbinskn

• update API reference for model compression in doc

• Import shortcut @ultmaster

• Pipeline (fix ut numpy dependency, check it) @ultmaster

• Fix NNI manager warning @J-shang

Stretch Goals & Deferred Items

• tensorboard backend support, 5 days @J-shang (@SparkSnail help)

• multi-objective optimization algorithm, NNI多目标搜索算法 #2525 , 5 days - @ultmaster

• Cream2NNI @hwpengms , pr Request for Integrating the new NAS algorithm: Cream #2705 , under refactoring (review @QuanluZhang )

• Update TF model compression: support checkpoint, 3 days @liuzhe-lz

• P2, 2.0-candidates, under design discussion support post training quantization speedup, 14 days - dev owner: @linbinskn @J-shang

• P2 Update node/python JSON handling, 3 days - @liuzhe-lz

  • survey node/python JSON handling
  • fix the JSON parse twice issue, 1 day
  • Flatten nested JSON in RPC protocol, 5 days

• FBnetV3 tuning algorithm, Can we do hyperparameter optimization while doing NAS? #2591 , improve PBT tuner #2357 , 7 days - dev owner: @J-shang

• HyperBand further refactor @J-shang

• Merge .config and db.experimentProfile.param @J-shang

• Support code coverage check, add this check to IT, 7 days @chicm-ms

• test classic nas tuners on NNI NAS benchmark and report evaluation results, ? days, @98may

• Search trial by parameter (issue Refine search by paramters #3024) @sharonxingyufan @Lijiaoa

• Bump dependency version, @liuzhe-lz 5 days

Backlog

• Tutorial to use tuner directly instead of using nni manager (like hyperopt?) (maybe @liuzhe-lz next release)
• NAS visualization improvements: refactor graph representation @ultmaster (v2.1 candidate)
• AutoML for data augmentation (image and NLP) (research)
• Write search space pruning with flops in a separated component (research)

[liuzhe-lz]

Bug Bash

• Document convertion @ yiru
• Retiarii
  • Graph gen @ultmaster
  • Weight sharing @QuanluZhang
  • Dedup @colorjam
  • end-to-end @colorjam
• Heterogenous training service @liuzhe-lz
• NAS NLP benchmark @ linbin
• Web UI new layout @ yiru
• Experiment management @SparkSnail
• Model compression
  • PyTorch 1.6 support @J-shang
  • FPGM for AGP @J-shang
  • Flops counter @ linbin
• Launch from Python @J-shang
  • fix (note: add dispatcher log) PR Fix RemoteConfig bug and add save log to dispatcher.log for nni.Experiment #3245
• New configuration @ultmaster
  • fix
• Customized tuner install @liuzhe-lz
• Experiment resume bugfix @QuanluZhang
  • fix
• Experiment duration under TUNER_NO_MORE_TRIAL @98may
• HyperBand refactor @ ayan
• ADL @SparkSnail
• Quantization export @QuanluZhang
• NNI manager warning @QuanluZhang
• Check other PRs not listed in tasks @98may
• Kill trial @liuzhe-lz
• Logging (check if NNI manager, dispatcher, trial log contains logging and printing output, including multiprocess scenario) @ultmaster
• nnictl save , load , log commands @98may
• quantization bug (PR fix quant grad function calculation error #3160) @J-shang
• Cream @ultmaster

Documentation

• Retiarii @QuanluZhang
  • ETA for v1 @QuanluZhang 12/30; Reviewers: @colorjam @ultmaster @kvartet @scarlett2018
• Heterogenous
  • [heterogenous --> hybrid] ETA for the 12/31 @kvartet @scarlett2018
• Launch from Python PR ready @J-shang add launch from python code doc #3217 ; Reviewers: @liuzhe-lz @QuanluZhang @kvartet; ETA for review: 12/31;
• New experiment config @liuzhe-lz Experiment config doc #3222; Reviewer: @SparkSnail ; TODO: @liuzhe-lz fix reviewer comments; + @kvartet (comments: there are lots of TODO in the doc); ETA for doc update: 12/31;
• New custom algorithm register mechanism ask @chicm-ms (@J-shang) PR update customized algos doc #3241
• Translation @kvartet; ETA for the PR: 12/31;
• README @scarlett2018 ETA for 1/4

Document Update & Regression Test

• HPO
  • mnist-{a,b}... , networkmorphism, autogbdt, gasquad @QuanluZhang
  • other mnist @ linbin
  • other trials @98may
    • fix: Update doc, remove paiYarn
  • custom tuner examples (TODO)
  • System autotuning @QuanluZhang > @ xiaotian
• NAS @ultmaster
• Model Compression
  • Regression @colorjam
  • Speedup @ ningxin
• Training Service @J-shang
  • kubeflow @ linbin & @liuzhe-lz
• Reference @liuzhe-lz
• Overview @ yiru
  • Install from source code doc
• Others
  • Feature Engineering @98may

Pipelines

[liuzhe-lz]

Bugs Found in Bugbash

• Quantization BNN & Dorefa @ linbin PR Fix dorefa bnn #3247
• Configuration v2 error handling @ultmaster PR Add config v2 example and refine validation error message #3248
• Retiarii output @QuanluZhang update doc
• Python experiment termination
  • Linux @liuzhe-lz
  • Windows @J-shang
• OpenPAI windows mountpoint check @liuzhe-lz
• Remove PAI yarn doc @SparkSnail
• Reuse in remote training service doc @SparkSnail
• Update hybrid doc that windows does not support local @SparkSnail
• Web UI doc update @Lijiaoa Update webui doc image and add experiments management module #3229
• Install from source doc @liuzhe-lz
• Change quickstart example to torch @liuzhe-lz
• Change experiment management page order @Lijiaoa
• nnictl trial ls timestamp @J-shang nnictl get port and pid from .experiment #3235
• hybrid v2 config @QuanluZhang
• add new config format example file in mnist-tf and mnist-pytorch examples @ultmaster

[liuzhe-lz]

• Retiarii strategy doc @QuanluZhang
• Retiarii output doc @QuanluZhang
• Release note @ yiru
• Pipeline @liuzhe-lz