2020 Sept ~ Oct release plan

[scarlett2018]

This plan captures the work in Sept 2020. This is a 6 week iteration major focus on system features and improvements, and we will ship it in the beginning of August.

Release Manager

@abuccts

Endgame

Feature freeze: 9/30
Code freeze: 10/15
Scrum demo date: 9/30 ~ 10/12
Endgame & retrospective date: tbd

Plan Items

Below is a summary of the top level plan items.

marketplace related items microsoft/openpaimarketplace#60

• Grammar check @debuggy
• Fix data storage config mismathcing issue @debuggy
• COVID 19 bug fix @debuggy
• X-Ray Chest Check @yiyione
• Distributed Job @hzy46

test owner: @TobeyQin

HiveD scheduler

• Cluster autoscaler with HiveD scheduler on aks. @abuccts Add HiveD scheduler config adapter for CA #4868
• Support dynamic sku types for different vc on webportal @abuccts [Webportal] Support dynamic sku types for different vc #4900

test owner: @yiyione

• test case: on submission page, choose one vc, check sku type dropdown list, change another vc, check if dropdown list changes.

advanced job debug mode:

Per Task Retry History:

• FC generates Task history snapshots Expose Task History frameworkcontroller#62 @yqwang-ms
• Fluentd + DB to collect above snapshots @hzy46
• RestServer enrich job detail API by supporting history job attempt and including history task attempts @suiguoxin
• WebPortal expose the Per Task Retry History @debuggy Add task retry info in job detail page #4966
• FC release and update PAI @yqwang-ms

test owner: @yqwang-ms
Issues:

• View Exit Diagnostics misplaced
• when switch job attempt, table content and more diag button is unmatched
• After click stop button sucessfully, the button is still enabled until the job becomes Stopping Job stop button no feedback after click successfully #5023

Expose K8s events:

• event watcher to support Enrich job debugging info #4649 @hzy46 PR: [display event] add event watcher in database controller #4939
• add event api @hzy46
• show events in job detail page @debuggy Add job event page #4975

Test cases in PR #4939
test owner: @abuccts

GPU monitoring and utilization

• GPU utilization monitoring & handling GPU utilization monitoring & handling #4789 @suiguoxin

test cases: cordon-nodes / send email to user and admin / tag job
test owner: @Binyang2014

DShuttle

• Webportal support DShuttle type @yiyione Webportal use new storage API and support DShuttle type #4605
• DShuttle integration and internal deploy & usage @Binyang2014 (depending on the above item; ) Dshuttle integration Plan #4599 Integrate Dshuttle into PAI #4584

test case:

1. Upload data into AzureBlob which connected to DShuttle.
2. Submit job which mount DShuttle.
3. Consume the data through DShuttle mounted path

For failure pattern, please refer to #4599 (comment)

Documentation

• Use PAI DRI notebook, DRI tickets and Wiki to update document @hzy46 Use PAI DRI notebook, DRI tickets, Wiki to update document #4760 broken link of pai wiki maintain machine & service #1763 PR: Fix document using DRI tickets #4828 Add distributed training examples of PyTorch #4821
• Component owner brownbag plan @scarlett2018

Web Portal Experience

• Add help info for items in submit job page @debuggy Add help info #4950
• User experience survey

test owner: @hzy46

---

Other Candidates

postponed from last release

• GPU Utilization improve training cost ROI for EDU scenario @hzy46 Use alert-manager to alert/kill low GPU efficiency job #4623
• 22. Following up items of DB
  • P1 archive (delete) jobs (SHAIIC) Support delete/archive jobs #4453 @suiguoxin
  • P2 Job tags (SHAIIC) @suiguoxin
  • [comments] DB replication; Master/Slave; SQL Azure; Recover method; P2 //(pending)job GC

test owner: @debuggy
test cases: Job tags: add / delete tag api

• job tag test issues #5009

Backlog

Moving to #4988