2021 Jan release plan

[scarlett2018]

This plan captures the work in Jan 2021. This is a 6 week iteration major focus on system features and improvements, and we will ship it at to-be-decided.

Release Manager

@debuggy

Endgame

Feature freeze: Dec. 24
Code freeze: Dec. 21 Dec. 24
Scrum demo date: Dec.24
Bug Bash date: Dec.28 ~ Jan. 8
Release date & retrospective date: TBD

Test Plan: #5218

Work Items

Web Portal Experience test owner: @suiguoxin

• Responsive UI for surface book or other mobile device @debuggy @yangou1988 test: @TobeyQin
  responsive design principles
• P0 Home page overlap issue webportal home page responsive layout #5086 Refine pivot of home page #5180 @debuggy test: @suiguoxin test cases: Issues: webportal home page responsive layout #5086
• P0 Job detail page. Test cases: [Webportal] Add filter, search box and export csv file in task detail list #5175 (comment) @suiguoxin
  • Add sort, filter and search in task detail list Job detail page cannot be searched by browser #4917 @yiyione ETA: 12/14
  • Add export csv function @yiyione ETA: 12/14
• Job Submission page test: @suiguoxin @TobeyQin
  • P1 The defaulting should only be done after submission phase The defaulting should only be done after submission phase #4576 Create a new page for yaml editor #5172 @debuggy ETA: Dec. 22 Demo: Dec. 24
  • P1 New Job Submission Page Design @yangou1988 @debuggy ETA: Dec. 22 Demo: Dec. 24

Marketplace Items: microsoft/openpaimarketplace#152 test owner: @TobeyQin

Support different types of computing hardware #5138 @mydmdm test owner: @abuccts test cases suiyuan

• add support to enflame DTO add support to enflame DTO #5140

Deployment process refinement @suiguoxin @hzy46

• P0 Installation entry point: ETA: 3 days
  • master.csv + worker.csv -> layout.yaml
  • move config.yaml, layout.yaml under quick-start folder, remove all the argument parse logic
  • doc refactor
  • layout.yaml schema change
• P0 unify version requirements: pai version, pai image tag ETA: 1 day
  • set default value in config files
  • create a wiki for release manager : wiki
• add / remove node
  • P0 fix current remove-node doc: do not work
• P1 generate hiveD config with layout.yaml ETA: 3 days generate hived config from layout.yaml #5179
  • generate one vc for every machine-sku, leave the first vc as default vc
  • calculate free resources with specification in layout.yaml
• P1 check layout before installing k8s: ETA: 3 days
  • one-master constraint, master-node / worker-node can not be in the same machine layout schema check #5184
  • resource requirements, support CPU / provide interfaces for other computing devices [Multiple Hardwares] [Installation] Check if resource matches layout.yaml #5181
• P1 config folder structure arrangement
• P1 Refine installation logs: ETA: 1 day
  • add log for key steps
  • remove verbose & useless log
• P1 add skip service list argument add skip-service-list arg #5193

test owner: @debuggy @hzy46
Test Cases:

• deploy from scratch: follow doc @debuggy @TobeyQin
  • pure CPU cluster
  • CPU / GPU mixed cluster
  • pure GPU cluster
• upgrade from v1.4.1 to v1.5.0 doc @debuggy paictl will check version alignment when push config #5208
  • test skip-service-list argument
• add / remove nodes : doc
• uninstallation: doc

Log manager test owner: @yiyione

• P1 get logs api return code Get logs API return 500 not 404 when log not exists #5125 @Binyang2014 ETA: dec. 18

Release Tracking test owner: @debuggy

• feature proposal issue template including motivate and test cases @debuggy
• P1 Deployment: doc for release branch deployment @suiguoxin

Dockerhub pull policy #5219

• Docker image pull frequency limit in dockerhub @qiufeng large job test
  Possible Solutions:
  • use Microsoft account; waiting for reply
  • docker image in China;
  • docker hub pro;

Backlog

• Rest-Server Job API Perf Tests and Improvements Rest-Server Job API Perf Tests and Improvements #5027
• Python 2/3.5 Deprecation Upgrade Python 2 to Python 3 #5042
• P1 add / remove node with with layout.yaml Add / Remove Nodes with layout.yaml #5167
• P2 dev-box can be inside master node: doc, testing
• Refine Alert System [Monitor] Alert System Refine #4810 P2 items
• P2 too many tokens created Too many identical tokens have been created. #5123
• P2 Group management & VC request management for user and admin @yiyione Group & VC request management for user and admin #4949
• P2 vscode bug @yiyione
• P1 (design) Submit job ui should show protocol prerequisites parts 2020 Oct. ~ Nov. Release Plan openpaimarketplace#73 @debuggy @yiyione @TobeyQin
• P2 code coverage concern when developing feature
• P1 Release Process: refine archived oneNote doc as github wiki @yiyione
• P1 Agile CI and split heavy tests with nightly-deployment [CI] Setup nightly deployment to do the e2e tests and agile CI #5173 @yiyione
  • Design how to split the CI
  • Setup the nightly-deployment and test

HiveD scheduler

• Cell as sku in hived scheduler. @abuccts (backbone support (config) for "Cell", submission form supports for Cell)
  ETA: design: in progress
• P2 UX for HiveD @yangou1988

Job protocol update

• P1 support data via extending prerequisite in job protocol extend prerequisite field in job protocol #5145

DB controller

• make DBController tolerant to wrong framework request make DBController tolerant to wrong framework request #4889 When user's job config is too large, OpenPAI job will be in Waiting or Stopping forever  #5093