Pure K8S Beta Release Plan - v0.17

[scarlett2018]

This is the placeholder for the candidate external release, more plan item will coming later.

Release Manager

@Binyang2014

Endgame

• TBD Code freeze for the endgame
• TBD Endgame done

The endgame details for this iteration are tracked in TBD.

Plan Items

Below is a summary of the top level plan items (by priority).

• P00 Deployment should be limited within few hours to at most 1 day in general @ydye P00 - Deployment story for Beta release #4022
  • Deployment pre-requisit/benchmark/pre-check list P00 - part of 4022 - Prerequisites/node check #1893
  • Clear goal for deployment ready for release
  • System requirement
  • Service dependency
  • Configurations and default values
• P0 Remove framework owner reference for priority class [Rest Server] Update job priority class #4131 @abuccts
• P0 Default not to create priority class [Rest Server] Update job priority class #4131 @abuccts
• P0 tag and commit number Alpha version 0.16.0 - 1 day @ydye/@debuggy
• P0 Finalize protocol - Job UUID P0 - Job Name, UID and Description #3935 @yqwang-ms [Rest Server] Update job name encoding method #4069
• P0 Security - K8S Dashboard https enabled and login with kubeconfig - 2~3 weeks @Binyang2014 P0 - K8S Dashboard https enabled and login with kubeconfig #4025
• P0 Security - add REST API for checking node/pod status [Rest Server] Add nodes / pods info API #3892 @sunqinzheng 1 week
• P0 Security - security review items @qfyin P0 - security review items  #4026
  initContainer Related
• P0 VS Code Client Support for Beta release @yiyione 2 week
  • job simulation for v2 job VS Code simulate PAI Job Running for v2 job #3913
  • team wise storage explorer VS Code Team share storage support #3945 PR:[VS Code] Team share storage explorer #4104
  • Source code auto upload VS Code auto upload source code in OpenPAI K8S version #3989
  • vscode remote debug for pai job VS Code remote debug OpenPAI jobs #4097
• P0 Finalize document and help P0 - Documentation ready for Beta Release #4023 @scarlett2018
• P0 Finalize UX for beta release @yangou1988 P0 - Finalize UX for beta release #4024
• P0 Candidate - Grafana Dashboard: [v0.16.0] Failed to fetch nodes and gpu info  #4020
• P0.5: Support to add Pod Creation http error pattern to errorspec Support to add Pod Creation http error pattern to errorspec #4125 @yqwang-ms
• P0.5: Watchdog auto delete leak priorityclass P0.5: Watchdog auto delete leak priorityclass #3866 @Binyang2014
• P0.5 Remove yarn version components in k8s version object model - 1-2 weeks @ydye P0.5 - Remove yarn version components in k8s version object model #4027
• P0.5 elastic search alternative option (Azure search or DB search as switchable options) @hzy46 / @mydmdm P0.5 - elastic search alternative option (Azure search or DB search as switchable options)  #4028 PR: Use Postgresql as an alternative storage for job history #4164
• P0.5 storage manager constantly restart @Binyang2014 storage manager crashed for unknown reason #4081
• P1 Security - Pass secret from job yaml as a secrete and to runtime @abuccts @Binyang2014
• P1 Security - Check ACS Docker Image in initContainer K8S Pod keep waiting in creating container, such as ErrImagePull #3572 @Binyang2014
• P1 Disallow ApiServer HA in beds P0.5: Disallow ApiServer HA for Pod safety #4120 @Binyang2014
• P1 Job history dev log on server and clients - 1 day @debuggy
• P1 Replace code injection in webportal with new plugin implementation (storage and set tensorboard disable by default) @Binyang2014 [RestServer, Webportal, Runtime] Change tensorboard to use runtime plugin #4105
• P1 do not modify user submitted/cloned job yaml P1 - Job config is modified after it is imported/uploaded/cloned #3823 @debuggy
• P1 kube runtime may change user's python environment P1 - kube runtime may change user's python environment #3941 @abuccts
• P1 Include Pure K8s tests in Jenkins P1 - Will need a unified framework to do integration test in jenkins #2455 P1 - Jenkins hangs due to disk full #2280 @abuccts P1 - [CI/CD] Update Jenkins #4035 P1 - [CI/CD] Add Jenkins stages for pure k8s version #4059
• P2 Remove yarn version components in building k8s version images @ydye [Rest-server] Notify user the default group externalname when login failed. #4167
• P1 Candidate - a failed runtime plugin shouldn't fail the entire runtime P1 Candidate - a failed runtime plugin shouldn't fail the entire runtime #4014 @Binyang2014
• P2 Security - When cloning a job, if it is pulling docker from ACS and it is cloned by a different user, warn the user to fill in user/password, if it is cloned by the same user, auto-fill the user/password P2 Candidate - Docker Auth information disappear in clone job #3829
• P1 Job retry page issue job retry failed with unknown reason #4008 @debuggy
• P1 GPU Vendor @abuccts support different GPU vendor in one cluster #4090 Add support for AMD GPU #4093
• P1 For basic authentication mode, prepare document to explain how to create a group and associate users to the group Provide a way to setup storage in basic authentication mode #4130

Deploy Notices

• P0 Set and deploy v=2 log level for all k8s components Set and deploy v=2 log level for all k8s components #4123 @Binyang2014

---

Some features on the special branch

• unified storage interface for users
• job persistence: enhanced codeDir and outputDir
• ssh key management and simplifying user behavior of ssh plugin
• open ssh connection in browser
• Show ssh command in job detail page
• etcd to DB synchronizer - job (basic) / user
• favorite jobs

---

• Grafana metrics link does not have time interval
• Jupyter notebook integration in job detail page
• Implement JobEnv and pass user-defined environment variables to runtime (Binyang How to set environment variables or pass structure data when submitting job? #3956)
• Skip intra communication for elastic job (Binyang P1 - Skip intra communication for elastic tasks #3708)
• Bring back debug mode (Binyang Reserve container after job failed. #3643, Better Job Debugging Experience for End Users #2210)
• Support nested AAD group (Yundong, Get Recursive Nested AD Users in AAD Mode #3440)
• replace disk cleaner with k8s pod eviction policy ([pure-k8s] Disk usage monitor module #3340, [pure k8s] K8s's pod eviction may not trigger when job container uses too much disk space #3765)
• When job is waiting, show the detailed waiting reason somewhere in the job page. (Show queueing duration and running duration in job detail #3847)
• Archive logs to elastic search
• Add retry count to task and show retry history (Show per task retry count on job detail page #3938)
• Exec to job containers on webportal
• persist service log (persistent log for PAI services #2129)
• differentiate opportunistic job and normal jobs in VC usage page.
• improve user group adding to VC experience
• improve HiveD configuration experience
• Show job priority in webportal
• introduce a new job state "initialization" (A job (or a pod of a job) may get stuck in a state neither running nor waiting #4141)
• Admin home page only showing abnormal jobs, should show my jobs by default

[fanyangCS]

#4225

[scarlett2018]

made an internal release.