Migrate protocol spec (#4307)

* Migrate protocol spec

Migrate pai protocol spec to https://github.com/microsoft/openpai-protocol.

* Fix broken links

Fix broken links.

contrib/pai_vscode/documentation/edit_yaml_job_config.md

@@ -1,8 +1,8 @@
 # OpenPAI job config file edit features
 
-In OpenPAI, all jobs are represented by YAML, a markup language.  
-Base on VSCode editor [IntelliSense](https://code.visualstudio.com/docs/editor/intellisense) and [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), OpenPAI VS Code Client support some features to improve user experience for editing job config file.  
-For more details about the protocol of OpenPAI job, please refer to [PAI Job Protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml).
+In OpenPAI, all jobs are represented by YAML, a markup language.
+Base on VSCode editor [IntelliSense](https://code.visualstudio.com/docs/editor/intellisense) and [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), OpenPAI VS Code Client support some features to improve user experience for editing job config file.
+For more details about the protocol of OpenPAI job, please refer to [PAI Job Protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml).
 
 - [OpenPAI job config file edit features](#openpai-job-config-file-edit-features)
   - [Create job config file](#create-job-config-file)
@@ -45,7 +45,7 @@ User can create a simple job config YAML file by below ways:
 
 ## YAML validation
 
-Use [PAI Job Protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml) to do the validation for job config file.
+Use [PAI Job Protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml) to do the validation for job config file.
 
 ### Whitch YAML file is a PAI job config
 
@@ -74,7 +74,7 @@ Base on VSCode editor [IntelliSense](https://code.visualstudio.com/docs/editor/i
 
 ### 2. Code snippets
 
-We provide several code snippets for VSCode YAML editor, user can use it to form their job config easily.  
+We provide several code snippets for VSCode YAML editor, user can use it to form their job config easily.
 It could be trigger by typing or right click in the editor and select `OpenPAI: Insert job config`
 
 #### Trigger by typing
@@ -87,7 +87,7 @@ It could be trigger by typing or right click in the editor and select `OpenPAI:
 
 ## Insert OpenPAI Runtime Plugin
 
-OpenPAI support some runtime plugins, such as SSH plugin, Storage plugin and Tensorboard plugin, user can config it in their job and setup the service.  
+OpenPAI support some runtime plugins, such as SSH plugin, Storage plugin and Tensorboard plugin, user can config it in their job and setup the service.
 Here is an example of runtime plugin config:
 
 ```yaml
@@ -111,7 +111,7 @@ We provide several ways to help user insert the plugin config in YAML file.
 
 ### Insert by code snippet
 
-The snippet `OpenPAI Runtime Plugin` will include `"com.microsoft.pai.runtimeplugin:"` line, and will ask user to select the first plugin type and generate it.  
+The snippet `OpenPAI Runtime Plugin` will include `"com.microsoft.pai.runtimeplugin:"` line, and will ask user to select the first plugin type and generate it.
 Typing `"- plugin:"` will trigger `OpenPAI: Insert a runtime plugin config`, it will help user to add other plugins.
 
 ![Insert by code snippet](../assets/auto_completion_runtime_plugin_snippet.gif)
@@ -124,6 +124,6 @@ Right click the editor and select `OpenPAI: Insert job config`, and select `Open
 
 ## Reference
 
-[PAI Job Protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml)  
-[Submit Jobs on OpenPAI](https://github.com/microsoft/pai/blob/master/docs/user/job_submission.md#job-workflow)  
+[PAI Job Protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml)
+[Submit Jobs on OpenPAI](https://github.com/microsoft/pai/blob/master/docs/user/job_submission.md#job-workflow)
 [Troubleshoot jobs](https://github.com/microsoft/pai/blob/master/docs/user/troubleshooting_job.md)

contrib/pai_vscode/documentation/submit_job.md

@@ -1,6 +1,6 @@
 # Submit job to OpenPAI by VSCode Extension
 
-This document is a tutorial for OpenPAI job submission on VSCode Extension.  
+This document is a tutorial for OpenPAI job submission on VSCode Extension.
 Before learning this document, make sure you have an OpenPAI cluster already, and already install VSCode.
 
 - [Submit job to OpenPAI by VSCode Extension](#submit-job-to-openpai-by-vscode-extension)
@@ -33,7 +33,7 @@ In VSCode Extension Marketplace search [OpenPAI VS Code Client](https://marketpl
 
 ## Submit a Hello World Job
 
-The job of OpenPAI defines how to execute code(s) and command(s) in specified environment(s). A job can be run on single node or distributedly.  
+The job of OpenPAI defines how to execute code(s) and command(s) in specified environment(s). A job can be run on single node or distributedly.
 The following process submits a model training job implemented by TensorFlow on CIFAR-10 dataset. It downloads data and code from internet and helps getting started with OpenPAI.
 
 ### Create a job config file
@@ -77,12 +77,12 @@ taskRoles:
       - python train_image_classifier.py --dataset_name=cifar10 --dataset_dir=/tmp/data --max_number_of_steps=1000
 ```
 
-The `OpenPAI VS Code Client` support some features to improve user experience for editing job config file, please refer to [OpenPAI job config file edit features](edit_yaml_job_config.md).  
+The `OpenPAI VS Code Client` support some features to improve user experience for editing job config file, please refer to [OpenPAI job config file edit features](edit_yaml_job_config.md).
 To learn more about this job, please refer to [Learn the Hello World Job](https://github.com/microsoft/pai/blob/master/docs/user/job_submission.md#Learn-the-Hello-World-Job)
 
 ### Submit it
 
-Finish editing the config file, save it and right click on the editor and select `Submit Job to PAI Cluster`.  
+Finish editing the config file, save it and right click on the editor and select `Submit Job to PAI Cluster`.
 After the information `Successfully submitted job.` pop up, you can click the `Open job page` button at right bottom corner, and view you job on website.
 
 ![Submit](../assets/submit.gif)
@@ -93,26 +93,26 @@ Most model training and other kinds of jobs need to transfer files between runni
 
 ### Teamwise Storage
 
-OpenPAI admin can define Team-wise storage through [Storage Plugin](https://github.com/microsoft/pai/tree/master/contrib/storage_plugin).  
+OpenPAI admin can define Team-wise storage through [Storage Plugin](https://github.com/microsoft/pai/tree/master/contrib/storage_plugin).
 User's job container can mount to the storage if user add it in job config file, for how to insert storage plugin into job config, please refer to [Insert OpenPAI Runtime Plugin](edit_yaml_job_config.md#Insert-OpenPAI-Runtime-Plugin).
 
 ### Storage explorer
 
-To manage user's data in Team-wise storage, `OpenPAI VS Code Client` support a `STORAGE EXPLORER` in vscode, User can manage data in the explorer.  
-We also support an `Auto Upload` feature in VSCode, the client will auto upload user's project file to the storage before submit job.  
+To manage user's data in Team-wise storage, `OpenPAI VS Code Client` support a `STORAGE EXPLORER` in vscode, User can manage data in the explorer.
+We also support an `Auto Upload` feature in VSCode, the client will auto upload user's project file to the storage before submit job.
 For more detail, refer to [Storage Explorer and Auto Upload](storage_explorer_and_auto_upload.md)
 
 ![Storage Explorer](../assets/storage.gif)
 
 ### Source code auto upload
 
-VSCode is a very powerful editor, user can use it to edit their source code, the auto upload feature make the source code to PAI job easily.  
+VSCode is a very powerful editor, user can use it to edit their source code, the auto upload feature make the source code to PAI job easily.
 For more detail, refer to [Storage Explorer and Auto Upload](storage_explorer_and_auto_upload.md#Auto-Upload)
 
 ![Source code auto upload](../assets/source_code_auto_upload.gif)
 
 ## Reference
 
-[PAI Job Protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml)  
-[Submit Jobs on OpenPAI](https://github.com/microsoft/pai/blob/master/docs/user/job_submission.md#job-workflow)  
+[PAI Job Protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml)
+[Submit Jobs on OpenPAI](https://github.com/microsoft/pai/blob/master/docs/user/job_submission.md#job-workflow)
 [Troubleshoot jobs](https://github.com/microsoft/pai/blob/master/docs/user/troubleshooting_job.md)

contrib/python-sdk/README.md

@@ -228,7 +228,7 @@ opai job list -a <cluster-alias> [<job-name>]
 
 ### How to submit a job from existing job config file
 
-If you already has a job config file, you could submit a job based on it directly. The job config file could be in the format of `json` or `yaml`, and it must be compatible with [job configuration specification v1](https://github.com/microsoft/pai/blob/master/docs/job_tutorial.md) or [pai-job-protocol v2](https://github.com/Microsoft/pai/blob/master/docs/pai-job-protocol.yaml).
+If you already has a job config file, you could submit a job based on it directly. The job config file could be in the format of `json` or `yaml`, and it must be compatible with [job configuration specification v1](https://github.com/microsoft/pai/blob/master/docs/job_tutorial.md) or [pai-job-protocol v2](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml).
 
 ```bash
 opai job submit -a <cluster-alias> <config-file>

contrib/python-sdk/README_zh_CN.md

@@ -207,7 +207,7 @@ opai job list -a <cluster-alias> [<job-name>]
 
 ### How to submit a job from existing job config file
 
-If you already has a job config file, you could submit a job based on it directly. The job config file could be in the format of `json` or `yaml`, and it must be compatible with [job configuration specification v1](https://github.com/microsoft/pai/blob/master/docs/job_tutorial.md) or [pai-job-protocol v2](https://github.com/Microsoft/pai/blob/master/docs/pai-job-protocol.yaml).
+If you already has a job config file, you could submit a job based on it directly. The job config file could be in the format of `json` or `yaml`, and it must be compatible with [job configuration specification v1](https://github.com/microsoft/pai/blob/master/docs/job_tutorial.md) or [pai-job-protocol v2](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml).
 
 ```bash
 opai job submit -a <cluster-alias> <config-file>

contrib/python-sdk/docs/scenarios-and-user-stories.md

@@ -4,7 +4,7 @@
 
 - **User can easily access `OpenPAI` resources in scripts (`Python` or `Shell`) and `Jupyter` notebooks**
 
-The SDK provides classes to describe the clusters (`openpaisdk.core.Cluster`) and jobs (`openpaisdk.job.Job`). The Cluster class wraps necessary REST apis for convenient operations. The Job class is an implementation of the [protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml), with which user can easily organize (add or edit) the content of job `yaml` and `json` configuration.
+The SDK provides classes to describe the clusters (`openpaisdk.core.Cluster`) and jobs (`openpaisdk.job.Job`). The Cluster class wraps necessary REST apis for convenient operations. The Job class is an implementation of the [protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml), with which user can easily organize (add or edit) the content of job `yaml` and `json` configuration.
 
 Besides the wrapping of APIs, the SDK also provides functions to facilitate user to utilize `OpenPAI`. Such functions includes *cluster management*, *storage accessing*, *execution environment detection (local or in a job container)*.
 

contrib/python-sdk/docs/scenarios-and-user-stories_zh_CN.md

@@ -4,7 +4,7 @@
 
 - **User can easily access `OpenPAI` resources in scripts (`Python` or `Shell`) and `Jupyter` notebooks**
 
-The SDK provides classes to describe the clusters (`openpaisdk.core.Cluster`) and jobs (`openpaisdk.job.Job`). The Cluster class wraps necessary REST apis for convenient operations. The Job class is an implementation of the [protocol](https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml), with which user can easily organize (add or edit) the content of job `yaml` and `json` configuration.
+The SDK provides classes to describe the clusters (`openpaisdk.core.Cluster`) and jobs (`openpaisdk.job.Job`). The Cluster class wraps necessary REST apis for convenient operations. The Job class is an implementation of the [protocol](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml), with which user can easily organize (add or edit) the content of job `yaml` and `json` configuration.
 
 Besides the wrapping of APIs, the SDK also provides functions to facilitate user to utilize `OpenPAI`. Such functions includes *cluster management*, *storage accessing*, *execution environment detection (local or in a job container)*.
 

contrib/python-sdk/openpaisdk/job.py

@@ -114,7 +114,7 @@ def parse_list(lst: List[str]):
 
 class Job:
     """
-    the data structure and methods to describe a job compatible with https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml
+    the data structure and methods to describe a job compatible with https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml
     external methods:
     - I/O
         - save(...) / load(...): store and restore to the disk
@@ -131,7 +131,7 @@ class Job:
     """
 
     def __init__(self, name: str=None, **kwargs):
-        self.protocol = dict()  # follow the schema of https://github.com/microsoft/pai/blob/master/docs/pai-job-protocol.yaml
+        self.protocol = dict()  # follow the schema of https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml
         self._client = None  # cluster client
         self.new(name, **kwargs)
 

contrib/submit-job-v2/README.md

@@ -8,7 +8,7 @@ This plugin is used to submit PAI job v2 on web portal.
 
 User can upload a job v2 yaml file from disk, choose a job v2 from marketplace, or use the submission form to fill in a job v2 config.
 
-Please refer to [PAI protocol spec](../../docs/pai-job-protocol.yaml) for more details.
+Please refer to [PAI protocol spec](https://github.com/microsoft/openpai-protocol/blob/master/schemas/v2/schema.yaml) for more details.
 
 ## Build