Support pai and paiYarn trainingservice #1853

SparkSnail · 2019-12-15T14:29:16Z

test image sparksnail/nni:paiYarn

merge master

Update evolution doc (microsoft#1493)

merge master

augment pylintrc (microsoft#1643)

fix console.log (microsoft#1636)

merge master

Filter prune algo implementation (microsoft#1655)

merge master

…ai-nfs

merge master

…ai-nfs

merge master

…ai-k8s

chicm-ms · 2019-12-20T03:41:49Z

src/nni_manager/training_service/pai_base/paiBaseTrainingService.ts

+ * Training Service implementation for OpenPAI (Open Platform for AI)
+ * Refer https://github.com/Microsoft/pai for more info about OpenPAI
+ */
+@component.Singleton


Remove this @component.Singleton

chicm-ms · 2019-12-20T07:59:32Z

src/nni_manager/training_service/pai/paiJobRestServer.ts


    @Inject
-    private readonly paiTrainingService: PAITrainingService;
+    protected readonly paiTrainingService: PAITrainingService;


@Inject should be removed if it is assigned in constructor.

chicm-ms · 2019-12-20T09:36:57Z

src/nni_manager/training_service/pai/paiTrainingService.ts

-    private isMultiPhase: boolean = false;
-    private authFileHdfsPath: string | undefined = undefined;
-    private portList?: string | undefined;
+abstract class PAITrainingService implements TrainingService {


Remove this @component.Singleton

chicm-ms · 2019-12-20T09:40:46Z

src/nni_manager/training_service/pai/paiTrainingService.ts

-
-        return deferred.promise;
-    }
-
    public getClusterMetadata(key: string): Promise<string> {


It seems this getClusterMetadata is not implemented, then throw not implemented error.

chicm-ms · 2019-12-20T09:42:16Z

src/nni_manager/training_service/pai/paiTrainingService.ts

        try {
-            await restServer.stop();
+            await this.paiJobRestServer.stop();


Remove this unnecessary deferred

chicm-ms · 2019-12-20T09:44:43Z

src/nni_manager/training_service/pai/paiTrainingService.ts

-            if (restServer.getErrorMessage !== undefined) {
-                throw new Error(restServer.getErrorMessage);
+            if (this.paiJobRestServer === undefined) {
+                throw new Error('paiBaseJobRestServer not implemented!');


error message is not accurate, should be paiJobRestServer is undefined.

chicm-ms · 2019-12-20T09:54:22Z

src/nni_manager/training_service/pai/paiK8S/paiK8STrainingService.ts

+    }
+
+    public async setClusterMetadata(key: string, value: string): Promise<void> {
+        const deferred: Deferred<void> = new Deferred<void>();


Remove the deferred

chicm-ms · 2019-12-20T10:01:58Z

src/nni_manager/training_service/pai/paiK8S/paiK8STrainingService.ts

+                    await validateCodeDir(this.paiTrialConfig.codeDir);
+                } catch (error) {
+                    this.log.error(error);
+                    deferred.reject(new Error(error));


this error should have already been logged by rest server, no need to catch and re-throw, please check it.

chicm-ms · 2019-12-20T10:03:27Z

src/nni_manager/training_service/pai/paiK8S/paiK8STrainingService.ts

+        if (this.paiClusterConfig === undefined) {
+            throw new Error(`paiClusterConfig not initialized!`);
+        }
+        const deferred: Deferred<PAITrialJobDetail> = new Deferred<PAITrialJobDetail>();


Remove this deferred

chicm-ms · 2019-12-20T10:06:32Z

src/nni_manager/training_service/pai/paiTrainingService.ts

            return Promise.reject(`trial job ${trialJobId} not found`);
        }

-        return Promise.resolve(paiTrialJob);
+        return Promise.resolve(paiBaseTrialJob);


return paiBaseTrialJob directly

chicm-ms · 2019-12-20T10:07:25Z

src/nni_manager/training_service/pai/paiTrainingService.ts


-        if (paiTrialJob === undefined) {
+        if (paiBaseTrialJob === undefined) {


throw new Error to keep style consistent

chicm-ms · 2019-12-20T10:29:49Z

src/nni_manager/training_service/pai/paiYarn/paiYarnTrainingService.ts

+            trialJobDetail.status = 'FAILED'; // eslint-disable-line require-atomic-updates
+            deferred.resolve(true);
+
+            return deferred.promise;


return true directly

chicm-ms · 2019-12-22T06:04:26Z

tools/nni_cmd/launcher_utils.py

@@ -261,7 +262,7 @@ def validate_machine_list(experiment_config):

 def validate_pai_trial_conifg(experiment_config):
    '''validate the trial config in pai platform'''
-    if experiment_config.get('trainingServicePlatform') == 'pai':
+    if experiment_config.get('trainingServicePlatform') in ['pai', 'PAIYarn']:


It seems PAIYarn should be paiYarn

chicm-ms · 2019-12-22T06:12:01Z

src/nni_manager/training_service/pai/paiYarn/paiYarnTrainingService.ts

+        if (this.paiClusterConfig === undefined) {
+            throw new Error(`paiBaseClusterConfig not initialized!`);
+        }
+        const deferred: Deferred<PAITrialJobDetail> = new Deferred<PAITrialJobDetail>();


Remove this deferred.

chicm-ms · 2019-12-22T06:12:56Z

src/nni_manager/training_service/pai/paiYarn/paiYarnTrainingService.ts

+                // Validate to make sure codeDir doesn't have too many files
+                try {
+                    await validateCodeDir(this.paiTrialConfig.codeDir);
+                } catch (error) {


Remove this try catch

chicm-ms · 2019-12-22T06:13:18Z

src/nni_manager/training_service/pai/paiYarn/paiYarnTrainingService.ts

+    }
+
+    public async setClusterMetadata(key: string, value: string): Promise<void> {
+        const deferred: Deferred<void> = new Deferred<void>();


Remove this deferred.

chicm-ms · 2019-12-22T06:14:53Z

src/nni_manager/training_service/pai/paiYarn/paiYarnTrainingService.ts

+    protected postParameterFileMeta(parameterFileMeta: ParameterFileMeta): Promise<void> {
+        const deferred: Deferred<void> = new Deferred<void>();
+        if (this.paiJobRestServer === undefined) {
+            throw new Error('paiBaseJobRestServer not implemented!');


Error messge inaccurate.

SparkSnail and others added 28 commits August 6, 2019 18:58

Merge pull request #200 from microsoft/master

704b50e

merge master

Merge pull request #204 from microsoft/master

5b0034e

merge master

Merge pull request #205 from microsoft/master

8fe2588

Update evolution doc (microsoft#1493)

Merge pull request #206 from microsoft/master

9fae194

merge master

Merge pull request #207 from microsoft/master

c785655

merge master

Merge pull request #208 from microsoft/master

2f5272c

merge master

Merge pull request #209 from microsoft/master

1892bc2

augment pylintrc (microsoft#1643)

Merge pull request #210 from microsoft/master

7c1ab11

fix console.log (microsoft#1636)

Merge pull request #211 from microsoft/master

8c203f3

merge master

check pylint for nni_cmd

d7a62f6

fix id error

e259d10

Merge pull request #212 from microsoft/master

4997295

merge master

Merge pull request #213 from microsoft/master

c037a7c

merge master

init

f9d3312

Merge pull request #214 from microsoft/master

7620e7c

merge master

Merge pull request #215 from microsoft/master

d16dbe9

Filter prune algo implementation (microsoft#1655)

Merge pull request #216 from microsoft/master

9ce751d

merge master

Merge branch 'master' of https://github.com/SparkSnail/nni into dev-p…

69cef11

…ai-nfs

Merge pull request #217 from microsoft/master

a0846f2

merge master

Merge pull request #218 from microsoft/master

cd3a912

merge master

Merge pull request #219 from microsoft/master

32efaa3

merge master

Merge branch 'master' of https://github.com/SparkSnail/nni into dev-p…

90dcd3d

…ai-nfs

Merge pull request #220 from microsoft/master

543239c

merge master

fix conflict

9006669

basic work version

27032cc

fix typo

f0b4c4c

fix eslint

e7a9495

refactor code

8b69167

SparkSnail changed the title ~~Support PAI on K8S version~~ Support pai and paiYarn trainingservice Dec 15, 2019

add doc

6ac3e18

SparkSnail added 3 commits December 19, 2019 21:49

Merge branch 'master' of https://github.com/SparkSnail/nni into dev-p…

5943e43

…ai-k8s

fix comments

8fab408

fix pipeline

9a82fcc

chicm-ms reviewed Dec 20, 2019

View reviewed changes

refactor folder structure

217f454

chicm-ms reviewed Dec 20, 2019

View reviewed changes

SparkSnail added 2 commits December 20, 2019 17:37

remove paiYarnJobRestserver and paiK8SJobRestserver

cc802b0

remove singleton

14ba08a

chicm-ms reviewed Dec 20, 2019

View reviewed changes

SparkSnail added 2 commits December 20, 2019 19:53

add logPath

0bf5f69

fix comments

d09769e

chicm-ms reviewed Dec 22, 2019

View reviewed changes

fix comments

2b57042

xuehui1991 approved these changes Dec 23, 2019

View reviewed changes

chicm-ms approved these changes Dec 23, 2019

View reviewed changes

SparkSnail merged commit 9cbbf6f into microsoft:master Dec 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support pai and paiYarn trainingservice #1853

Support pai and paiYarn trainingservice #1853

SparkSnail commented Dec 15, 2019 •

edited

Loading

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 20, 2019

SparkSnail Dec 20, 2019

chicm-ms Dec 22, 2019

SparkSnail Dec 22, 2019

chicm-ms Dec 22, 2019

SparkSnail Dec 22, 2019

chicm-ms Dec 22, 2019

SparkSnail Dec 22, 2019

chicm-ms Dec 22, 2019

SparkSnail Dec 22, 2019

chicm-ms Dec 22, 2019

SparkSnail Dec 22, 2019


		if (paiTrialJob === undefined) {
		if (paiBaseTrialJob === undefined) {

Support pai and paiYarn trainingservice #1853

Support pai and paiYarn trainingservice #1853

Conversation

SparkSnail commented Dec 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkSnail commented Dec 15, 2019 •

edited

Loading