Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

NNI does not support shmMB parameter in pai mode #823

Closed
gemfield opened this issue Mar 8, 2019 · 1 comment
Closed

NNI does not support shmMB parameter in pai mode #823

gemfield opened this issue Mar 8, 2019 · 1 comment
Assignees
Labels

Comments

@gemfield
Copy link

gemfield commented Mar 8, 2019

For a standalone PAI jobconfig, I can configure shmMB in the config yaml file on web, which works ok for my training;
However, for NNI PAI mode, if I configure shmMB in the exp_pai.yml, and run the command:

nnictl create --config exp_pai.yml

then the error occurs:

ERROR: Your config file is not correct, please check your config file content!
Key 'trial' error:
Wrong key 'shmMB' in {'gpuNum': 2, 'shmMB': 8196, 'command': 'bash test.sh', 'codeDir': '/bigdata/gemfield/.', 'outputDir': 'hdfs://192.168.1.121:9000/nni', 'dataDir': 'hdfs://192.168.1.121:9000/DataSet/data2016', 'memoryMB': 8196, 'cpuNum': 2, 'image': 'gemfield/pytorch4hdfs:1.0.1'}

If I decide to not set shmMB in exp_pai.yml, then the job will fail during the training phase because of the shm limit.

@gemfield gemfield changed the title NNI dose not support shmMB parameter in pai mode NNI does not support shmMB parameter in pai mode Mar 8, 2019
@SparkSnail
Copy link
Contributor

SparkSnail commented Mar 13, 2019

NNI does not support this param yet, we will support this feature in next release.
#842

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants