Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

getIPV4Address failed when using PAIK8S mode #2008

Closed
xuehui1991 opened this issue Feb 6, 2020 · 3 comments
Closed

getIPV4Address failed when using PAIK8S mode #2008

xuehui1991 opened this issue Feb 6, 2020 · 3 comments
Assignees

Comments

@xuehui1991
Copy link
Contributor

xuehui1991 commented Feb 6, 2020

When using the PAIK8S mode, I got some error:

Training service error:
getIPV4Address() fail because os.networkinterfaces().eth0 is undefined.

By the way, I already set the "nniManagerIp" in config correctly.

Short summary about the issue/question:

Brief what process you are following:

How to reproduce it:

nni Environment:

  • nni version: 1.3
  • nni mode(local|pai|remote): paik8s
  • OS: linux
  • python version: .36
  • is conda or virtualenv used?: conda
  • is running in docker?: yes

need to update document(yes/no):

Anything else we need to know:

@SparkSnail
Copy link
Contributor

SparkSnail commented Feb 7, 2020

This error is caused by the python environment. There are two NNI environment in user's machine, and one env is for 'local python', one is for 'virtual environment python', the two environment set nnictl's path to the same place, and the 'local python' nnictl path override the 'virtual environment python' nnictl path. When user use nnictl command, the system will always use 'local python' nnictl, while user want to use 'virtual environment python' nnictl.

@ICEORY
Copy link

ICEORY commented Feb 8, 2020

Hi, I also get the same error: getIPV4Address() fail because os.networkinterfaces().eth0 is undefined; I also test nni 1.2, and it works fine.

log

[2/8/2020, 6:20:18 AM] INFO [ 'NNIManager setClusterMetadata, key: nni_manager_ip, value: {"nniManagerIp":"10.10.1.139"}' ]
[2/8/2020, 6:20:18 AM] ERROR [ 'Uknown key: nni_manager_ip' ]
{ Error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined.
    at Object.getIPV4Address (/usr/local/nni/common/utils.js:183:15)
    at PAIK8STrainingService.submitTrialJobToPAI (/usr/local/nni/training_service/pai/paiK8S/paiK8STrainingService.js:170:103)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  name: '',
  cause:
   Error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined.
       at Object.getIPV4Address (/usr/local/nni/common/utils.js:183:15)
       at PAIK8STrainingService.submitTrialJobToPAI (/usr/local/nni/training_service/pai/paiK8S/paiK8STrainingService.js:170:103)
       at process._tickCallback (internal/process/next_tick.js:68:7) }

environment:

  • nni 1.3
  • python 3.6
  • openpai 0.14
  • ubuntu 16.04
  • running in docker
  • without anaconda

@SparkSnail
Copy link
Contributor

Hi @iceroy, this issue is fixed by #1987, will release this fix on v1.4.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants