Failed Eval when model training is in progress

[zhangpengshan]

We'd better to have the feature when in training but can run eval in parallel. Training is usually a long time running job and sometimes in the middle of running, eval can be leveraged to check model performance till checkpointed model.

2020-08-13 08:38:55: ERROR EvalModelProcessor [Eval-MAR] - Exception in eval:
java.io.FileNotFoundException: File models does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:444)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1548)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1591)
at ml.shifu.shifu.util.ModelSpecLoaderUtils.findModels(ModelSpecLoaderUtils.java:541)
at ml.shifu.shifu.util.ModelSpecLoaderUtils.locateBasicModels(ModelSpecLoaderUtils.java:359)
at ml.shifu.shifu.util.ModelSpecLoaderUtils.loadBasicModels(ModelSpecLoaderUtils.java:217)
at ml.shifu.shifu.core.processor.EvalModelProcessor.validateEvalColumnConfig(EvalModelProcessor.java:794)
at ml.shifu.shifu.core.processor.EvalModelProcessor.runEval(EvalModelProcessor.java:849)
at ml.shifu.shifu.core.processor.EvalModelProcessor.access$200(EvalModelProcessor.java:57)
at ml.shifu.shifu.core.processor.EvalModelProcessor$2.run(EvalModelProcessor.java:695)