Multiphase job hangs when search space is exhuasted with NoMoreTrialError raised

[chicm-ms]

Short summary about the issue/question:

Brief what process you are following:

How to reproduce it:
Batch tuner and Gridsearch tuner raises NoMoreTrialError when search space exhuasted, but nni does not handle it properly for multiphase jobs.

I reproduced it using nni/test/config_test/multi_phase/multi_phase_batch.test.yml.
For example using a small search space with batch tuner:

{
    "test":
    {
        "_type" : "choice",
        "_value" : [1, 100]
    }
}

Then request more than 2 trials, multiphase job hangs.

[06/25/2019, 11:33:28 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.ReportMetricData], data: [OrderedDict([('type', 'REQUEST_PARAMETER'), ('sequence', 0), ('parameter_index', 1), ('trial_job_id', 'xl6az')])]
[06/25/2019, 11:33:28 AM] ERROR (nni.msg_dispatcher_base/Thread-1) no more parameters now.
Traceback (most recent call last):
File "/home/quzha/anaconda3/envs/nni/lib/python3.7/site-packages/nni/msg_dispatcher_base.py", line 102, in command_queue_worker
self.process_command(command, data)
File "/home/quzha/anaconda3/envs/nni/lib/python3.7/site-packages/nni/msg_dispatcher_base.py", line 160, in process_command
command_handlerscommand
File "/home/quzha/anaconda3/envs/nni/lib/python3.7/site-packages/nni/msg_dispatcher.py", line 146, in handle_report_metric_data
param = self.tuner.generate_parameters(param_id, trial_job_id=data['trial_job_id'])
File "/home/quzha/anaconda3/envs/nni/lib/python3.7/site-packages/nni/batch_tuner/batch_tuner.py", line 90, in generate_parameters
raise nni.NoMoreTrialError('no more parameters now.')
nni.NoMoreTrialError: no more parameters now.

nni Environment:

• nni version:
• nni mode(local|pai|remote):
• OS:
• python version:
• is conda or virtualenv used?:
• is running in docker?:

need to update document(yes/no):

Anything else we need to know:

[rabbit008]

#1539 fixed this issue