Relationship between maxTrialNum and population_size

[martsalz]

Short summary about the issue/question:

What is the difference between maxTrialNum and population_size in the Evolution tuner?
Assuming: maxTrialNum = 5 and population_size = 10

Is there an initial population of 10 available for each of the 5 experiments?

[xuehui1991]

Hello @martsalz.
The maxTrialNum in config means 'maxTrialNum specifies the max number of trial jobs created by NNI, including succeeded and failed jobs.' Usually, we will set a large number, like 1000. You could reference here

And the population_size means the initial size of the population(trial num) in evolution tuner. For example, if population_size is 5, means the evolution tuner will randomly generate 5 trials as its initial population, then evolve from this population by mutation.

If you set maxTrialNum = 5 and population_size = 10, I guess it won't work.

Thank you for your questions~ And feel free to ask me anything.

[martsalz]

Thanks.

At which point is an experiment completed? Is there a maximum number of generations per experiment that can be set?

[xuehui1991]

Hello @martsalz. The maxTrialNum is the max number of an experiment. So if the trial number exceeds the maxTrialNum, the experiment will be stopped.

Thank you.

[martsalz]

At which point is an trial completed? Is there a maximum number of generations per trial that can be set? (for Evolution as tuner)

[xuehui1991]

I am not sure I understand your question totally. But I try to answer it.

In nni, trial code is written by users. Which means you get params(or config) generated by tuner, and used in your trial code. Then return the metric(like acc) to tuner.

So the trial completed when execute your all trial code.

"Is there a maximum number of generations per trial that can be set" --> For this question, if you want to ask how to set the number of mutation trial in one iteration, I think you could set trialConcurrency in config.yml. Which could specify the max num of trial trials run simultaneously.

Thank you~

[martsalz]

Assume Evolution is selected as tuner and population_size is specified.

Models are trained -> individuals are selected from the population -> crossover / mutation -> insert into the population.

How often is this process performed?

Thanks

[leelaylay]

@martsalz
maxTrialNum means the max number of trial jobs created by NNI.
trialConcurrency means the max num of trial jobs run simultaneously.

The process of Naive Evolution Tuner:
Initial population -> models are trained -> new individuals are inserted into the population -> individuals are selected from the population (random select 2 individuals from the population, the individual with better result can crossover and then it pops up)-> crossover / mutation -> generate parameter for model to train -> models are trained -> ... (loop)

NNI does mutation when it generates parameter from config so the frequency of this process is related to trialConcurrency. The source code of the mutation is here

The implementation is based on the paper Large-Scale Evolution of Image Classifiers.
Also, you can refer to the document of NNI for more details.