Create Optimizer in BO Suggestion only for the first run

[andreyvelich]

Fixes: #1039.

In this PR I added is_first_run flag.
We must create Optimizer instance using search_space only for the first GetSuggestions call.

/assign @gaocegege @johnugeorge

---

This change is 

[andreyvelich]

See comment: #1039 (comment)

[gaocegege]

/lgtm

It is a great optimization.

[andreyvelich]

@gaocegege Thanks!

[andreyvelich]

/hold
Hold until I test it on long-term experiment

[andreyvelich]

After testing BO on big amount of Trials I found few problems.

1. Since we are using one Optimizer instance during whole experiment, each GetSuggestion call will run tell method which will add information about all Trials to Optimizer. Xi and yi will have duplicate values (See attributes here: https://scikit-optimize.github.io/modules/generated/skopt.Optimizer.html#skopt.Optimizer) which is not correct and slow performance.
   To prevent it, I add recorded_trials dict, where key = loss value and value = List of Trial Assignment list for this value. Before running tell method, I am checking if this information already contains in the dict and I don't run tell method for the recorded Trials.

2. We don't need to iterate over trials if count of succeeded Trials is the same. For some cases, Katib controller can run GetSuggestion without new succeeded trials (For Example, some trials failed). We don't need to run tell method in that situation.

3. I tried to change ask method with n_points=request_number, see (https://scikit-optimize.github.io/modules/generated/skopt.Optimizer.html#skopt.Optimizer.ask). It indicates how many Trials do we want to create. Unfortunately, after 35 Trials this method takes so much time to run (more than 1 minute). I am not sure how we can fix it. Any thoughts @gaocegege ?

Also, I added few logging prints in BO.

/hold cancel

/cc @johnugeorge

[andreyvelich]

According to the first point, instead of map it is easier to have list of recorded trials name, since they are unique.
I just forgot that we send Trial name to the Suggestion.

[andreyvelich]

/retest

[johnugeorge]

/retest

[gaocegege]

/lgtm

[ever-cheng]

Doesn't SkoptService exist as a singleton instance for the system, to accept all GetSuggestions request?
How can it accept experiment change if it was bound to a fixed base_service?
if self.is_first_run:
self.base_service = BaseSkoptService(...)
...
self.is_first_run = False

[johnugeorge]

@ever-cheng No. algorithm service is deployed per experiment

[johnugeorge]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment