Optimize Chocolate Suggestion

[andreyvelich]

I changed Chocolate Suggestion to be consistent with other Suggestions (Hyperopt, skopt).

1. Chocolate suggestion should analyse search space only for the first run.
2. I changed mechanism of saving information to sqlite3 DB. We don't need to override all Trial data every GetSuggestions call. We just update current DB data with the new Trial loss and Trial name for appropriate _chocolate_id. For this, I use list of recorded dictionaries self.created_trials variable. I update this list when receive new information about Trials.
3. Because of that, Chocolate sampler generates correct _chocolate_id for each new assignment and Chocolate algorithms should work correct. User can check data in sqlite3 DB.
4. Added few logs steps.
5. Removed warnings from SQLAlchemy about different threads by adding: check_same_thread=False flag.

I tested this Suggestion on grid, chocolate-quasirandom and chocolate-mocmaes. All Experiments were succeeded. Also, I tested grid on 150 Trials and return Experiment functionality.

/assign @gaocegege
/cc @johnugeorge

While testing I found one thing. In very rare cases, Katib controller calls GetSuggestion and doesn't create appropriate jobs.

It might be because of these sequence of actions in the controller:

1. Katib controller calls GetSuggestion with request_number = 1.
2. Katib controller gets response from GetSuggestion
3. Katib controller gets these errors. grid-lg9jzkgw.- Previously created Trial.

"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"suggestion-controller","request":"kubeflow/grid","error":"Operation cannot be fulfilled on suggestions.kubeflow.org \"grid\": the object has been modified; please apply your changes to the latest version and try again"
"logger": "trial-controller","msg":"Reconcile trial error","Trial":"kubeflow/grid-lg9jzkgw","error":"jobs.batch \"grid-lg9jzkgw\" not found"

4. At the same time Katib controller calls GetSuggestion with request_number = 2, because activeCount<parallelCount and got new suggestion with 2 Trials.
5. Katib controller created new Trials from the latest GetSuggestion response.

So we lose the first suggested Trial.

[kubeflow-bot]

This change is 

[andreyvelich]

/retest

[andreyvelich]

/retest

[andreyvelich]

/retest

[andreyvelich]

/retest

[gaocegege]

/retest

[andreyvelich]

/retest

[andreyvelich]

/retest

[andreyvelich]

/retest

[gaocegege]

/retest

[andreyvelich]

CI passed. @gaocegege Can you take a look again, please?

[gaocegege]

/lgtm
/approve

Thanks for your contribution! 🎉 👍

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment