Add validation for algortihms (a.k.a suggestions)

[gaocegege]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

We should validate algorithm settings in the suggestion services.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/priority p1
/kind feature

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.99

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen

[anencore94]

Could you explain more information about this issue please? Thanks in advance :) @andreyvelich

[andreyvelich]

Could you explain more information about this issue please? Thanks in advance :) @andreyvelich

Sure. We have ValidateAlgorithmSettings gRPC call which is used to validate spec.algorithmSettings.
For example, in hyperopt Suggestion: https://github.com/kubeflow/katib/blob/master/pkg/suggestion/v1beta1/hyperopt/service.py#L57.
We run this request once Suggestion is created: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/suggestion/suggestion_controller.go#L271.

If ValidateAlgorithmSettings fails, Katib controller changes Suggestion and Experiment status to failed.

We should implement this feature in each Suggestion which Katib supports.

In addition to that, we have pending PR to implement it in Chocolate suggestion: #1205.

We would appreciate if you could help us with the implementation for other Suggestions, for example: skopt: https://github.com/kubeflow/katib/tree/master/pkg/suggestion/v1beta1/skopt.

[anencore94]

Thank you for such a friendly and detailed reply 👍 @andreyvelich
Sure, I'd like to implement validation for another suggestion algorithms. I agree this feature needs to be implemented. I'll take a look and will send a PR soon.

[gaocegege]

Thanks!

[tenzen-y]

I take this issue about optuna.

[tenzen-y]

I take darts.

[tenzen-y]

From what I can see, we have completed implementing validation for all suggestions.
I think that we can close this issue. What do you think? @kubeflow/wg-automl-leads

[johnugeorge]

Looks great.

Thanks @tenzen-y and @anencore94

[johnugeorge]

Closing this issue as validations are added for all algorithms