Add Katib ROADMAP 2022/2023

[andreyvelich]

I created initial draft for Katib ROADMAP 2022/2023.
I tried to consolidated features that we implemented in 2022, since we didn't update the ROADMAP in 2022 (if I missed some significant features that we've done, please let me know).

Please let me know if you have better ideas how to name headers.

Let's discuss the items during next month and finalise the ROADMAP.

/cc @johnugeorge @gaocegege @tenzen-y @anencore94 @kimwnasptd @elenzio9 @orfeas-k @apo-ger @d-gol @shaowei-su @zhixian82 @fischor @keisuke-umezawa @jbottum @DnPlas @juliusvonkohout

cc @kubeflow/wg-training-leads

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: d-gol, shaowei-su, fischor, keisuke-umezawa, zhixian82.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

I created initial draft for Katib ROADMAP 2022/2023.
I tried to consolidated features that we implemented in 2022, since we didn't update the ROADMAP in 2022.

Please let me know if you have better ideas how to name headers.

Let's discuss the items during next month and finalise the ROADMAP.

/cc @johnugeorge @gaocegege @tenzen-y @anencore94 @kimwnasptd @elenzio9 @orfeas-k @apo-ger @d-gol @shaowei-su @zhixian82 @fischor @keisuke-umezawa @jbottum @DnPlas @juliusvonkohout

cc @kubeflow/wg-training-leads

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andreyvelich]

/hold for review

[tenzen-y]

Thank you for compiling our roadmap!

Also, I'm thinking of creating a separate CRD for the Nas might be better. Since currently Experiment CRD seems to be a bit complex.

[tenzen-y]

IIUC, we already supported the TPE a long time ago, right?

https://github.com/kubeflow/katib/blob/279f6794dc6e71c5efedf4ef997d6346f60fe833/examples/v1beta1/hp-tuning/tpe.yaml

[andreyvelich]

That's right @tenzen-y.
Since this ROADMAP for 2022/2023, I tried to outline features that we already implemented in 2022 there, so users will have overview of what we do in 2022-2023.

[tenzen-y]

Ah, I see.
That's great! Thanks for clarifying!

[andreyvelich]

Also, I'm thinking of creating a separate CRD for the Nas might be better. Since currently Experiment CRD seems to be a bit complex.

@tenzen-y Yes, we were discussing about this before, how to separate various AutoML features from the Kubernetes CRD perspective:

1. HyperParameter tuning
2. NAS
3. Auto Feature Engineering
4. Auto Model Compression

Then, it will be easier to maintain APIs for every aspect of AutoML.
I think, we need to decide on APIs structure before moving Katib Component to v1 version.

[tenzen-y]

Also, I'm thinking of creating a separate CRD for the Nas might be better. Since currently Experiment CRD seems to be a bit complex.

@tenzen-y Yes, we were discussing about this before, how to separate various AutoML features from the Kubernetes CRD perspective:

1. HyperParameter tuning
2. NAS
3. Auto Feature Engineering
4. Auto Model Compression

Then, it will be easier to maintain APIs for every aspect of AutoML. I think, we need to decide on APIs structure before moving Katib Component to v1 version.

Oh, I see. Thank you for letting me know :) It would be worth creating an issue for discussion.

[johnugeorge]

Also, I'm thinking of creating a separate CRD for the Nas might be better. Since currently Experiment CRD seems to be a bit complex.

@tenzen-y Yes, we were discussing about this before, how to separate various AutoML features from the Kubernetes CRD perspective:

1. HyperParameter tuning
2. NAS
3. Auto Feature Engineering
4. Auto Model Compression

Then, it will be easier to maintain APIs for every aspect of AutoML. I think, we need to decide on APIs structure before moving Katib Component to v1 version.

I feel, we need to make API changes in this release if we are planning to do so.
Example: Can "Experiment" resource always refer to HP tuning? Should we rename the API resource? One other option is to add explicity resource names to other AutoML features

[tenzen-y]

I feel, we need to make API changes in this release if we are planning to do so.

If we can collect contributors to do that, we can do that in this release. However, changing the API name is so significant work...

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gaocegege]

Should we merge this?

[andreyvelich]

Sure, I removed Katib CRDs naming (Experiment, Suggestion, Trial) from the ROADMAP, since we need to discuss what naming we are going to have in our V1 APIs version.
We discussed this initially during our latest community call.
I will create an issue for this soon.

If other items look good, we can merge it.
/assign @gaocegege @tenzen-y @johnugeorge

[gaocegege]

LGTM!

[tenzen-y]

Thanks Andrey!
/lgtm

[andreyvelich]

/hold cancel

[tenzen-y]

We need to set up CNI due to kubernetes/minikube#16143.

[andreyvelich]

Makes sense, let me add it back.
Not sure why we are getting the following error in our CI:

chmod: cannot access '/etc/cni/net.d': No such file or directory
Error: The process '/usr/bin/sudo' failed with exit code 1

Any ideas ?

[tenzen-y]

I'm not sure why the error occurs. How about upgrading the Ubuntu version?

katib/.github/workflows/e2e-test-enas-cifar10.yaml

Line 14 in 373f6e6

runs-on: ubuntu-20.04

[andreyvelich]

Let me try that, 23.04

[andreyvelich]

Alright 22.04 work, let me submit separate PR to update the ubuntu version for our E2Es.

[tenzen-y]

Thank you 🎉

[terrytangyuan]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

/hold

[andreyvelich]

/hold cancel

[tenzen-y]

/lgtm