Create a basic example for using Katib

[jlewi]

Do we have a simple example that walks through using Katib with a simple model (e.g. mnist)?

I'd like a simple example that we can ask folks to follow to provide a friction log.

/cc @YujiOshima

[lluunn]

I was using https://github.com/kubeflow/katib/blob/master/examples/client-example.go

[YujiOshima]

I think @jlewi said about training script.
In the example @lluunn referred, we use mxnet python mnist example.
But it is inconvenient for users to find input and log format.
We may be good to store the training script example in this repo.
@jlewi Is my understanding correct?

[jlewi]

I think I'm looking for something higher level. I'd like instructions that walk people through using Katib. So the goal would be to show how customers can use Katib to do hyperparameter tuning for a simple problem (e.g. mnist). The algorithm could also be very simple (e.g. grid search). This example might refer to the pieces mentioned above. So my expectation is that the instructions would be something like the following

1. Deploy Kubeflow
2. Enable/Deploy Kubeflow
3. Run grid search for sample problem (e.g. mnist)
4. Open the UI to browse the results.

[jlewi]

@ankushagarwal Will investigate to see what we can do in the next sprint to integrate with TFJob and create an example.

[jlewi]

https://github.com/kubeflow/kubeflow/pull/989/files

Describes running client-example.go.

A simple thing to do would be to turn that into a K8s job and include that as a prototype within the Katib package.

An advantage of running it on cluster should be that we don't have to port-forward any ports to make the client work.

[jlewi]

The GKE demo is actually using our GitHub issue summarization example
https://github.com/kubeflow/katib/blob/master/examples/GKEDemo/GKE-Demo.md

Looks like the code for the model is actually being duplicated to that location.

[lluunn]

Now that we have https://github.com/kubeflow/examples/tree/master/github_issue_summarization/hp-tune, what's the TODO here?

[jlewi]

@lluunn it doesn't actually work.

See for example kubeflow/examples#163 and kubeflow/examples#162

The go program you indicated doesn't successfully create the jobs for training (I think the issue was not supplying GCP credentials as a volume) and not extracting the metrics.

[wukong1992]

@jlewi
Now I want to test other examples. When I m going to write an example of a training model, I don't know how to define its log format, and katib is based on log to get accuracy.

[YujiOshima]

@wukong1992 Currently Yes Katib will collect metrics from stdout.
You should print logs {metrics name}={value} style.
For example when your objective value name is loss and the metrics are recall and precision, your training code should print like this.

epoch 1:
loss=0.3
recall=0.5
precision=0.4

epoch 2:
loss=0.2
recall=0.55
precision=0.5

Katib will collect all log of metrics.

[YujiOshima]

Since we have a simple example of random, grid and hyperband, we can close this.
/close

[k8s-ci-robot]

@YujiOshima: Closing this issue.

In response to this:

Since we have a simple example of random, grid and hyperband, we can close this.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.