Error while Running MNIST Example

[vivekbhadouria]

Hello All:

I tried to run MNIST example on Google Cloud Kubernetes Cluster using go run client-example.go -s 192.168.99.100:30678, following the steps proposed in the documentation, but I am getting the following error:
./client-example.go:31:27: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/kubeflow/katib/vendor/google.golang.org/grpc".ClientConn in argument to api.NewManagerClient

Any help will be greatly appreciated.

[YujiOshima]

@vivekbhadouria The address of GKE cluster probably not be 192.168.99.100. Please check the node IP of your cluster.
And GKE cannot access from outside in default config. You need to create Ingress or set ACL like gcloud compute firewall-rules create katibservice --allow tcp:30678.

[patzhang]

@YujiOshima I had the same error when running with minikube on Mac today.

$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

Also, the web UI at http://192.168.99.100:30080/projects can be loaded successfully.
Will the error related go dependencies or go setup? I installed minikube, go, VM with homebrew.
Any ideas? Thanks.

[YujiOshima]

@patzhang Could you show me the result of minikube service list and kubectl -n katib get pod ?

[patzhang]

@YujiOshima Sure.

❯ minikube service list
|-------------|-------------------------------------|-----------------------------|
|  NAMESPACE  |                NAME                 |             URL             |
|-------------|-------------------------------------|-----------------------------|
| default     | kubernetes                          | No node port                |
| katib       | modeldb-backend                     | No node port                |
| katib       | modeldb-db                          | No node port                |
| katib       | modeldb-frontend                    | http://192.168.99.100:30080 |
| katib       | vizier-core                         | http://192.168.99.100:30678 |
| katib       | vizier-db                           | No node port                |
| katib       | vizier-earlystopping-medianstopping | No node port                |
| katib       | vizier-suggestion-grid              | No node port                |
| katib       | vizier-suggestion-random            | No node port                |
| kube-system | kube-dns                            | No node port                |
| kube-system | kubernetes-dashboard                | http://192.168.99.100:30000 |
|-------------|-------------------------------------|-----------------------------|

❯ kubectl -n katib get pod
NAME                                                   READY     STATUS    RESTARTS   AGE
modeldb-backend-6855d95fb4-bxdgf                       1/1       Running   0          6h
modeldb-db-6cf5bb764-gfzhp                             1/1       Running   0          6h
modeldb-frontend-88b6b5468-7rkxp                       1/1       Running   0          6h
vizier-core-655f5b7669-7c9tk                           1/1       Running   2          6h
vizier-db-64557596dc-zcm2z                             1/1       Running   0          6h
vizier-earlystopping-medianstopping-846d9f75d8-72phr   1/1       Running   0          6h
vizier-suggestion-grid-df8d7959b-nzmsn                 1/1       Running   0          6h
vizier-suggestion-random-6b4d6db6-fwhbg                1/1       Running   0          6h

[YujiOshima]

Hmm... It looks katib components are correctly running.
Which version are you using? Latest master branch or v0.1.2-alpha? And using vendored packages?
That error isn't seen in my environment.

[patzhang]

Hi @YujiOshima, I think the problem I had is related to vendored packages. I am new to Go Lang, so I do not know how to use the vendor package. I am using go version go1.10.3 darwin/amd64.

I cloned the katib repo and run .deploy. Then I run the following commands:

~/DL_Cluster/katib/examples master
❯ go run client-example.go -a random
client-example.go:10:2: cannot find package "github.com/kubeflow/katib/pkg/api" in any of:
        /usr/local/go/src/github.com/kubeflow/katib/pkg/api (from $GOROOT)
        /Users/patrick.zhang/go/src/github.com/kubeflow/katib/pkg/api (from $GOPATH)
client-example.go:11:2: cannot find package "google.golang.org/grpc" in any of:
        /usr/local/go/src/google.golang.org/grpc (from $GOROOT)
        /Users/patrick.zhang/go/src/google.golang.org/grpc (from $GOPATH)
client-example.go:12:2: cannot find package "gopkg.in/yaml.v2" in any of:
        /usr/local/go/src/gopkg.in/yaml.v2 (from $GOROOT)
        /Users/patrick.zhang/go/src/gopkg.in/yaml.v2 (from $GOPATH)

~/DL_Cluster/katib/examples master
❯ go get github.com/kubeflow/katib/pkg/api

~/DL_Cluster/katib/examples master 18s
❯ go get google.golang.org/grpc
go get google.golang.org/grpc  8.29s user 4.01s system 52% cpu 23.534 total

~/DL_Cluster/katib/examples master 24s
❯ go get gopkg.in/yaml.v2

~/DL_Cluster/katib/examples master
❯ go run client-example.go -a random
# command-line-arguments
./client-example.go:31:27: cannot use conn (type *"google.golang.org/grpc".ClientConn) as type *"github.com/kubeflow/katib/vendor/google.golang.org/grpc".ClientConn in argument to api.NewManagerClient

However, if I go to my $GOPATH and run from there, it works.

~/go/src/github.com/kubeflow/katib/examples master
❯ go run client-example.go -a random
2018/06/11 22:01:18 Study ID b23fe285a3f4ca6f
... ...

1. Do I need to use go get to install the dependencies? Or can I tell Go to use the vendor packages?

2. Should I run client-example.go from the cloned repo or the go/src directory?

3. Another issue is the google doc for the details of the system flow for the client and katib components are not shared. https://github.com/kubeflow/katib/blob/master/examples/MinikubeDemo.md

Looking forward to your explanation. Thank you in advance.

[YujiOshima]

Hi @patzhang, you should source in $GOPATH.
And you don't need to go get all depended packages.
Go will search packages in vendor directory when the source is in $GOPATH.
https://golang.org/cmd/go/#hdr-Vendor_Directories

the google doc
Oh, sorry. I will fix it.

[patzhang]

Thanks, @YujiOshima I checked out the code under ~/go/src, so Go can pick up packages from the vendor directory now. However, I still a problem with the api package.

~/go/src/katib/examples master
❯ go run client-example.go -a random
client-example.go:10:2: cannot find package "github.com/kubeflow/katib/pkg/api" in any of:
        /Users/patrick.zhang/go/src/katib/vendor/github.com/kubeflow/katib/pkg/api (vendor tree)
        /usr/local/go/src/github.com/kubeflow/katib/pkg/api (from $GOROOT)
        /Users/patrick.zhang/go/src/github.com/kubeflow/katib/pkg/api (from $GOPATH)

Do I need to build the api package?

[YujiOshima]

Please put katib dir under ~/go/src/github.com/kubeflow.

$ mkdir -p ~/go/src/github.com/kubeflow
$ cd ~/go/src/github.com/kubeflow
$ git clone https://github.com/kubeflow/katib
$ cd katib/examples
$ go run client-example.go -a random

[patzhang]

@YujiOshima, thanks a lot for all the detailed commands. I finally got the example (release v0.1.2-alpha) running successfully. Because I am not familiar with Go, I did not expect to create the directory ~/go/src/github.com/kubeflow and check out the code there. It is worth adding these commands to MinikubeDemo.md, because it is useful to people who just want to run the demo.

I also tried the code in master for the MinikubeDemo, but the new changes using port-forward does not seem to work. I guess I missed some other setup in the cluster. Anyway, it could be something I can test for the next release.

Thanks again for all your prompt help.

[jlewi]

@YujiOshima Now that we use the StudyController and users no longer need to write and run go code; does that address the issues in this issue?

[YujiOshima]

We can close this.
/close

[k8s-ci-robot]

@YujiOshima: Closing this issue.

In response to this:

We can close this.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.