vizier-core stuck in CrashLoopBackoff due to failed pod checks

[pdmack]

#270 implemented gRPC health checking but I'm trying to make sense of why vizier-core (0.4.0) falls into CLB due to the failed readiness/liveness checks in my master+3 compute node deployment.

Events:
  Type     Reason     Age                  From                                 Message
  ----     ------     ----                 ----                                 -------
  Normal   Pulled     1h (x261 over 15h)   kubelet, node2.REDACTED.internal  Container image "gcr.io/kubeflow-images-public/katib/vizier-core:v0.4.0" already present on machine
  Warning  Unhealthy  46m (x800 over 15h)  kubelet, node2.REDACTED.internal  Readiness probe failed: timeout: failed to connect service ":6789" within 1s
  Warning  BackOff    6m (x3388 over 15h)  kubelet, node2.REDACTED.internal  Back-off restarting failed container
  Warning  Unhealthy  1m (x866 over 15h)   kubelet, node2.REDACTED.internal  Liveness probe failed: timeout: failed to connect service ":6789" within 1s

vizier-core                              NodePort    172.30.193.128   <none>        6789:30681/TCP      1d
vizier-core-rest                         ClusterIP   172.30.154.220   <none>        80/TCP              1d
vizier-db                                ClusterIP   172.30.8.147     <none>        3306/TCP            1d
vizier-suggestion-bayesianoptimization   ClusterIP   172.30.68.61     <none>        6789/TCP            1d
vizier-suggestion-grid                   ClusterIP   172.30.241.37    <none>        6789/TCP            1d
vizier-suggestion-hyperband              ClusterIP   172.30.251.108   <none>        6789/TCP            1d
vizier-suggestion-random                 ClusterIP   172.30.26.32     <none>        6789/TCP            1d

[johnugeorge]

@pdmack My deployment looks ok with 0.4.0. Did you change anything in your environment? Btw, are other Katib pods up?

[pdmack]

Yes the other vizier pods are running. Is this a local check within the pod that the vizier-manager is up and running?

[johnugeorge]

@pdmack
Yes. This is to check if the katib manager is ready to accept requests. https://github.com/kubeflow/katib/blob/master/cmd/manager/main.go#L346

1. Can you verify if vizier-db pod is up and active? I have seen a similar situation when the db pod was pending due to pvc issues
2. Can you check the logs of vizier-core pod? Can you see this print - https://github.com/kubeflow/katib/blob/master/cmd/manager/main.go#L343

[pdmack]

This is OpenShift 3.11 with generous permissions, but I'm thinking there's something subtle I'm missing. A standalone 3.11 env (AIO) doesn't exhibit this problem.

[pdmack]

@johnugeorge yeah turned out to be vizier-db was the culprit although it reported as Running. I got around this with permission modifications for the storage provisioner backing store.

I'll try to remember to file something for the health/ready checks on vizier-db.