Unable to enable dashboard on k8s v1.14.0 (addon-manager misbehaving?)

[LarsKumbier]

Current master seems to be unable to enable the dashboard addon with kubernetes v1.14.0.

Reproduce steps:

• build minikube from master (commit tested: f74751c)
• provision a cluster: minikube start --vm-driver=kvm2 --kubernetes-version v1.14.0
• try to enable the dashboard: minimube dashboard --v=7

$ minikube dashboard --v=7
Found binary path at /usr/local/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:41149
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
-   Enabling dashboard ...
Found binary path at /usr/local/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:37685
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
Found binary path at /usr/local/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:40377
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Making call to close driver server
(minikube) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(minikube) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
-   Verifying dashboard health ... Temporary Error: Error getting service kubernetes-dashboard: services "kubernetes-dashboard" not found
Temporary Error: Error getting service kubernetes-dashboard: services "kubernetes-dashboard" not found
Temporary Error: Error getting service kubernetes-dashboard: services "kubernetes-dashboard" not found                                                                                                      
[...last message repeated a few dozen times...]

Minikube logs

minikube.log

Versions

• Ubuntu 18.10
• docker v18.06.2-ce
• kvm2-driver

Notes

Problem has been confirmed with the same result by afbjorklund in minikube's slack channel.

[tstromberg]

What version of minikube is this? This should work from HEAD, but v1.14 support isn't complete in any of the releases.

[tstromberg]

I can confirm the behavior at head.

% kubectl get pods --all-namespaces --show-labels 
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE     LABELS
kube-system   coredns-fb8b8dccf-f7xrr            1/1     Running   0          5m34s   k8s-app=kube-dns,pod-template-hash=fb8b8dccf
kube-system   coredns-fb8b8dccf-hlsr9            1/1     Running   0          5m34s   k8s-app=kube-dns,pod-template-hash=fb8b8dccf
kube-system   etcd-minikube                      1/1     Running   0          4m41s   component=etcd,tier=control-plane
kube-system   kube-addon-manager-minikube        1/1     Running   0          4m17s   component=kube-addon-manager,kubernetes.io/minikube-addons=addon-manager,version=v8.6
kube-system   kube-apiserver-minikube            1/1     Running   0          4m40s   component=kube-apiserver,tier=control-plane
kube-system   kube-controller-manager-minikube   1/1     Running   0          4m42s   component=kube-controller-manager,tier=control-plane
kube-system   kube-proxy-55f6s                   1/1     Running   0          5m34s   controller-revision-hash=b7775b676,k8s-app=kube-proxy,pod-template-generation=1
kube-system   kube-scheduler-minikube            1/1     Running   0          4m19s   component=kube-scheduler,tier=control-plane

It appears that the dashboard pod isn't even being scheduled. I checked the logs for the addon manager and saw:

INFO: == Kubernetes addon ensure completed at 2019-03-26T17:28:51+0000 ==
INFO: == Reconciling with deprecated label ==
Error from server (NotFound): the server could not find the requested resource
INFO: == Reconciling with addon-manager label ==
Error from server (NotFound): the server could not find the requested resource
INFO: == Kubernetes addon reconcile completed at 2019-03-26T17:28:51+0000 ==
Error from server (NotFound): the server could not find the requested resource
WRN: == Failed to start /opt/namespace.yaml in namespace  at 2019-03-26T17:28:56+0000. 44 tries remaining. ==

$ docker ps -a | grep addon
f44d30813922        9c16409588eb           "/opt/kube-addons.sh"    10 minutes ago      Up 10 minutes                           k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_59c05d925476f9adfb2188d6da4bcbd1_0
39023a401795        k8s.gcr.io/pause:3.1   "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-addon-manager-minikube_kube-system_59c05d925476f9adfb2188d6da4bcbd1_0

$ docker logs f44d30813922
INFO: == Kubernetes addon manager started at 2019-03-26T17:19:37+0000 with ADDON_CHECK_INTERVAL_SEC=60 ==
Error from server (NotFound): the server could not find the requested resource
WRN: == Failed to start /opt/namespace.yaml in namespace  at 2019-03-26T17:19:40+0000. 99 tries remaining. ==
Error from server (NotFound): serviceaccounts "default" not found
WRN: == Error getting default service account, retry in 0.5 second ==
...
WRN: == Failed to start /opt/namespace.yaml in namespace  at 2019-03-26T17:28:56+0000. 44 tries remaining. ==

[afbjorklund]

Had the same issue, PR fixed it for me.