e2e tests sometimes fail because we try to create clusterrolebinding before the etcd server is ready

[wallrj]

Test failed in #116

• https://travis-ci.org/jetstack/navigator/jobs/299702643
• log.txt

+kubectl get nodes
The connection to the server 127.0.0.1:8443 was refused - did you specify the right host or port?
++date +%s
+local current_time=1510242196
+local remaining_time=288
+[[ 288 -lt 0 ]]
+local sleep_time=10
+[[ 288 -lt 10 ]]
+sleep 10
+true
+kubectl get nodes
No resources found.
+return 0
+kubectl create clusterrolebinding cluster-admin:kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:default
Error from server: client: etcd member http://0.0.0.0:2379 has no leader

/kind bug

Obviously the test for kubectl get nodes isn't sufficient.
We need to find some other way to know that the API is ready to accept configuration changes.
Or just retry the configuration change until it works.

[munnerz]

Nice find.

I think for now, retrying this with a 1 minute timeout or something will work.
Otherwise, a kubectl get componentstatus should return ok/healthy for etcd. But it may be difficult to properly parse the output of this command reliably.