metrics server install-kube-system fix causing intermittent hang on new clusters

[davidmccormick]

It looks like there has been an issue within 'install-kube-system' when the metrics service is enabled which was documented and fixed in issue: #1039 (comment). We are now finding that when we spin up some new clusters that they get stuck within 'install-kube-system' which is not completing and allowing 'cfn-signal' to fire.

We see it get stuck in the logs...

Jul 26 09:41:37 ip-10-4-5-95.us-west-2.compute.internal retry[2152]: Sleeping 3 seconds.
Jul 26 09:41:40 ip-10-4-5-95.us-west-2.compute.internal retry[2152]: Sleeping 3 seconds.
Jul 26 09:41:43 ip-10-4-5-95.us-west-2.compute.internal retry[2152]: Sleeping 3 seconds.

Looking at 'install-kube-system' it looks to be stuck in this code: -

# See https://github.com/kubernetes-incubator/kube-aws/issues/1039#issuecomment-348978375
if ks get apiservice v1beta1.metrics.k8s.io && ! ps ax | grep '[h]yperkube proxy'; then
  echo "apiserver is up but kube-proxy isn't up. We have likely encountered #1039."
  echo "Temporary deleting the v1beta1.metrics.k8s.io apiservice as a work-around for #1039"
  ks delete apiservice v1beta1.metrics.k8s.io

  echo Waiting until controller-manager stabilizes and it creates a kube-proxy pod.
  until ps ax | grep '[h]yperkube proxy'; do
      echo Sleeping 3 seconds.
      sleep 3
  done
  echo kube-proxy stared. apiserver should be responsive again.
fi

It seems that this code executes before the code that installs the kube-proxy and so gets stuck in the inner loop waiting for the proxy to come up.