Kube-controller and kube-scheduler not working on kubernetes v1.18.8

[sumitKash]

What keywords did you search in kubeadm issues before filing this one?

Yes.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:10:16Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Environment: Production

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: On premise VMs

• OS (e.g. from /etc/os-release): VERSION="18.04.4 LTS (Bionic Beaver)"

• Kernel (e.g. uname -a): Linux mrprodkubesmaster2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

Installed Kubernetes version 1.18.8 on production server using kubeadm. I saw no error during the installtion. All the pods in kube-system are running fine with out any frequent restarts. However when i get error when i run below command i get error:

  kubectl get cs
    NAME                 STATUS      MESSAGE                                                                                     ERROR
    controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
    scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
    etcd-0               Healthy     {"health":"true"}

While troubleshooting i found that the ports are not being listened.

sudo netstat -tlpn |grep kube
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      132584/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      133300/kube-proxy
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      197705/kube-control
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      213741/kube-schedul
tcp6       0      0 :::10250                :::*                    LISTEN      132584/kubelet
tcp6       0      0 :::6443                 :::*                    LISTEN      132941/kube-apiserv
tcp6       0      0 :::10256                :::*                    LISTEN      133300/kube-proxy

What you expected to happen?

The cluster status should be healthy and port 10251 and 10252 should be bind to controller and scheduler process. Below is the example of the developemnt cluster on v1.17.4 which is running without any issue.

kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}

sudo netstat -tlpn |grep 102
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      2141/kube-controlle
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      2209/kube-scheduler
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1230/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2668/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      2668/kube-proxy
tcp6       0      0 :::10250                :::*                    LISTEN      1230/kubelet
tcp6       0      0 :::10251                :::*                    LISTEN      2209/kube-scheduler
tcp6       0      0 :::10252                :::*                    LISTEN      2141/kube-controlle

How to reproduce it (as minimally and precisely as possible)?

Install kubernetes version v1.18.8 on Ubuntu 18.04 using kubeadm

Anything else we need to know?

My cluster setup has 3 Master and 3 Worker nodes with external kube-api loadbalancer. I am using conatinerd as a runtime.

[fabriziopandini]

/triage support
AFAIK components status is being deprecated, and recently started to report errors even if the underlying components are healthy. Let me search for the issue

[fabriziopandini]

kubernetes/enhancements#553
#2222
kubernetes/kubernetes#93342
kubernetes/kubernetes#93570

[sumitKash]

Thank you for the reply.
Even if i overlook the error from Kubectl get componentstatus command. I still don't understand why port 10251 and 10252 are not being listened for kube-controller and kube-scheduler. Is this a new change in Kubernetes version1.18.8?

[neolit123]

Yes CS was deprecaded in 1.19 but we also disabled insecure serving of these components for older versions. To restore that functionality you can remove the --port=0 from their manifests. Not recommended as this can expose their metrics to the whole internet. /close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

Yes CS was deprecaded in 1.19 but we also disabled insecure serving of
these components for older versions. To restore that functionality you can
remove the --port=0 from their manifests. Not recommended as this can
expose their metrics to the whole internet.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sumitKash]

Yes CS was deprecaded in 1.19 but we also disabled insecure serving of these components for older versions. To restore that functionality you can remove the --port=0 from their manifests. Not recommended as this can expose their metrics to the whole internet. /close

Thank you for the reply. I double checked it by commenting the --port=0. There are few scripts in our env that used kubectl get cs output to check status. Guess i need to change it.

[Abhishek627]

@neolit123 Sorry for commenting on this older post but what is the recommended way to check status of these components now ? Currently we are checking 10251/healthz and 10251/healthz while deploying k8s.

[neolit123]

you can use kubectl get po ... and check if the pods are Running.
https://stackoverflow.com/questions/45851945/how-to-get-status-of-a-particular-pod-or-container-kubectl-get-pods-using-json