Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minikube failing to start with containerd and --wait=all flag #11101

Closed
spowelljr opened this issue Apr 14, 2021 · 4 comments · Fixed by #11209
Closed

minikube failing to start with containerd and --wait=all flag #11101

spowelljr opened this issue Apr 14, 2021 · 4 comments · Fixed by #11209
Assignees
@prezha
Labels
co/runtime/containerd kind/regression Categorizes issue or PR as related to a regression from a prior release. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@spowelljr
Copy link
Member

Starting minikube with containerd and without the --wait=all flag completes successfully.

But since #10985, if you start containerd with the --wait=all flag minikube fails to start with the following error

E0414 13:29:33.328256   76460 pod_ready.go:66] WaitExtra: waitPodCondition: timed out waiting for the condition
E0414 13:29:33.331163   76460 pod_ready.go:66] WaitExtra: waitPodCondition: error getting pod "coredns-74ff55c5b-ngsjr" in "kube-system" namespace: pods "coredns-74ff55c5b-ngsjr" not found

❌  Exiting due to GUEST_START: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns

The coredns pod is failing to start, the logs of the coredns pods are

  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Warning  FailedCreatePodSandBox  21m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f8ad333be454f7a02489bbc08856bd9032fab763d68069c0c785e32846796857": failed to set bridge addr: could not add IP address to "cni0": permission denied
@spowelljr spowelljr added co/runtime/containerd kind/regression Categorizes issue or PR as related to a regression from a prior release. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Apr 14, 2021
@medyagh
Copy link
Member

medyagh commented Apr 14, 2021

seems like when I specify the CNI manually it works

$ minikube start --container-runtime=containerd --cni=cilium
😄 minikube v1.19.0 on Darwin 11.2.3
✨ Automatically selected the docker driver. Other choices: hyperkit, parallels, virtualbox, ssh
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
> index.docker.io/kicbase/sta...: 357.67 MiB / 357.67 MiB 100.00% 25.24 Mi
❗ minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.20, but successfully downloaded kicbase/stable:v0.0.20 as a fallback image
🔥 Creating docker container (CPUs=2, Memory=4000MB) ...
📦 Preparing Kubernetes v1.20.2 on containerd 1.4.4 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring Cilium (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-dmj4w 0/1 Running 0 62s
kube-system cilium-operator-7c755f4594-gzljb 1/1 Running 0 62s
kube-system coredns-74ff55c5b-w9vtl 0/1 Running 0 62s
kube-system etcd-minikube 1/1 Running 0 73s
kube-system kube-apiserver-minikube 1/1 Running 0 73s
kube-system kube-controller-manager-minikube 1/1 Running 0 73s
kube-system kube-proxy-txzm2 1/1 Running 0 62s
kube-system kube-scheduler-minikube 0/1 Running 0 73s
kube-system storage-provisioner 1/1 Running 1 74s

@spowelljr
Copy link
Member Author

@medyagh The --wait=all flag wasn't passed

@medyagh
Copy link
Member

medyagh commented Apr 14, 2021 •
edited
Loading

so it seems like when we try with containerd and --wait=all it waits long long time and warns a lot and then Exits 0 succesfully.

could it be we are waiting on wrong things ? @prezha

$ minikube start --container-runtime=containerd --cni=cilium --wait=all
😄  minikube v1.19.0 on Darwin 11.2.3
✨  Automatically selected the docker driver. Other choices: hyperkit, parallels, virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > index.docker.io/kicbase/sta...: 357.67 MiB / 357.67 MiB  100.00% 26.23 Mi
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.20, but successfully downloaded kicbase/stable:v0.0.20 as a fallback image
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
📦  Preparing Kubernetes v1.20.2 on containerd 1.4.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Cilium (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
E0414 16:49:58.947751   94626 pod_ready.go:66] WaitExtra: waitPodCondition: error getting pod "coredns-74ff55c5b-nhk6w" in "kube-system" namespace: pods "coredns-74ff55c5b-nhk6w" not found
❌  Problems detected in kubelet:
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966843    1573 reflector.go:138] object-"kube-system"/"cilium-aws": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-aws" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966985    1573 reflector.go:138] object-"kube-system"/"cilium-operator-token-69qjh": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-operator-token-69qjh" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
❌  Problems detected in kubelet:
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966843    1573 reflector.go:138] object-"kube-system"/"cilium-aws": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-aws" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966985    1573 reflector.go:138] object-"kube-system"/"cilium-operator-token-69qjh": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-operator-token-69qjh" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
❌  Problems detected in kubelet:
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966843    1573 reflector.go:138] object-"kube-system"/"cilium-aws": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-aws" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
    Apr 14 23:48:48 minikube kubelet[1573]: E0414 23:48:48.966985    1573 reflector.go:138] object-"kube-system"/"cilium-operator-token-69qjh": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cilium-operator-token-69qjh" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'minikube' and this object
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ kc get pods
No resources found in default namespace.
16:50:38 medya/workspace/minikube
windows_ps ✓
$ kc get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   cilium-kslzh                       1/1     Running   0          112s
kube-system   cilium-operator-7c755f4594-zpc42   1/1     Running   0          112s
kube-system   coredns-74ff55c5b-9lgfb            1/1     Running   0          112s
kube-system   etcd-minikube                      1/1     Running   0          2m2s
kube-system   kube-apiserver-minikube            1/1     Running   0          2m2s
kube-system   kube-controller-manager-minikube   1/1     Running   0          2m2s
kube-system   kube-proxy-4cfkp                   1/1     Running   0          112s
kube-system   kube-scheduler-minikube            1/1     Running   0          2m2s
kube-system   storage-provisioner                1/1     Running   0          110s
16:50:40 medya/workspace/minikube

@prezha
Copy link
Contributor

prezha commented Apr 15, 2021 •
edited
Loading

@medyagh yes, you are right - we are waiting on wrong things:

according to your logs above, depending on the order we get in the list of (coredns) pods, we might end up waiting for the initially created ("coredns-74ff55c5b-nhk6w" in this case) pod that got terminated, instead of waiting for the newly created one that replaced it ("coredns-74ff55c5b-9lgfb" in this case); i'll fix that in the upcoming pr

i think that Problems detected in kubelet is a bit misleading here (ie, those errors in the logs are expected/irrelevant)

@prezha prezha mentioned this issue Apr 15, 2021
Closed
@prezha prezha self-assigned this Apr 15, 2021
@prezha prezha mentioned this issue Apr 29, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prezha prezha

Labels
co/runtime/containerd kind/regression Categorizes issue or PR as related to a regression from a prior release. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
3 participants
@medyagh @prezha @spowelljr