-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,7 +3,8 @@ | |
This page provides techniques for troubleshooting common Canonical Kubernetes | ||
issues. | ||
|
||
## Kubectl error: "dial tcp 127.0.0.1:6443: connect: connection refused" | ||
|
||
## Kubectl error: `dial tcp 127.0.0.1:6443: connect: connection refused` | ||
|
||
### Problem | ||
|
||
|
@@ -33,6 +34,51 @@ Use `k8s config` instead of `k8s kubectl config` to generate a kubeconfig file | |
that is valid for use on external machines. | ||
|
||
|
||
## Kubelet Error: `failed to initialize top level QOS containers: root container [kubepods] doesn't exist` | ||
Check failure on line 37 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 37 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
|
||
### Problem | ||
|
||
This is related to the `kubepods` cgroup not getting the cpuset controller up on the kubelet. | ||
Check failure on line 41 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 41 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
kubelet needs a feature from cgroup and the kernel may not be set up appropriately to provide | ||
Check failure on line 42 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 42 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
the cpuset feature. | ||
|
||
|
||
### Explanation | ||
|
||
An excellent deep-dive of the issue exists at [kubernetes/kubernetes #122955][kubernetes-122955]. | ||
Check failure on line 48 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 48 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
|
||
Commenter [@haircommander][] [states][kubernetes-122955-2020403422] | ||
> basically: we've figured out that this issue happens because libcontainer | ||
> doesn't initialize the cpuset cgroup for the kubepods slice when the kubelet | ||
> initially calls into it to do so. This happens because there isn't a cpuset | ||
> defined on the top level of the cgroup. however, we fail to validate all of | ||
> the cgroup controllers we need are present. It's possible this is a | ||
> limitation in the dbus API: how do you ask systemd to create a cgroup that is | ||
> effectively empty? | ||
> if we delegate: we are telling systemd to leave our cgroups alone, and not | ||
> remove the "unneeded" cpuset cgroup. | ||
|
||
### Solution | ||
|
||
This is in the process of being fixed upstream via [kubernetes/kuberetes #125923][kubernetes-125923]. | ||
Check failure on line 65 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 65 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
|
||
In the meantime, the best solution is to create a `Delegate=yes` configuration in systemd. | ||
Check failure on line 67 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
Check failure on line 67 in docs/src/snap/reference/troubleshooting.md GitHub Actions / markdown-lintLine length
|
||
|
||
```bash | ||
mkdir -p /etc/systemd/system/snap.k8s.kubelet.service.d | ||
cat /etc/systemd/system/snap.k8s.kubelet.service.d/delegate.conf <<EOF | ||
[Service] | ||
Delegate=yes | ||
EOF | ||
reboot | ||
``` | ||
|
||
<!-- LINKS --> | ||
|
||
[kubeconfig file]: https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/ | ||
[kubernetes-122955]: https://github.com/kubernetes/kubernetes/issues/122955 | ||
[kubernetes-125923]: https://github.com/kubernetes/kubernetes/pull/125923 | ||
[kubernetes-122955-2020403422]: https://github.com/kubernetes/kubernetes/issues/122955#issuecomment-2020403422 | ||
[@haircommander]: https://github.com/haircommander |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,5 +7,3 @@ k8s::common::setup_env | |
k8s::remove::containers | ||
|
||
k8s::remove::network | ||
|
||
k8s::remove::kubelet |