Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kind cluster on WSL2 does not restart when Docker restarts #3244

Open
inliquid opened this issue May 22, 2023 · 18 comments
Open

kind cluster on WSL2 does not restart when Docker restarts #3244

inliquid opened this issue May 22, 2023 · 18 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@inliquid
Copy link

What happened:

Hey! I have created a cluster on Ubuntu@WSL2, but found that it's not restarted when Docker restarts.

In order to start the cluster I have to run manually:

$ docker start kind-control-plane

I have also tried changing RestartPolicy to unless-stopped and always - it didn't work. There are a couple of containers with unless-stopped policy, and they all restart as expected, problem is only with kind-control-plane.

When I run docker start kind-control-plane everything seems fine, so the problem is only with restart, I don't see any failure.

What you expected to happen:

Kind cluster being started with Docker restart.

How to reproduce it (as minimally and precisely as possible):

If I'm correct:

  1. Install Ubuntu on latest WSL2@Windows 11
  2. Install Docker Desktop and configure it to integrate with Ubuntu@WSL2
    изображение
  3. Create kind cluster
  4. Restart Docker Desktop
  5. Check if kind cluster restarted

Anything else we need to know?:

Environment:

  • kind version: (use kind version):
$ kind version
kind v0.19.0 go1.20.4 linux/amd64
  • Runtime info: (use docker info or podman info):
$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.10.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 12
  Running: 4
  Paused: 0
  Stopped: 8
 Images: 13
 Server Version: dev
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 15.47GiB
 Name: docker-desktop
 ID: 6b560505-4d0d-44ec-bd08-ec5179f075b5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
  • OS (e.g. from /etc/os-release):
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy
  • Kubernetes version: (use kubectl version):
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-05-12T19:03:40Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
  • Any proxies or other special environment settings?:

No

@inliquid inliquid added the kind/bug Categorizes issue or PR as related to a bug. label May 22, 2023
@BenTheElder
Copy link
Member

We're going to need logs from when it doesn't start to see why it exited or if docker even started it

docker logs kind-control-plane + docker inspect kind-control-plane before your apply your workaround.

@BenTheElder
Copy link
Member

BenTheElder commented May 22, 2023

But generally speaking short of a docker bug failing to restart it for some reason, I'd expect the entrypoint failed and that happens when something is rather broken in the environment and it is by design that it doesn't constantly keep restarting if that happens.

@BenTheElder
Copy link
Member

Please also see the pinned issue about WSL2: #3180

@inliquid
Copy link
Author

inliquid commented May 22, 2023

$ docker logs kind-control-plane
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v1
WARN: cgroupns not enabled! Please use cgroup v2, or cgroup v1 with cgroupns enabled.
INFO: fix cgroup mounts for all subsystems
INFO: unmounting and removing /sys/fs/cgroup/misc
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.20.0.2
INFO: Detected old IPv4 address: 172.20.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
INFO: Detected old IPv6 address: fc00:f853:ccd:e793::2
systemd 247.3-7+deb11u2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization wsl.
Detected architecture x86-64.
Failed to create symlink /sys/fs/cgroup/net_prio: File exists
Failed to create symlink /sys/fs/cgroup/net_cls: File exists
Failed to create symlink /sys/fs/cgroup/cpu: File exists
Failed to create symlink /sys/fs/cgroup/cpuacct: File exists

Welcome to Debian GNU/Linux 11 (bullseye)!

Set hostname to <kind-control-plane>.
Queued start job for default target Graphical Interface.
[  OK  ] Created slice slice used to run Kubernetes / Kubelet.
[  OK  ] Created slice system-modprobe.slice.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Sockets.
         Mounting Huge Pages File System...
         Mounting Kernel Debug File System...
         Mounting Kernel Trace File System...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module fuse...
         Starting Journal Service...
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Trace File System.
modprobe@configfs.service: Succeeded.
[  OK  ] Finished Load Kernel Module configfs.
modprobe@fuse.service: Succeeded.
[  OK  ] Finished Load Kernel Module fuse.
systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
systemd-modules-load.service: Failed with result 'exit-code'.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Mounting FUSE Control File System...
         Starting Apply Kernel Variables...
         Starting Create System Users...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Finished Create System Users.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timers.
         Starting containerd container runtime...
         Starting kubelet: The Kubernetes Node Agent...
[  OK  ] Started containerd container runtime.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

@inliquid
Copy link
Author

$ docker inspect kind-control-plane
[
    {
        "Id": "fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be",
        "Created": "2023-05-18T12:44:00.481268157Z",
        "Path": "/usr/local/bin/entrypoint",
        "Args": [
            "/sbin/init"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 30047,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-05-22T20:11:58.662618999Z",
            "FinishedAt": "2023-05-22T12:11:04.70435659Z"
        },
        "Image": "sha256:b7d12ed662b873bd8510879c1846e87c7e676a79fefc93e17b2a52989d3ff42b",
        "ResolvConfPath": "/var/lib/docker/containers/fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be/hostname",
        "HostsPath": "/var/lib/docker/containers/fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be/hosts",
        "LogPath": "/var/lib/docker/containers/fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be/fca8f43a0a044c396ad39d43dc87d8ef67bd6cb4ce22e2a8c027884cb13d17be-json.log",
        "Name": "/kind-control-plane",
        "RestartCount": 0,
        "Driver": "stargz",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/lib/modules:/lib/modules:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "kind",
            "PortBindings": {
                "6443/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "38727"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined",
                "apparmor=unconfined",
                "label=disable"
            ],
            "Tmpfs": {
                "/run": "",
                "/tmp": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null,
            "Init": false
        },
        "GraphDriver": {
            "Data": null,
            "Name": "stargz"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/lib/modules",
                "Destination": "/lib/modules",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "4b8bcf626a9216a47e277c8223133fa6a682fc0666b49eed61323ff2ab0a25c8",
                "Source": "/var/lib/docker/volumes/4b8bcf626a9216a47e277c8223133fa6a682fc0666b49eed61323ff2ab0a25c8/_data",
                "Destination": "/var",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "kind-control-plane",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "6443/tcp": {}
            },
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER",
                "KUBECONFIG=/etc/kubernetes/admin.conf",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=docker"
            ],
            "Cmd": null,
            "Image": "kindest/node:v1.27.1@sha256:b7d12ed662b873bd8510879c1846e87c7e676a79fefc93e17b2a52989d3ff42b",
            "Volumes": {
                "/var": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "/usr/local/bin/entrypoint",
                "/sbin/init"
            ],
            "OnBuild": null,
            "Labels": {
                "desktop.docker.io/wsl-distro": "Ubuntu",
                "io.x-k8s.kind.cluster": "kind",
                "io.x-k8s.kind.role": "control-plane"
            },
            "StopSignal": "SIGRTMIN+3"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9399397f4d38a5017b748f692d4e13604ecb86c0f1f1062b4c47fb9e693d3130",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "6443/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "38727"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/9399397f4d38",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "fca8f43a0a04",
                        "kind-control-plane"
                    ],
                    "NetworkID": "4fa292a56e1f236a09ed21bcdc914c8c4e5685ea355913c7fc64635460327724",
                    "EndpointID": "900b1cecee25b706ba3c972185d3db74190feec64f3d779f26e47e68f9d03611",
                    "Gateway": "172.20.0.1",
                    "IPAddress": "172.20.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::2",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "02:42:ac:14:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

@inliquid
Copy link
Author

@BenTheElder thank you! I collected some outputs (see above), hope that helps.

@BenTheElder
Copy link
Member

Unfortunately these logs show that the container is running without exiting?

We need the logs / exit status from when it fails to start running.

I think some of the issues are fixed by either #3180 or #3241. For the latter it's not released yet and technically a breaking change but could be tested with https://kind.sigs.k8s.io/docs/user/quick-start/#installing-from-source

@inliquid
Copy link
Author

inliquid commented May 23, 2023

Unfortunately these logs show that the container is running without exiting?

Not sure, the problem is that container is not started on Docker Desktop restart no matter which RestartPolicy is set. At the same time, all other containers running along with it with unless-stopped or always are restarted normally.

So, technically, it could be logs from previous normal run, but they might not reflect the reason of kind not being restarted.

@BenTheElder
Copy link
Member

Not sure, the problem is that container is not started on Docker Desktop restart no matter which RestartPolicy is set. At the same time, all other containers running along with it with unless-stopped or always are restarted normally.

So, if it never started at all following host restart, that's not a kind bug, that's a bug in docker desktop or a behavior change in docker specifically.

But before saying docker desktop has a bug, I would guess that it could have started and failed, because the policy is to only start once.

If it is started once and failed, we need the logs from that.

Otherwise we need evidence that docker has broken this behavior and won't fix it before we can commit to changing our approach to something like always with the drawbacks that entails.

@BenTheElder
Copy link
Member

Are we sure these logs are immediately after docker desktop was restarted without manually starting the kind cluster again?

Because these logs appear normal, the container is in running state.

@BenTheElder BenTheElder added the triage/needs-information Indicates an issue needs more information in order to work on it. label May 23, 2023
@inliquid
Copy link
Author

Here is what I have immediately after restart (other containers running as expected):

$ docker inspect kind-control-plane | grep -i -A 1 restartpolicy
            "RestartPolicy": {
                "Name": "always",
$ docker ps -a | head -n 2
CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS                       PORTS                        NAMES
cf53894badac   kindest/node:v1.27.1      "/usr/local/bin/entr…"   33 minutes ago   Exited (128) 3 minutes ago   127.0.0.1:42097->6443/tcp    kind-control-plane
$ docker logs kind-control-plane
<...>

INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v1
WARN: cgroupns not enabled! Please use cgroup v2, or cgroup v1 with cgroupns enabled.
INFO: fix cgroup mounts for all subsystems
INFO: unmounting and removing /sys/fs/cgroup/misc
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.20.0.2
INFO: Detected old IPv4 address: 172.20.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
INFO: Detected old IPv6 address: fc00:f853:ccd:e793::2
systemd 247.3-7+deb11u2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization wsl.
Detected architecture x86-64.
Failed to create symlink /sys/fs/cgroup/cpuacct: File exists
Failed to create symlink /sys/fs/cgroup/cpu: File exists
Failed to create symlink /sys/fs/cgroup/net_prio: File exists
Failed to create symlink /sys/fs/cgroup/net_cls: File exists

Welcome to Debian GNU/Linux 11 (bullseye)!

Set hostname to <kind-control-plane>.
Queued start job for default target Graphical Interface.
[  OK  ] Created slice slice used to run Kubernetes / Kubelet.
[  OK  ] Created slice system-modprobe.slice.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Sockets.
         Mounting Huge Pages File System...
         Mounting Kernel Debug File System...
         Mounting Kernel Trace File System...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module fuse...
         Starting Journal Service...
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Trace File System.
modprobe@configfs.service: Succeeded.
[  OK  ] Finished Load Kernel Module configfs.
modprobe@fuse.service: Succeeded.
[  OK  ] Finished Load Kernel Module fuse.
systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
systemd-modules-load.service: Failed with result 'exit-code'.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Mounting FUSE Control File System...
         Starting Apply Kernel Variables...
         Starting Create System Users...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Journal Service.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timers.
         Starting containerd container runtime...
         Starting kubelet: The Kubernetes Node Agent...
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started containerd container runtime.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Update UTMP about System Runlevel Changes.

@BenTheElder
Copy link
Member

$ docker inspect kind-control-plane | grep -i -A 1 restartpolicy
"RestartPolicy": {
"Name": "always",

OK, this is with the modified container settings though because kind does not set always.

The logs don't make sense, maybe docker/for-linux#293

Have you checked the WSL version as mentioned in #3180 ?

@BenTheElder
Copy link
Member

The original post says:

Install Ubuntu on latest WSL2@Windows 11

But as far as I know, latest WSL2 is known to be broken on their end, in the pinned issue linked here.

@inliquid
Copy link
Author

OK, this is with the modified container settings though because kind does not set always.

Yes, default didn't work so I tried all of them and left "always".

Have you checked the WSL version as mentioned in #3180 ?

It's 1.2.5.0.

@BenTheElder
Copy link
Member

1.2.5.0 is known to be broken per follow-up from another user (and updated title etc), can you try the steps there to switch to 1.2.4.0?

1.2.0.0 was also broken, and there was no change on our end, just a fix in WSL2.

@inliquid
Copy link
Author

Interesting observation: sometimes kind-control-plane restarts fine.

@inliquid
Copy link
Author

@BenTheElder after several attempts I got this error message:

$ docker start kind-control-plane
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: failed to write "a *:* rwm": write /sys/fs/cgroup/devices/docker/cf53894badacc76043c19145026f5d7862652008b9b28a1fcd09d8f267c150ba/devices.allow: invalid argument: unknown
Error: failed to start containers: kind-control-plane

and after a couple retries kind has been started w/o error message. Can this help?

@BenTheElder
Copy link
Member

BenTheElder commented May 24, 2023

This is interesting, but can we confirm if it still happens on WSL2 1.2.4.0? As mentioned previously we have multiple reports of WSL2 shipping broken 1.2.X releases and 1.2.4.0 is considered working but 1.2.5.0 and 1.2.0.0 at least are not.

There are instructions in the linked and pinned issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Development

No branches or pull requests

2 participants