Microk8s 1.19.0 never ready, not working.

[WereCatf]

Installing microk8s with sudo snap install microk8s --classic appears to go fine, but microk8s status always reports as microk8s not running. microk8s kubectl get all --all-namespaces shows everything as not ready and pending.

Ubuntu 20.04.1 Desktop, network-interfaces are ens33, lo, virbr0 and virbr0-nic, so nothing particularly exciting there. The flannel - interface you'd get with an earlier microk8s - version is a no-show here.

Already tried the solution mentioned in #1554 but that doesn't work.
inspection-report-20200916_031614.tar.gz

[ktsakalozos]

Hi @WereCatf

In the logs of kubelet (journalctl -u snap.microk8s.daemon-kubelet) there are the following errors:

syys 16 03:16:10 ubuvm microk8s.daemon-kubelet[20094]: E0916 03:16:10.011947   20094 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": failed to get device for dir "/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": could not find device with major: 0, minor: 28 in cached partitions map.

and

syys 16 03:16:10 ubuvm microk8s.daemon-kubelet[20094]: F0916 03:16:10.141628   20094 kubelet.go:1296] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/snap/microk8s/common/var/lib/kubelet": could not find device with major: 0, minor: 28 in cached partitions map

I assume you are on btrfs and kubelet cannot query the stats of the underling filesystem. I wonder what our options are here? Is moving to another file system an option or do we need to further digg into this? I am not familiar with btrfs but this is a possibly related issue: [1]

[1] kubernetes/kubernetes#65204 (comment)

[WereCatf]

@ktsakalozos Wait, are you...are you serious? Using Btrfs breaks Microk8s?? Jesus Christ, that's definitely something I'd classify as a major issue.

[GavinB-hpe]

Exactly the same problem. / uses btrfs and I get the same errors in the kubelet logs. Ubuntu 18.04 LTS based system.

[ktsakalozos]

Hi @GavinB-hpe

Could you try one of the two suggestions recommended in #1587 (comment)

[GavinB-hpe]

Thanks @ktsakalozos

--feature-gates="LocalStorageCapacityIsolation=false"

Fixed things for me.

[ktsakalozos]

Do you think this is something it should be automated? Can we reliably detect the cases where this is needed?

[WereCatf]

@ktsakalozos Reliably detecting where this workaround is specifically absolutely needed, I dunno, but detecting cases where it might be needed could be as simple as: if [ $(df --output=fstype "/path/to/directory"|tail -n 1) = btrfs ]; then applyworkaround; fi

Personally, I don't think automating such a workaround should be done at all and instead the issue should be fixed properly at the source, but I don't really have a grasp of where it happens or who is in charge of the code and if they even care.

[ktsakalozos]

We could detect btrfs in microk8s inspect command and propose a fix [1]. We could also have a docs section in [2] to suggest such configurations.

[1] https://github.com/ubuntu/microk8s/blob/master/scripts/inspect.sh#L106
[2] https://microk8s.io/docs/troubleshooting#heading--common-issues

[exi]

Setting: --feature-gates="LocalStorageCapacityIsolation=false" does not fix it for me on a btrfs root filesystem.
Is there any workaround?

[exi]

To answer my own question: k8s 1.19 will not work, even with the feature-gates flag.
Downgrading to 1.18 solved the issue for me and my containers are starting with the feature-gates flag.

[liuxh0]

I tried feature-gates flag with microk8s 1.18/stable (v1.18.15), still not working.

[jcdutton]

I have a similar problem. on microk8s 1.20.2 on ubuntu.
I am pretty sure this is BTRFS filesystem specific.

[WereCatf]

This seems to have been fixed recently. I just installed 1.21.0 in a VM with a Btrfs-filesystem and Microk8s does seem to be working as it's supposed to. Not that I tested it in depth, but at least it now states the node is ready.

Closing this as solved.

[gaetanquentin]

just installed mirok8s on 20.04.1 LTS with btrfs.
Env:

juju 2.9.10 16909 latest/stable canonical✓ classic
lxd 4.17 21260 latest/stable canonical✓ -
microk8s v1.21.3 2346 1.21/stable canonical✓ classic

I don't see any btrfs subvol created by micro8s: does that mean that btrfs is not used?

by the way, how to see containers created by microk8s? docker ps won't show anything.

Regards.

[exi]

I don't see any btrfs subvol created by micro8s: does that mean that btrfs is not used?

The default is to not use the btrfs snapshotter so this is completely expected.

by the way, how to see containers created by microk8s? docker ps won't show anything.

microk8s does not use docker as the container runtime but instead uses containerd.

To see all running pods in k8s you can try doing:

microk8s kubectl get pods --all-namespaces

[gaetanquentin]

I don't see any btrfs subvol created by micro8s: does that mean that btrfs is not used?

The default is to not use the btrfs snapshotter so this is completely expected.

thank you for your answers.

how to know where are stored the containers ?

[exi]

In MicroK8s container data are kept under /var/snap/microk8s/common/var/lib/containerd and /var/snap/microk8s/common/run/

[gaetanquentin]

In MicroK8s container data are kept under /var/snap/microk8s/common/var/lib/containerd and /var/snap/microk8s/common/run/

ok thank you. last question: how do you activate brtfs snapshotter?

[exi]

In MicroK8s container data are kept under /var/snap/microk8s/common/var/lib/containerd and /var/snap/microk8s/common/run/

ok thank you. last question: how do you activate brtfs snapshotter?

Short answer: better not.

See containerd/containerd#4217 (comment)

The btrfs snapshotter is currently considered broken.