unmount: invalid argument

[smira]

Bug Report

While preparing for an upgrade, we hit this err:

[1919483.106390] [talos] task unmountPodMounts (2/2): failed: error unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/03a545ee: invalid argument

Description

Logs

[1919482.887461] [talos] task unmountOverlayFilesystems (1/2): starting
[1919482.896608] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/8b65a629-36e2-489f-ad29-23a53b1a15ba/volumes/kubernetes.io~empty-dir/keys-tmp-dir
[1919482.912182] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/3d84aee2-9647-42d4-a73b-8d83701b7b79/volumes/kubernetes.io~secret/rook-csi-cephfs-plugin-sa-token-p56wp
[1919482.929888] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/3d84aee2-9647-42d4-a73b-8d83701b7b79/volumes/kubernetes.io~empty-dir/keys-tmp-dir
[1919482.944994] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/4eff467b-2fad-43f6-94fe-a312dcbb83a7/volumes/kubernetes.io~secret/calico-node-token-2ktwj
[1919482.961416] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/8b65a629-36e2-489f-ad29-23a53b1a15ba/volumes/kubernetes.io~secret/rook-csi-rbd-plugin-sa-token-9g76r
[1919482.978159] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/volumes/kubernetes.io~projected/kube-api-access-kggmn
[1919482.994597] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/7e0961bd-619b-4e5f-b5b7-c981f7baa9ab/volumes/kubernetes.io~projected/kube-api-access-qhb2v
[1919483.010506] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/a280077c-4f3d-485b-96fc-839abea47bd9/volumes/kubernetes.io~projected/kube-api-access-t2sb4
[1919483.026803] [talos] task unmountOverlayFilesystems (1/2): done, 144.852361ms
[1919483.034292] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/416f7599
[1919483.048764] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/416f7599
[1919483.063339] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/03a545ee
[1919483.077756] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/416f7599
[1919483.092254] [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/03a545ee
[1919483.106390] [talos] task unmountPodMounts (2/2): failed: error unmounting /var/lib/kubelet/pods/090c2b2f-c594-4ac3-88ad-10b58dc21bce/containers/debug-container/03a545ee: invalid argument
[1919483.123733] [talos] phase unmount (5/11): failed
[1919483.128761] [talos] upgrade sequence: failed

Environment

• Talos version: [talosctl version --nodes <problematic nodes>]
• Kubernetes version: [kubectl version --short]
• Platform: