Massive error when losing connection to meta

[yujunz]

When meta is down, the driver has problem cleaning up existing mount points

I0314 03:32:54.829147       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/159f0ab4-1337-4249-8577-9b446909246d/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.829214       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/159f0ab4-1337-4249-8577-9b446909246d/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point
I0314 03:32:54.829312       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/babd88dc-ed30-4fe5-a814-9c71b1e53de2/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.829343       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/babd88dc-ed30-4fe5-a814-9c71b1e53de2/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point
I0314 03:32:54.849151       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/7635a844-d00d-4208-a03e-eb95a7493e92/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.849200       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/7635a844-d00d-4208-a03e-eb95a7493e92/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point
I0314 03:32:54.849258       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/311d1dbf-ddd1-43a4-9e13-f9828803cd7f/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.849290       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/311d1dbf-ddd1-43a4-9e13-f9828803cd7f/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point
I0314 03:32:54.849333       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/0d627b6c-156e-4ed6-8fcf-8a907c2e9f24/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.849359       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/0d627b6c-156e-4ed6-8fcf-8a907c2e9f24/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point
I0314 03:32:54.849401       1 node.go:150] NodeUnpublishVolume: called with args volume_id:"pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94" target_path:"/var/lib/kubelet/pods/9ec56f3f-2ac1-4d0d-9b37-738a9215a8ac/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount"
E0314 03:32:54.849429       1 driver.go:53] GRPC error: rpc error: code = Internal desc = Error checking "/var/lib/kubelet/pods/9ec56f3f-2ac1-4d0d-9b37-738a9215a8ac/volumes/kubernetes.io~csi/pvc-719bbf50-9ca9-41cd-b537-9301d1b07d94/mount" is not mount point

[yujunz]

Workaround: exec shell into the pod and run unmount.

[chengyu-l]

采用这个方法可以解决

[davies]

@yujunz how can we reproduce this?

[yujunz]

I don't remember the steps. Try killing meta and see what happens?

[davies]

can't reproduce, close it for now.

[ggaaooppeenngg]

Which version fix this problem?

[davies]

@ggaaooppeenngg Can you reproduce this one?

[ggaaooppeenngg]

We are using 0.7.0. When a node is facing disk pressure and begin to taint itself to evict pods, we find these mount points problem. At this point, the csi-driver is running but the node is in a tainted state. I am not sure if disk pressure recover, these problems will disappear, I will give feedback then.