Reconcile when plugin pods are killed and recreated

[yujunz]

Existing volumes should reconcile even when plugin pods are killed and recreated.

[yujunz]

When node driver pod is deleted, the mounted volume will fail and unable to reconcile.

Application log

juicefs-app-dynamic app /bin/sh: /data/out.txt: Transport endpoint is not connected
...
juicefs-app-reader-dynamic app tail: read error: Transport endpoint is not connected
...

Driver log

juicefs-csi-node-mfscc node-driver-registrar E1027 07:30:05.637381       1 connection.go:129] Lost connection to unix:///csi/csi.sock.
- juicefs-csi-node-mfscc
- juicefs-csi-node-mfscc
+ juicefs-csi-node-krvh5 › node-driver-registrar
+ juicefs-csi-node-krvh5 › juicefs-plugin
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.200898       1 main.go:110] Version: v1.1.0-0-g80a94421
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.200964       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.200978       1 connection.go:151] Connecting to unix:///csi/csi.sock
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.202082       1 main.go:127] Calling CSI driver to discover driver name
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.202103       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.202110       1 connection.go:181] GRPC request: {}
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.222933       1 connection.go:183] GRPC response: {"name":"csi.juicefs.com"}
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.225001       1 connection.go:184] GRPC error: <nil>
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.228642       1 main.go:137] CSI driver name: "csi.juicefs.com"
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.874173       1 driver.go:29] Driver: csi.juicefs.com Version:
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.229121       1 node_register.go:54] Starting Registration Server at: /registration/csi.juicefs.com-reg.sock
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.229320       1 node_register.go:61] Registration Server started at: /registration/csi.juicefs.com-reg.sock
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:15.750589       1 main.go:77] Received GetInfo call: &InfoRequest{}
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.876078       1 mount_linux.go:174] Cannot run systemd-run, assuming non-systemd OS
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.876104       1 mount_linux.go:175] systemd-run failed with: exit status 1
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.876115       1 mount_linux.go:176] systemd-run output: Failed to create bus connection: No such file or directory
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.877927       1 mount_linux.go:174] Cannot run systemd-run, assuming non-systemd OS
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.877948       1 mount_linux.go:175] systemd-run failed with: exit status 1
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.878024       1 mount_linux.go:176] systemd-run output: Failed to create bus connection: No such file or directory
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:14.878502       1 driver.go:66] Listening for connection on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:16.752697       1 main.go:77] Received GetInfo call: &InfoRequest{}
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:16.760304       1 node.go:177] NodeGetInfo: called with args
juicefs-csi-node-krvh5 node-driver-registrar I1027 07:30:16.783322       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
juicefs-csi-node-krvh5 juicefs-plugin I1027 07:30:33.662996       1 node.go:160] NodeGetCapabilities: called with args

[yujunz]

Related discussion: https://groups.google.com/d/msg/kubernetes-sig-storage-wg-csi/VKwH_jj97x0/LipZHmAgBwAJ

4. In-tree volume provisioners can implement
   Attacher.VolumesAreAttached and BulkVolumeVerifier to make sure that
   the volumes are periodically monitored and reconciled. I don't see
   such a support in CSI. Does CSI support monitoring/reconciling volumes?

Not yet. We assume that the driver itself keeps the volumes attached.
There is no way how a driver would tell Kubernetes that something is not
attached (mounted, ...) any longer.

[yujunz]

Work around in https://github.com/juicedata/juicefs-csi-driver#juicefs-csi-volumes-can-not-reconcile-14

[chengyu-l]

在NodePublishVolume方法中，先执行umount，再执行mount，可以解决这个问题。建议将JfsMount方法放置到NodeStageVolume实现。NodePublishVolume方法当中只调用Mount

[davies]

done