Delay PublishVolumeRequest until OFFLINE-resizable Volume is resized

[zuzzas]

If a plugin has a VolumeExpansion.OFFLINE, then a resize process should happen after ControllerUnpublishVolume and before ControllerPublishVolume.

Currently, this is not the case, and if I am to delete Pod, then external-resizer and external-attacher would race each other. If external-resizer wins and calls ControllerPublishVolume before a resize operation is finished, then we have to delete a Pod again. Which is incredibly frustrating.

I've hacked together a hacky "solution". But I think that OFFLINE resize should work properly on the CSI side, not on the plugin side.

I propose not Publishing a Volume to a Node if a PVC is currently resizing.

[zuzzas]

Here is the hack:

1. Lock volume if there is a resize operation pending: https://github.com/flant/yandex-csi-driver/blob/master/driver/controller.go#L559
2. Refuse to ControllerPublishVolume if it is locked: https://github.com/flant/yandex-csi-driver/blob/master/driver/controller.go#L270-L272
3. Unlock volume if it is resized: https://github.com/flant/yandex-csi-driver/blob/master/driver/controller.go#L564

[jsafrane]

I think this should be solved on the driver level.

Kubernetes is a dynamic system and does not offer any mechanism to "lock" objects on controller level and clean up the locks when a controller dies / restarts.

[zuzzas]

Oh, but there are examples of locking throughout the Kubernetes. Leases (and Endpoints previously) that are used for leader election is a good example.

[jsafrane]

Leader election works per controller (i.e. one object per cluster). You want to have a lock per PV. That's completely different scale.

We tried to add annotation with a lock to PVCs to be able to run multiple provisioners in parallel, but adding/removing/checking the annotation on every PVC basically overloaded API server.
See kubernetes-retired/external-storage#892, kubernetes-csi/external-provisioner#68, kubernetes-retired/external-storage#553. It does not scale.

[zuzzas]

Okay, I see, thanks for the references.

Let's approach the issue from another direction. Has there been a successful CSI OFFLINE ExpandVolume which does not suffer from issues that I've outlined on the top? Because I am not sure how that can be fixed on the driver side, if the driver is not aware (when Publishing Volume) that there is a resize pending.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.