Workflow for moving a writer workload into a node with an existing reader workload

[sumukhs]

In the following scenario, could you please clarify the right behavior w.r.t ControllerPublishVolume call semantics:

Setup:
0. Volume v1 is created with multi node reader/writer capability.

1. ControllerPublishVolume for node N1, volume v1 is invoked with capability as Read only = true. CO invokes NodeStageVolume on N1, followed by NodePublishVolume.
2. ControllerPublishVolume for node N2, volume v1 is invoked with capability as a Writer. CO invokes NodeStageVolume on N2, followed by NodePublishVolume.

Scenario:
The CO intends to move workload on N2 to N1.

Behavior 1:

1. CO invokes ControllerPublishVolume for node N1, volume v1 with capability as Writer.
2. CO then invokes NodePublishVolume on node N1, volume v1 with capability as Writer with the same target staging path as earlier and a new target path.

Behavior 2:

1. CO directly invokes NodePublishVolume on node N1, volume v1 with capability as Writer with the same target staging path as earlier and a new target path.

What is the expected contract per CSI?
I would expect 2, but wanted to double check with the community.

[sumukhs]

I read this in the kubernetes docs
Important! A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.

In my scenario above, a volume's node capability is being modified from being read-only to read-write - based on the text above it is unsupported from Kubernetes?

[jdef]

You probably need to ask this question in some k8s SIG Storage forum, as it seems pretty CO-specific (because you're asking about the k8s implementation of CSI).

…

On Tue, Nov 9, 2021 at 10:47 AM Sumukh Shivaprakash < ***@***.***> wrote: I read this in the kubernetes docs <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes> Important! A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time. In my scenario above, a volume's node capability is being modified from being read-only to read-write - based on the text above it is unsupported from Kubernetes? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#497 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR5KLFFVMASFLXDEENO3B3ULE7CXANCNFSM5HUNUNAQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- James DeFelice 585.241.9488 (voice) 650.649.6071 (fax)

[bswartz]

@sumukhs The limitation is worse than that for K8s. In K8s you can only have one access mode per PVC, if you use CSI. The existing sidecars prohibit mixing access modes.

You can use the RWX access mode, and make individual pods read only, and ControllerPublish will be called with readOnly=false and NodePublish will be called with readOnly=true. This gives you the desired behavior, I think.