Provision vsphere volume as per zone #72731

skarthiksrinivas · 2019-01-09T15:39:29Z

What type of PR is this?
/kind bug

What this PR does / why we need it:
Currently vsphere cloud provider (VCP) insists on provisioning a volume only on a globally shared datastore. Hence, in a zoned environment, even in presence of a shared datastore within a specific zone, the volume provisioning can fail if that datastore is not shared across all the zones hosting kubernetes nodes. This change fixes this issue by considering the zone information provided in allowedTopologies for selection of the datastore. If allowedTopologies is not provided, the current behaviour is retained as-is.
This PR addresses one part of issue #67703. The other part to attach zone labels to the created volumes is here - #72687
Which issue(s) this PR fixes:
Fixes #

Does this PR introduce a user-facing change?:
Yes

This change ensures that volumes get provisioned based on the zone information provided in allowedTopologies.

Storage class spec:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fastpolicy1
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: zeroedthick
    storagePolicyName: vSAN Default Storage Policy
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - zone1

PV creation Logs:
I0109 11:17:52.321372       1 vsphere.go:1147] Starting to create a vSphere volume with volumeOptions: &{CapacityKB:1048576 Tags:map[kubernetes.io/created-for/pvc/namespace:default kubernetes.io/created-for/pvc/name:pvcsc-1-policy kubernetes.io/created-for/pv/name:pvc-34650c12-1400-11e9-aef4-005056804cc9] Name:kubernetes-dynamic-pvc-34650c12-1400-11e9-aef4-005056804cc9 DiskFormat:zeroedthick Datastore: VSANStorageProfileData: StoragePolicyName:vSAN Default Storage Policy StoragePolicyID: SCSIControllerType: Zone:[zone1]}
...
I0109 11:17:59.430113       1 vsphere.go:1334] The canonical volume path for the newly created vSphere volume is "[vsanDatastore] 98db185c-6683-d8c7-bc55-0200435ec5da/kubernetes-dynamic-pvc-34650c12-1400-11e9-aef4-005056804cc9.vmdk"

Ran regression tests (no zone) and they passed.

k8s-ci-robot · 2019-01-09T15:39:32Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2019-01-09T15:39:38Z

Hi @skarthiksrinivas. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

skarthiksrinivas · 2019-01-09T15:47:31Z

/cc @abrarshivani @SandeepPissay @divyenpatel

frapposelli · 2019-01-09T15:58:42Z

/ok-to-test

frapposelli · 2019-01-09T15:58:59Z

/sig vmware

gnufied · 2019-01-09T16:41:39Z

pkg/volume/vsphere_volume/vsphere_volume_util.go

@@ -104,6 +104,7 @@ func (util *VsphereDiskUtil) CreateVolume(v *vsphereVolumeProvisioner) (volSpec
 		Name:       name,
 	}

+	volumeOptions.Zone = selectedZone


So if I understand this correctly, this feature does not uses NodeAffinity field of PVs and hence does not uses topology aware provisioning?

Yes. Your understanding is correct. The scope of this fix is limited to honouring the allowedTopologies zones during volume provisioning.

So if I don't specify allowedTopology in my default storageClass and my nodes are spread across zones then it will basically result in PVCs which can not be used by pods. Previously - we blocked/errored out on volume provisioning altogether if datastore being used is not shared with all VMs (#72497). How does this interact with that bug?

The issue in #72497 is that volume provisioning will fail if there is no shared datastore available across all kubernetes node VMs. Now, with this change, by specifying allowedTopologies in SC, the volume provisioning can happen as long as there is a shared datastore available for the nodes within the zone.
There is no change in behaviour when allowedTopology is not specified. It will continue to work in the same way today, i.e by looking for shared datastore across all nodes and succeed or fail as the case is.

gnufied · 2019-01-09T16:48:45Z

pkg/cloudprovider/providers/vsphere/nodemanager.go

+	nm.zoneInfoLock.Lock()
+	nm.zoneInfoMap[nodeName] = zone
+	nm.zoneInfoLock.Unlock()
+}


Should use defer pattern?

Makes sense. Done.

gnufied · 2019-01-09T16:49:14Z

pkg/cloudprovider/providers/vsphere/nodemanager.go

+
+	nm.zoneInfoLock.Lock()
+	delete(nm.zoneInfoMap, node.ObjectMeta.Name)
+	nm.zoneInfoLock.Unlock()
 }


lets use defer if possible.

Still not fixed.

My bad. Fixed the code to use defer pattern for all access to zoneInfoLock.

skarthiksrinivas · 2019-01-10T15:54:32Z

Addressed comments.

k8s-ci-robot · 2019-02-13T12:48:33Z

@skarthiksrinivas: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kops-aws	a7f8f66e9dbcc397251ad7c40f963c9735c628c1	link	`/test pull-kubernetes-e2e-kops-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

skarthiksrinivas · 2019-02-13T13:33:00Z

/retest

SandeepPissay · 2019-02-15T19:02:44Z

/approve

k8s-ci-robot · 2019-02-15T19:05:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: frapposelli, SandeepPissay, skarthiksrinivas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/cloudprovider/providers/vsphere/OWNERS~~ [SandeepPissay,frapposelli]
~~pkg/volume/vsphere_volume/OWNERS~~ [SandeepPissay]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vladimirvivien · 2019-02-15T22:04:45Z

@SandeepPissay will this be cherry picked to earlier k8s version ?

frapposelli · 2019-02-16T11:07:48Z

@vladimirvivien I believe this will not be backported.

/lgtm

rhockenbury · 2019-02-16T21:22:47Z

Does anyone know if there are plans to implement theWaitForFirstConsumer option for the volume binding mode for vsphere?

frapposelli · 2019-02-18T07:32:49Z

/lgtm

frapposelli · 2019-02-18T11:04:33Z

🎉

tedyu · 2019-02-18T16:23:23Z

pkg/cloudprovider/providers/vsphere/vsphere.go

+					return "", err
+				}
+
+				if err != nil {


Shouldn't this be ahead of the if block starting at line 1260 ?

Yes. That's correct. Thanks for pointing out. The current sequence makes this check a no-op. Fixed it. Have created a PR for this - #74263

tedyu · 2019-02-18T16:28:07Z

pkg/cloudprovider/providers/vsphere/vsphere_util.go

+
+		for _, host := range hosts {
+			var hostSystemMo mo.HostSystem
+			host.Properties(ctx, host.Reference(), []string{"datastore"}, &hostSystemMo)


What's the effect of this call (I don't see assignment) ?

The last parameter in the this method call is an OUT parameter which will be set and that's how hostSystemMo gets assigned. However, this method returns error which is not processed in this step. I have fixed that now in the PR mentioned above.

skarthiksrinivas · 2019-02-19T17:07:06Z

Does anyone know if there are plans to implement theWaitForFirstConsumer option for the volume binding mode for vsphere?

Yes. We do have that task in the pipeline.

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 9, 2019

k8s-ci-robot added sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 9, 2019

k8s-ci-robot requested review from jsafrane and rootfs January 9, 2019 15:40

k8s-ci-robot requested review from abrarshivani, divyenpatel and SandeepPissay January 9, 2019 15:47

sandeeppsunny mentioned this pull request Jan 9, 2019

Added e2e tests for multi-zone volume provisioning through VCP #72732

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 9, 2019

k8s-ci-robot added the area/provider/vmware Issues or PRs related to vmware provider label Jan 9, 2019

gnufied reviewed Jan 9, 2019

View reviewed changes

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 10, 2019

skarthiksrinivas force-pushed the vsphere_volume_zone branch from e00fc45 to 64c82ec Compare February 13, 2019 11:30

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 13, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 15, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 16, 2019

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 17, 2019

Provision vsphere volume honoring zones

a309d8a

skarthiksrinivas force-pushed the vsphere_volume_zone branch from 64c82ec to a309d8a Compare February 18, 2019 06:11

k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 18, 2019

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Feb 18, 2019

k8s-ci-robot merged commit 701f914 into kubernetes:master Feb 18, 2019

tedyu reviewed Feb 18, 2019

View reviewed changes

embano1 mentioned this pull request Feb 18, 2019

vSphere Cloud Provider does not work when deployed across Zones with zone-local Storage #67703

Closed

4 tasks

skarthiksrinivas deleted the vsphere_volume_zone branch February 19, 2019 04:29

skarthiksrinivas restored the vsphere_volume_zone branch February 19, 2019 04:29

skarthiksrinivas mentioned this pull request Feb 19, 2019

Fix error handling logic in vsphere volume provisioning #74263

Merged

rhockenbury mentioned this pull request Feb 20, 2019

Add labels to vsphere cloud config kubernetes-sigs/kubespray#4275

Merged

subramanian-neelakantan mentioned this pull request Feb 27, 2019

places vsphere volumes as per zones #74654

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provision vsphere volume as per zone #72731

Provision vsphere volume as per zone #72731

skarthiksrinivas commented Jan 9, 2019 •

edited

Loading

k8s-ci-robot commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

skarthiksrinivas commented Jan 9, 2019

frapposelli commented Jan 9, 2019

frapposelli commented Jan 9, 2019

gnufied Jan 9, 2019

skarthiksrinivas Jan 10, 2019

gnufied Jan 17, 2019

skarthiksrinivas Jan 18, 2019

gnufied Jan 9, 2019

skarthiksrinivas Jan 10, 2019

gnufied Jan 9, 2019

skarthiksrinivas Jan 10, 2019

gnufied Jan 11, 2019

skarthiksrinivas Jan 17, 2019

skarthiksrinivas commented Jan 10, 2019

k8s-ci-robot commented Feb 13, 2019 •

edited

Loading

skarthiksrinivas commented Feb 13, 2019

SandeepPissay commented Feb 15, 2019

k8s-ci-robot commented Feb 15, 2019

vladimirvivien commented Feb 15, 2019

frapposelli commented Feb 16, 2019

rhockenbury commented Feb 16, 2019

frapposelli commented Feb 18, 2019

frapposelli commented Feb 18, 2019

tedyu Feb 18, 2019

skarthiksrinivas Feb 19, 2019

tedyu Feb 18, 2019

skarthiksrinivas Feb 19, 2019

skarthiksrinivas commented Feb 19, 2019

Provision vsphere volume as per zone #72731

Provision vsphere volume as per zone #72731

Conversation

skarthiksrinivas commented Jan 9, 2019 • edited Loading

k8s-ci-robot commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

skarthiksrinivas commented Jan 9, 2019

frapposelli commented Jan 9, 2019

frapposelli commented Jan 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skarthiksrinivas commented Jan 10, 2019

k8s-ci-robot commented Feb 13, 2019 • edited Loading

skarthiksrinivas commented Feb 13, 2019

SandeepPissay commented Feb 15, 2019

k8s-ci-robot commented Feb 15, 2019

vladimirvivien commented Feb 15, 2019

frapposelli commented Feb 16, 2019

rhockenbury commented Feb 16, 2019

frapposelli commented Feb 18, 2019

frapposelli commented Feb 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skarthiksrinivas commented Feb 19, 2019

skarthiksrinivas commented Jan 9, 2019 •

edited

Loading

k8s-ci-robot commented Feb 13, 2019 •

edited

Loading