Wait for volume status in e2e test #34

bertinatto · 2018-10-03T13:49:14Z

These are the bits missing in the unfinished e2e test.

I ran this test in a loop for about 20 minutes and it didn't flake.

k8s-ci-robot · 2018-10-03T13:49:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bertinatto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bertinatto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

leakingtapan

One general comment, could you add some simple description on how do you run this e2e test on EC2? We are looking at automate running this. If you are doing it manually, it could be our good first step. If its already automated, it will be great.

leakingtapan · 2018-10-04T21:45:14Z

tests/e2e/e2e_test.go

+		// Read the file and check if content is correct
+		data, err := ioutil.ReadFile(testFile)
+		Expect(err).To(BeNil(), "Failed to read file")
+		Expect(data).To(Equal(testFileContents), "File content is incorrect")
 	}

 	// Unmount Disk


Should move the umount to be right after mount disk succeeds. So that any failures in side file read/write won't cause file is not umounted.

Good catch, thanks.

leakingtapan · 2018-10-04T21:46:29Z

tests/e2e/e2e_test.go

-		Factor:   1.2,
-		Steps:    21,
+		Duration: 1 * time.Second,
+		Factor:   1.8,


This value looks a bit tricky. Why is 1.8 better than 1.2? Could you add some docs around this?

Indeed. These values come from Kubernetes and are mostly related to attachment/detachment from an "exponential backoff" strategy.

In order to verify if the volume was created, Kubernetes fetches the volume state every second for 30 seconds (no exponential backoff). This sounds like too many requests for the purposes of this test, so I decided to use the same approach for both attach/detach and provision/deletion.

In any case, I added a comment (taken from Kubernetes) to take the magic away of these numbers.

leakingtapan · 2018-10-04T22:02:39Z

tests/e2e/e2e_test.go

-			},
-		}
-		waitForVolumes(descParams, 1 /* number of expected volumes */)
+		waitForVolumeState(volume.Id, "available")


Feels to me its better to move the watiForVolume creation and deletion inside driver CreateVolume and DeleteVolume so that the volume is guaranteed to be available/removed after the call completes. This also void the case where ControllerPublishVolume is called during volume is still being created.

We could do similar thing for attach (saw your TODO comment on AttachDisk) and detach.

I agree that we need to have similar functionality in the cloud package (e.g., when creating, deleting, attaching and detaching volumes). However, it seems like the implementation of these functions in the driver should be a lot more sophisticated than this. For instance, the function that waits for the volume attachment status in Kubernetes has some magic [1] that I don't quite understand yet and can be tricky to write (in the right way).

Also, assuming that CreateDisk only returns when the volume is ready, e.g., with available status, I this test should make sure that the volume is indeed created here.

[1] https://github.com/bertinatto/kubernetes/blob/5fdb0c7177010711ca7b463c7e1d4f32a163a344/pkg/cloudprovider/providers/aws/aws.go#L1862

Got ya. We can punt this to #38 when we have time to properly implement this.

coveralls · 2018-10-08T13:09:43Z

Pull Request Test Coverage Report for Build 24

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 47.15%

Totals
Change from base Build 23:	0.0%
Covered Lines:	455
Relevant Lines:	965

💛 - Coveralls

bertinatto

Thanks for looking into this, @leakingtapan. I added a few comments and fixed some issues that you pointed out. Please take a look.

One small complement about the wait-for-volume-status functions: I'll start working on the implementation to be used in the driver, I just need to understand better how Kubernetes is handling this at the moment. After that I'll probably rename waitForVolumeStatus to something like validateVolumeStatus.

bertinatto · 2018-10-08T12:38:39Z

tests/e2e/e2e_test.go

-		Factor:   1.2,
-		Steps:    21,
+		Duration: 1 * time.Second,
+		Factor:   1.8,


Indeed. These values come from Kubernetes and are mostly related to attachment/detachment from an "exponential backoff" strategy.

In order to verify if the volume was created, Kubernetes fetches the volume state every second for 30 seconds (no exponential backoff). This sounds like too many requests for the purposes of this test, so I decided to use the same approach for both attach/detach and provision/deletion.

In any case, I added a comment (taken from Kubernetes) to take the magic away of these numbers.

bertinatto · 2018-10-08T13:01:11Z

tests/e2e/e2e_test.go

-			},
-		}
-		waitForVolumes(descParams, 1 /* number of expected volumes */)
+		waitForVolumeState(volume.Id, "available")


I agree that we need to have similar functionality in the cloud package (e.g., when creating, deleting, attaching and detaching volumes). However, it seems like the implementation of these functions in the driver should be a lot more sophisticated than this. For instance, the function that waits for the volume attachment status in Kubernetes has some magic [1] that I don't quite understand yet and can be tricky to write (in the right way).

Also, assuming that CreateDisk only returns when the volume is ready, e.g., with available status, I this test should make sure that the volume is indeed created here.

[1] https://github.com/bertinatto/kubernetes/blob/5fdb0c7177010711ca7b463c7e1d4f32a163a344/pkg/cloudprovider/providers/aws/aws.go#L1862

bertinatto · 2018-10-08T13:01:40Z

tests/e2e/e2e_test.go

+		// Read the file and check if content is correct
+		data, err := ioutil.ReadFile(testFile)
+		Expect(err).To(BeNil(), "Failed to read file")
+		Expect(data).To(Equal(testFileContents), "File content is incorrect")
 	}

 	// Unmount Disk


Good catch, thanks.

leakingtapan · 2018-10-10T05:13:31Z

/lgtm

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 3, 2018

k8s-ci-robot requested review from d-nishi and leakingtapan October 3, 2018 13:49

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 3, 2018

bertinatto added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Oct 3, 2018

bertinatto assigned leakingtapan Oct 3, 2018

leakingtapan mentioned this pull request Oct 3, 2018

Fix bot #20

Closed

leakingtapan reviewed Oct 4, 2018

View reviewed changes

Wait for volume status in e2e test

f110126

bertinatto force-pushed the e2e branch from 4917df0 to f110126 Compare October 8, 2018 13:05

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 8, 2018

bertinatto commented Oct 8, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 10, 2018

k8s-ci-robot merged commit 17a60a2 into kubernetes-sigs:master Oct 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait for volume status in e2e test #34

Wait for volume status in e2e test #34

bertinatto commented Oct 3, 2018

k8s-ci-robot commented Oct 3, 2018

leakingtapan left a comment

leakingtapan Oct 4, 2018

bertinatto Oct 8, 2018

leakingtapan Oct 4, 2018

bertinatto Oct 8, 2018

leakingtapan Oct 4, 2018

bertinatto Oct 8, 2018 •

edited

Loading

leakingtapan Oct 10, 2018

coveralls commented Oct 8, 2018 •

edited

Loading

bertinatto left a comment

bertinatto Oct 8, 2018

bertinatto Oct 8, 2018 •

edited

Loading

bertinatto Oct 8, 2018

leakingtapan commented Oct 10, 2018

Wait for volume status in e2e test #34

Wait for volume status in e2e test #34

Conversation

bertinatto commented Oct 3, 2018

k8s-ci-robot commented Oct 3, 2018

leakingtapan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bertinatto Oct 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Oct 8, 2018 • edited Loading

Pull Request Test Coverage Report for Build 24

💛 - Coveralls

bertinatto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bertinatto Oct 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leakingtapan commented Oct 10, 2018

bertinatto Oct 8, 2018 •

edited

Loading

coveralls commented Oct 8, 2018 •

edited

Loading

bertinatto Oct 8, 2018 •

edited

Loading