Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add e2e test for mounting different paths on same volume on same node #173

Merged
merged 1 commit into from
Jun 24, 2020

Conversation

wongma7
Copy link
Contributor

@wongma7 wongma7 commented May 28, 2020

Is this a bug fix or adding new feature? fix #105

What is this PR about? / Why do we need it? #100 was fixed but we need an e2e test for it.

What testing is done?
Testing on my EKS cluster:

------------------------------
[efs-csi] EFS CSI [Driver: efs.csi.aws.com] 
  should mount different paths on same volume on same node
  /Users/mattwon/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:182
[BeforeEach] [efs-csi] EFS CSI
  /Users/mattwon/go/pkg/mod/k8s.io/kubernetes@v1.17.6/test/e2e/framework/framework.go:151
STEP: Creating a kubernetes client
Jun  1 16:00:48.241: INFO: >>> kubeConfig: /Users/mattwon/.kube/config
STEP: Building a namespace api object, basename efs
Jun  1 16:00:48.320: INFO: Found PodSecurityPolicies; assuming PodSecurityPolicy is enabled.
Jun  1 16:00:48.362: INFO: Found ClusterRoles; assuming RBAC is enabled.
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in efs-6908
STEP: Waiting for a default service account to be provisioned in namespace
[It] should mount different paths on same volume on same node
  /Users/mattwon/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:182
STEP: Creating efs pvc & pv with no subpath
STEP: Creating pod to make subpaths /a and /b
Jun  1 16:00:48.562: INFO: Waiting up to 5m0s for pod "pvc-tester-frqh5" in namespace "efs-6908" to be "success or failure"
Jun  1 16:00:48.577: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 15.387249ms
Jun  1 16:00:50.596: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.03421632s
Jun  1 16:00:52.612: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.050449793s
Jun  1 16:00:54.628: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.066294845s
Jun  1 16:00:56.645: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.082881893s
Jun  1 16:00:58.662: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.1000315s
Jun  1 16:01:00.683: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 12.121066843s
Jun  1 16:01:02.699: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 14.13670026s
Jun  1 16:01:04.715: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 16.153338265s
Jun  1 16:01:06.734: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 18.171695624s
Jun  1 16:01:08.753: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 20.191351553s
Jun  1 16:01:10.770: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 22.208355608s
Jun  1 16:01:12.787: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 24.225318444s
Jun  1 16:01:14.805: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 26.243291824s
Jun  1 16:01:16.824: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 28.262323831s
Jun  1 16:01:18.841: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 30.278845233s
Jun  1 16:01:20.858: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 32.295765653s
Jun  1 16:01:22.873: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 34.311190886s
Jun  1 16:01:24.891: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 36.328770405s
Jun  1 16:01:26.906: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 38.343785078s
Jun  1 16:01:28.922: INFO: Pod "pvc-tester-frqh5": Phase="Pending", Reason="", readiness=false. Elapsed: 40.360238146s
Jun  1 16:01:30.943: INFO: Pod "pvc-tester-frqh5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 42.38106425s
STEP: Saw pod success
Jun  1 16:01:30.943: INFO: Pod "pvc-tester-frqh5" satisfied condition "success or failure"
STEP: Creating efs pvc & pv with subpath /a
STEP: Creating efs pvc & pv with subpath /b
STEP: Creating pod on node "ip-192-168-83-235.us-west-2.compute.internal" to mount subpaths /a and /b
[AfterEach] [efs-csi] EFS CSI
  /Users/mattwon/go/pkg/mod/k8s.io/kubernetes@v1.17.6/test/e2e/framework/framework.go:152
Jun  1 16:02:13.136: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "efs-6908" for this suite.

• [SLOW TEST:84.927 seconds]
[efs-csi] EFS CSI
/Users/mattwon/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:173
  [Driver: efs.csi.aws.com]
  /Users/mattwon/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:181
    should mount different paths on same volume on same node
    /Users/mattwon/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:182
------------------------------

Testing on CI's kops cluster (https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_aws-efs-csi-driver/173/pull-aws-efs-csi-driver-e2e/1268250579340103680):

 [efs-csi] EFS CSI [Driver: efs.csi.aws.com] 
  should mount different paths on same volume on same node
  /home/prow/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:182
[BeforeEach] [efs-csi] EFS CSI
  /home/prow/go/pkg/mod/k8s.io/kubernetes@v1.17.6/test/e2e/framework/framework.go:151
STEP: Creating a kubernetes client
Jun  3 19:07:41.652: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename efs
STEP: Waiting for a default service account to be provisioned in namespace
[It] should mount different paths on same volume on same node
  /home/prow/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/test/e2e/e2e.go:182
STEP: Creating efs pvc & pv with no subpath
STEP: Creating pod to make subpaths /a and /b
Jun  3 19:07:42.041: INFO: Waiting up to 5m0s for pod "pvc-tester-lndpk" in namespace "efs-1888" to be "success or failure"
Jun  3 19:07:42.096: INFO: Pod "pvc-tester-lndpk": Phase="Pending", Reason="", readiness=false. Elapsed: 54.842217ms
Jun  3 19:07:44.154: INFO: Pod "pvc-tester-lndpk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.112854684s
Jun  3 19:07:46.209: INFO: Pod "pvc-tester-lndpk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.167987328s
Jun  3 19:07:48.264: INFO: Pod "pvc-tester-lndpk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.222731898s
Jun  3 19:07:50.319: INFO: Pod "pvc-tester-lndpk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.277866927s
Jun  3 19:07:52.375: INFO: Pod "pvc-tester-lndpk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 10.333033294s
STEP: Saw pod success
Jun  3 19:07:52.375: INFO: Pod "pvc-tester-lndpk" satisfied condition "success or failure"
STEP: Creating efs pvc & pv with subpath /a
STEP: Creating efs pvc & pv with subpath /b
STEP: Creating pod on node "ip-172-20-114-53.us-west-2.compute.internal" to mount subpaths /a and /b
[AfterEach] [efs-csi] EFS CSI
  /home/prow/go/pkg/mod/k8s.io/kubernetes@v1.17.6/test/e2e/framework/framework.go:152
Jun  3 19:08:12.988: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "efs-1888" for this suite. 

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 28, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 28, 2020
@wongma7 wongma7 force-pushed the subpath-test branch 2 times, most recently from 841b93d to 416d786 Compare May 28, 2020 18:48
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 28, 2020
@wongma7 wongma7 force-pushed the subpath-test branch 2 times, most recently from 718905b to 21de65d Compare June 1, 2020 23:05
@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 1, 2020
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 3, 2020
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 3, 2020
@wongma7 wongma7 changed the title WIP: Add e2e test for mounting different paths on same volume on same node Add e2e test for mounting different paths on same volume on same node Jun 3, 2020
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 3, 2020
@wongma7
Copy link
Contributor Author

wongma7 commented Jun 3, 2020

/test pull-aws-efs-csi-driver-e2e

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_aws-efs-csi-driver/173/pull-aws-efs-csi-driver-e2e/1268220967159599105 looks like an instance of #141. Saw similar 'mount.nfs4...blocked for more than 120 seconds.' messages in dmesg with no mounts of fs-286c882d on node ip-172-20-119-122.us-west-2.compute.internal ever succeeding

@wongma7
Copy link
Contributor Author

wongma7 commented Jun 3, 2020

/assign @leakingtapan
PTAL if you get a chance, this is a test to verify the fix you implemented for the path mounting issue. (#105). THX

test/e2e/e2e.go Outdated Show resolved Hide resolved
Copy link
Contributor

@jqmichael jqmichael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Some minor questions.

defer func() { _ = f.ClientSet.CoreV1().PersistentVolumes().Delete(pvRoot.Name, &metav1.DeleteOptions{}) }()

ginkgo.By(fmt.Sprintf("Creating pod to make subpaths /a and /b"))
pod := e2epod.MakePod(f.Namespace.Name, nil, []*v1.PersistentVolumeClaim{pvcRoot}, false, "mkdir /mnt/volume1/a && mkdir /mnt/volume1/b")
Copy link
Contributor

@jqmichael jqmichael Jun 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where "/mnt/volume1" is defined:

		volumeMounts[index] = v1.VolumeMount{Name: volumename, MountPath: "/mnt/" + volumename}

https://github.com/kubernetes/kubernetes/blob/7c161d6a5ff000de2147a33fd62225f398d76ae2/test/e2e/framework/pod/create.go#L190

framework.ExpectNoError(e2epod.WaitForPodSuccessInNamespace(f.ClientSet, pod.Name, f.Namespace.Name), "waiting for pod success")

ginkgo.By(fmt.Sprintf("Creating efs pvc & pv with subpath /a"))
pvcA, pvA, err := createEFSPVCPV(f.ClientSet, f.Namespace.Name, f.Namespace.Name+"-a", "/a")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wish Golang support keywords argument like Python does. It's not easy to tell what those parameters really represent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree lol. a pattern I have seen is to add /* */ after every arg but thats gets tiring and verbose fast. IDE can help here.

defer func() { _ = f.ClientSet.CoreV1().PersistentVolumes().Delete(pvRoot.Name, &metav1.DeleteOptions{}) }()

ginkgo.By(fmt.Sprintf("Creating pod to make subpaths /a and /b"))
pod := e2epod.MakePod(f.Namespace.Name, nil, []*v1.PersistentVolumeClaim{pvcRoot}, false, "mkdir /mnt/volume1/a && mkdir /mnt/volume1/b")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interface of MakePod

func MakePod(ns string, nodeSelector map[string]string, pvclaims []*v1.PersistentVolumeClaim, isPrivileged bool, command string) *v1.Pod {

https://github.com/kubernetes/kubernetes/blob/7c161d6a5ff000de2147a33fd62225f398d76ae2/test/e2e/framework/pod/create.go#L158

pod = e2epod.MakePod(f.Namespace.Name, nil, []*v1.PersistentVolumeClaim{pvcA, pvcB}, false, "")
pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod)
framework.ExpectNoError(err, "creating pod")
framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, pod.Name, f.Namespace.Name), "waiting for pod running")
Copy link
Contributor

@jqmichael jqmichael Jun 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is an additional check on whether the mount is accessible necessary, i.e. kubectl exec ... ls /mnt/volume1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, the pod from the initial step "ginkgo.By(fmt.Sprintf("Creating pod to make subpaths /a and /b"))" would have failed if somehow mnt is not accessible

ginkgo.By(fmt.Sprintf("Creating efs pvc & pv with no subpath"))
pvcRoot, pvRoot, err := createEFSPVCPV(f.ClientSet, f.Namespace.Name, f.Namespace.Name+"-root", "/")
framework.ExpectNoError(err, "creating efs pvc & pv with no subpath")
defer func() { _ = f.ClientSet.CoreV1().PersistentVolumes().Delete(pvRoot.Name, &metav1.DeleteOptions{}) }()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are PVC and Pods getting cleaned up automatically?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, since namespaces gets cleaned up automatically

@k8s-ci-robot
Copy link
Contributor

@jqmichael: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Some minor questions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nckturner
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 24, 2020
@wongma7
Copy link
Contributor Author

wongma7 commented Jun 24, 2020

/test pull-aws-efs-csi-driver-verify

@k8s-ci-robot k8s-ci-robot merged commit d4cb17e into kubernetes-sigs:master Jun 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add e2e test for mounting two folders on same EFS volume
5 participants