[root@master-1-cKPzXSI7IkXr5 e2e]# ./run-e2e-test.sh /root/kkk/csm-operator/tests/e2e W0515 21:44:20.795221 3778589 test_context.go:509] Unable to find in-cluster config, using default host : https://127.0.0.1:6443 May 15 21:44:20.795: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used. Running Suite: CSM Operator End-to-End Tests - /root/kkk/csm-operator/tests/e2e =============================================================================== Random Seed: 1715823811 Will run 1 of 1 specs ------------------------------ [BeforeSuite] /root/kkk/csm-operator/tests/e2e/e2e_test.go:82 STEP: Getting test environment variables @ 05/15/24 21:44:20.795 STEP: [authorization authorizationproxyserver] @ 05/15/24 21:44:20.795 STEP: Reading values file @ 05/15/24 21:44:20.795 STEP: Getting a k8s client @ 05/15/24 21:44:20.799 [BeforeSuite] PASSED [0.006 seconds] ------------------------------ [run-e2e-test] E2E Testing Running all test Given Test Scenarios /root/kkk/csm-operator/tests/e2e/e2e_test.go:123 STEP: Starting: Install PowerFlex Driver (With Authorization), Upgrade driver only @ 05/15/24 21:44:20.802 STEP: Executing Given an environment with k8s or openshift, and CSM operator installed @ 05/15/24 21:44:20.802 STEP: Executing Create [authorization-proxy-server] prerequisites from CR [1] @ 05/15/24 21:44:20.814 === Creating Authorization Proxy Server Prerequisites === STEP: Executing Apply custom resource [1] @ 05/15/24 21:44:44.026 May 15 21:44:44.027: INFO: Running '/usr/bin/kubectl --namespace=authorization apply --validate=true -f -' May 15 21:44:44.325: INFO: stderr: "" May 15 21:44:44.325: INFO: stdout: "containerstoragemodule.storage.dell.com/authorization created\nconfigmap/csm-config-params created\n" STEP: Executing Validate [authorization-proxy-server] module from CR [1] is installed @ 05/15/24 21:44:44.325 err: failed to check for AuthorizationProxyServer installation in default-source-cluster: no pod was found in authorization STEP: Executing Configure authorization-proxy-server for [powerflex] for CR [1] @ 05/15/24 21:45:14.354 === Configuring Authorization Proxy Server === === Generating Admin Token === === Writing Admin Token to Tmp File === Address: 10.247.103.251 === Creating Storage === === Storage === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml storage create --type powerflex --endpoint https://10.225.109.54 --system-id be84bfe40e796c0f --user admin --password Pass4you123! --array-insecure --insecure --addr 10.247.103.251:32402 === Creating Tenant === === Tenant === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml tenant create -n PancakeGroup --insecure --addr 10.247.103.251:32402 === Creating Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml tenant create -n PancakeGroup --insecure --addr 10.247.103.251:32402 === Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml role create --role=CSIGold=powerflex=be84bfe40e796c0f=SP1=300000000 --insecure --addr 10.247.103.251:32402 === Binding Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml rolebinding create --tenant PancakeGroup --role CSIGold --insecure --addr 10.247.103.251:32402 === Generating token === === Token === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml generate token --tenant PancakeGroup --insecure --addr 10.247.103.251:32402 --access-token-expiration 10m0s === Applying token === === Token Applied === STEP: Executing Create storageclass with name [op-e2e-vxflexos] and template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 05/15/24 21:45:24.557 STEP: Executing Set up secret with template [testfiles/powerflex-templates/csm-authorization-config.json] name [karavi-authorization-config] in namespace [test-vxflexos] for [pflexAuthSidecar] @ 05/15/24 21:45:25.008 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflex] @ 05/15/24 21:45:25.31 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflex] @ 05/15/24 21:45:25.67 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflexAuth] @ 05/15/24 21:45:25.765 STEP: Executing Apply custom resource [2] @ 05/15/24 21:45:26.065 May 15 21:45:26.065: INFO: Running '/usr/bin/kubectl --namespace=test-vxflexos apply --validate=true -f -' May 15 21:45:26.296: INFO: stderr: "" May 15 21:45:26.296: INFO: stdout: "containerstoragemodule.storage.dell.com/test-vxflexos configured\n" STEP: Executing Validate custom resource [2] @ 05/15/24 21:45:26.296 STEP: Executing Validate [powerflex] driver from CR [2] is installed @ 05/15/24 21:45:26.3 STEP: Executing Validate [authorization] module from CR [2] is installed @ 05/15/24 21:45:26.305 map[com.dell.karavi-authorization-proxy:true deployment.kubernetes.io/revision:1] map[com.dell.karavi-authorization-proxy:true deprecated.daemonset.template.generation:1] STEP: Executing Run custom test @ 05/15/24 21:45:26.335 May 15 21:45:26.335: INFO: Running ./cert-csi [test vio --sc op-e2e-vxflexos --chainNumber 2 --chainLength 2] [2024-05-15 21:45:26] INFO Starting cert-csi; ver. 0.8.1 [2024-05-15 21:45:26] INFO Using EVENT observer type [2024-05-15 21:45:26] INFO Using config from /root/.kube/config [2024-05-15 21:45:26] INFO Successfully loaded config. Host: https://10.247.103.251:6443 [2024-05-15 21:45:26] INFO Created new KubeClient [2024-05-15 21:45:26] INFO Running 1 iteration(s) [2024-05-15 21:45:26] INFO *** ITERATION NUMBER 1 *** [2024-05-15 21:45:26] INFO Starting VolumeIoSuite with op-e2e-vxflexos storage class [2024-05-15 21:45:26] INFO Successfully created namespace volumeio-test-1b227a87 [2024-05-15 21:45:26] INFO Using default number of volumes [2024-05-15 21:45:26] INFO Creating IO pod [2024-05-15 21:45:26] INFO Waiting for pod iowriter-test-74h6z to be READY [2024-05-15 21:45:26] INFO Waiting for pod iowriter-test-zrfgm to be READY [2024-05-15 21:47:50] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:47:50] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:47:52] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:47:52] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:47:58] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:47:58] INFO VolumeAttachment deleted [2024-05-15 21:47:58] INFO Waiting for pod iowriter-test-fvk8h to be READY [2024-05-15 21:47:58] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:47:58] INFO VolumeAttachment deleted [2024-05-15 21:47:58] INFO Waiting for pod iowriter-test-w8x79 to be READY [2024-05-15 21:48:04] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512] [2024-05-15 21:48:05] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512] [2024-05-15 21:48:12] INFO Hashes match [2024-05-15 21:48:12] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:48:12] INFO Hashes match [2024-05-15 21:48:12] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:48:14] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:48:15] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:48:18] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:48:18] INFO VolumeAttachment deleted [2024-05-15 21:48:20] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:48:20] INFO VolumeAttachment deleted [2024-05-15 21:48:20] INFO Deleting all resources in namespace volumeio-test-1b227a87 [2024-05-15 21:48:32] INFO Namespace volumeio-test-1b227a87 was deleted in 12.017141761s [2024-05-15 21:48:36] INFO SUCCESS: VolumeIoSuite in 3m10.361609477s [2024-05-15 21:48:36] INFO Started generating reports... Collecting metrics 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/sCollecting metrics Generating plots [2024-05-15 21:48:36] INFO Started generating reports... 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [-----------------------------------------------------------------------------------------------] 100.00% 58 p/s[2024-05-15 21:48:37] WARN No ResourceUsageMetrics provided [2024-05-15 21:48:37] ERROR no ResourceUsageMetrics provided report-test-run-f36b8acf: Name: test-run-f36b8acf Host: https://10.247.103.251:6443 StorageClass: op-e2e-vxflexos Minimum and Maximum EntityOverTime charts: /root/.cert-csi/reports/test-run-f36b8acf/PodsCreatingOverTime.png /root/.cert-csi/reports/test-run-f36b8acf/PodsReadyOverTime.png /root/.cert-csi/reports/test-run-f36b8acf/PodsTerminatingOverTime.png /root/.cert-csi/reports/test-run-f36b8acf/PvcsCreatingOverTime.png /root/.cert-csi/reports/test-run-f36b8acf/PvcsBoundOverTime.png Tests: -------------------------------------------------------------- 1. TestCase: VolumeIoSuite Started: 2024-05-15 21:45:26.418473784 -0400 -0400 Ended: 2024-05-15 21:48:36.781620548 -0400 -0400 Result: SUCCESS Stage metrics: PVCAttachment: Avg: 390.422699ms Min: 382.04204ms Max: 398.803359ms Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCAttachment.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCAttachment_boxplot.png PVCBind: Avg: 2m11.670049535s Min: 2m11.665541482s Max: 2m11.674557588s Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCBind.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCBind_boxplot.png PVCCreation: Avg: 2m12.46565907s Min: 2m12.42054386s Max: 2m12.510774281s Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCCreation.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCCreation_boxplot.png PVCDeletion: Avg: 12.867069ms Min: 11.523407ms Max: 14.210732ms Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCDeletion_boxplot.png PVCUnattachment: Avg: 4.436482659s Min: 4.411086651s Max: 4.461878668s Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCUnattachment.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PVCUnattachment_boxplot.png PodCreation: Avg: 1m13.819984581s Min: 4.821209529s Max: 2m23.263439328s Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PodCreation.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PodCreation_boxplot.png PodDeletion: Avg: 4.013199807s Min: 2.377979788s Max: 5.927230294s Histogram: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PodDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/PodDeletion_boxplot.png EntityNumberOverTime: /root/.cert-csi/reports/test-run-f36b8acf/VolumeIoSuite4/EntityNumberOverTime.png [2024-05-15 21:48:37] INFO Avg time of a run: 173.87s [2024-05-15 21:48:37] INFO Avg time of a del: 12.02s [2024-05-15 21:48:37] INFO Avg time of all: 190.36s [2024-05-15 21:48:37] INFO During this run 100.0% of suites succeeded STEP: Executing Upgrade from custom resource [2] to [3] @ 05/15/24 21:48:37.293 STEP: Executing Validate custom resource [3] @ 05/15/24 21:48:37.318 STEP: Executing Validate [powerflex] driver from CR [3] is installed @ 05/15/24 21:48:37.321 STEP: Executing Validate [authorization] module from CR [3] is installed @ 05/15/24 21:48:37.326 map[com.dell.karavi-authorization-proxy:true deployment.kubernetes.io/revision:1] map[com.dell.karavi-authorization-proxy:true deprecated.daemonset.template.generation:1] STEP: Executing Run custom test @ 05/15/24 21:48:37.35 May 15 21:48:37.350: INFO: Running ./cert-csi [test vio --sc op-e2e-vxflexos --chainNumber 2 --chainLength 2] [2024-05-15 21:48:37] INFO Starting cert-csi; ver. 0.8.1 [2024-05-15 21:48:37] INFO Using EVENT observer type [2024-05-15 21:48:37] INFO Using config from /root/.kube/config [2024-05-15 21:48:37] INFO Successfully loaded config. Host: https://10.247.103.251:6443 [2024-05-15 21:48:37] INFO Created new KubeClient [2024-05-15 21:48:37] INFO Running 1 iteration(s) [2024-05-15 21:48:37] INFO *** ITERATION NUMBER 1 *** [2024-05-15 21:48:37] INFO Starting VolumeIoSuite with op-e2e-vxflexos storage class [2024-05-15 21:48:37] INFO Successfully created namespace volumeio-test-7b7b2f4b [2024-05-15 21:48:37] INFO Using default number of volumes [2024-05-15 21:48:37] INFO Creating IO pod [2024-05-15 21:48:37] INFO Waiting for pod iowriter-test-gzzz7 to be READY [2024-05-15 21:48:37] INFO Waiting for pod iowriter-test-tlhrh to be READY [2024-05-15 21:49:15] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:49:15] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:49:17] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:49:17] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:49:23] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:49:23] INFO VolumeAttachment deleted [2024-05-15 21:49:23] INFO Waiting for pod iowriter-test-hfmss to be READY [2024-05-15 21:49:24] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:49:24] INFO VolumeAttachment deleted [2024-05-15 21:49:24] INFO Waiting for pod iowriter-test-ldr84 to be READY [2024-05-15 21:49:31] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512] [2024-05-15 21:49:32] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512] [2024-05-15 21:49:37] INFO Hashes match [2024-05-15 21:49:37] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:49:38] INFO Hashes match [2024-05-15 21:49:38] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:49:39] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:49:41] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:49:44] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:49:44] INFO VolumeAttachment deleted [2024-05-15 21:49:46] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:49:46] INFO VolumeAttachment deleted [2024-05-15 21:49:46] INFO Deleting all resources in namespace volumeio-test-7b7b2f4b [2024-05-15 21:49:58] INFO Namespace volumeio-test-7b7b2f4b was deleted in 12.013177751s Collecting metrics [2024-05-15 21:50:02] INFO SUCCESS: VolumeIoSuite in 1m25.208898459s [2024-05-15 21:50:02] INFO Started generating reports... 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/sCollecting metrics [2024-05-15 21:50:02] INFO Started generating reports... Generating plots 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [-----------------------------------------------------------------------------------------------] 100.00% 47 p/s[2024-05-15 21:50:03] WARN No ResourceUsageMetrics provided [2024-05-15 21:50:03] ERROR no ResourceUsageMetrics provided report-test-run-7efb24a0: Name: test-run-7efb24a0 Host: https://10.247.103.251:6443 StorageClass: op-e2e-vxflexos Minimum and Maximum EntityOverTime charts: /root/.cert-csi/reports/test-run-7efb24a0/PodsCreatingOverTime.png /root/.cert-csi/reports/test-run-7efb24a0/PodsReadyOverTime.png /root/.cert-csi/reports/test-run-7efb24a0/PodsTerminatingOverTime.png /root/.cert-csi/reports/test-run-7efb24a0/PvcsCreatingOverTime.png /root/.cert-csi/reports/test-run-7efb24a0/PvcsBoundOverTime.png Tests: -------------------------------------------------------------- 1. TestCase: VolumeIoSuite Started: 2024-05-15 21:48:37.468055998 -0400 -0400 Ended: 2024-05-15 21:50:02.680568404 -0400 -0400 Result: SUCCESS Stage metrics: PVCAttachment: Avg: 723.825094ms Min: 608.910647ms Max: 838.739541ms Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCAttachment.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCAttachment_boxplot.png PVCBind: Avg: 26.614033072s Min: 26.604538064s Max: 26.623528081s Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCBind.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCBind_boxplot.png PVCCreation: Avg: 27.852181261s Min: 27.744608838s Max: 27.959753685s Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCCreation.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCCreation_boxplot.png PVCDeletion: Avg: 14.370158ms Min: 10.432378ms Max: 18.307938ms Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCDeletion_boxplot.png PVCUnattachment: Avg: 555.87615ms Min: 454.097913ms Max: 657.654387ms Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCUnattachment.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PVCUnattachment_boxplot.png PodCreation: Avg: 22.255318389s Min: 7.432684172s Max: 36.978978582s Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PodCreation.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PodCreation_boxplot.png PodDeletion: Avg: 4.063226068s Min: 2.365573962s Max: 5.616234195s Histogram: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PodDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/PodDeletion_boxplot.png EntityNumberOverTime: /root/.cert-csi/reports/test-run-7efb24a0/VolumeIoSuite5/EntityNumberOverTime.png [2024-05-15 21:50:03] INFO Avg time of a run: 68.67s [2024-05-15 21:50:03] INFO Avg time of a del: 12.01s [2024-05-15 21:50:03] INFO Avg time of all: 85.20s [2024-05-15 21:50:03] INFO During this run 100.0% of suites succeeded STEP: Executing Enable forceRemoveDriver on CR [3] @ 05/15/24 21:50:03.204 STEP: Executing Delete custom resource [3] @ 05/15/24 21:50:03.221 STEP: Executing Delete custom resource [1] @ 05/15/24 21:50:03.236 STEP: Executing Restore template [testfiles/powerflex-templates/csm-authorization-config.json] for [pflexAuthSidecar] @ 05/15/24 21:50:03.246 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflexAuth] @ 05/15/24 21:50:03.359 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 05/15/24 21:50:03.445 STEP: Ending: Install PowerFlex Driver (With Authorization), Upgrade driver only @ 05/15/24 21:50:03.579 STEP: Starting: Install PowerFlex Driver (With Authorization), Upgrade driver and authorization sidecar @ 05/15/24 21:50:08.583 STEP: Executing Given an environment with k8s or openshift, and CSM operator installed @ 05/15/24 21:50:08.583 STEP: Executing Create [authorization-proxy-server] prerequisites from CR [1] @ 05/15/24 21:50:08.587 === Creating Authorization Proxy Server Prerequisites === STEP: Executing Apply custom resource [1] @ 05/15/24 21:50:21.087 May 15 21:50:21.088: INFO: Running '/usr/bin/kubectl --namespace=authorization apply --validate=true -f -' May 15 21:50:21.389: INFO: stderr: "" May 15 21:50:21.389: INFO: stdout: "containerstoragemodule.storage.dell.com/authorization created\nconfigmap/csm-config-params created\n" STEP: Executing Validate [authorization-proxy-server] module from CR [1] is installed @ 05/15/24 21:50:21.389 err: failed to check for AuthorizationProxyServer installation in default-source-cluster: no pod was found in authorization STEP: Executing Configure authorization-proxy-server for [powerflex] for CR [1] @ 05/15/24 21:50:51.423 === Configuring Authorization Proxy Server === === Generating Admin Token === === Writing Admin Token to Tmp File === Address: 10.247.103.251 === Creating Storage === === Storage === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml storage create --type powerflex --endpoint https://10.225.109.54 --system-id be84bfe40e796c0f --user admin --password Pass4you123! --array-insecure --insecure --addr 10.247.103.251:32491 === Creating Tenant === === Tenant === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml tenant create -n PancakeGroup --insecure --addr 10.247.103.251:32491 === Creating Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml tenant create -n PancakeGroup --insecure --addr 10.247.103.251:32491 === Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml role create --role=CSIGold=powerflex=be84bfe40e796c0f=SP1=300000000 --insecure --addr 10.247.103.251:32491 === Binding Role === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml rolebinding create --tenant PancakeGroup --role CSIGold --insecure --addr 10.247.103.251:32491 === Generating token === === Token === /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml generate token --tenant PancakeGroup --insecure --addr 10.247.103.251:32491 --access-token-expiration 10m0s === Applying token === === Token Applied === STEP: Executing Create storageclass with name [op-e2e-vxflexos] and template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 05/15/24 21:51:01.76 STEP: Executing Set up secret with template [testfiles/powerflex-templates/csm-authorization-config.json] name [karavi-authorization-config] in namespace [test-vxflexos] for [pflexAuthSidecar] @ 05/15/24 21:51:02.243 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflex] @ 05/15/24 21:51:02.535 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflex] @ 05/15/24 21:51:02.874 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflexAuth] @ 05/15/24 21:51:02.964 STEP: Executing Apply custom resource [2] @ 05/15/24 21:51:03.289 May 15 21:51:03.289: INFO: Running '/usr/bin/kubectl --namespace=test-vxflexos apply --validate=true -f -' May 15 21:51:03.513: INFO: stderr: "" May 15 21:51:03.513: INFO: stdout: "containerstoragemodule.storage.dell.com/test-vxflexos created\n" STEP: Executing Validate custom resource [2] @ 05/15/24 21:51:03.513 err: expected custom resource status to be Succeeded. Got: err: expected custom resource status to be Succeeded. Got: Failed err: expected custom resource status to be Succeeded. Got: Failed err: expected custom resource status to be Succeeded. Got: Failed err: expected custom resource status to be Succeeded. Got: Failed err: expected custom resource status to be Succeeded. Got: Failed STEP: Executing Validate [powerflex] driver from CR [2] is installed @ 05/15/24 21:52:03.575 STEP: Executing Validate [authorization] module from CR [2] is installed @ 05/15/24 21:52:03.58 map[com.dell.karavi-authorization-proxy:true deployment.kubernetes.io/revision:1] map[com.dell.karavi-authorization-proxy:true deprecated.daemonset.template.generation:1] STEP: Executing Run custom test @ 05/15/24 21:52:03.602 May 15 21:52:03.602: INFO: Running ./cert-csi [test vio --sc op-e2e-vxflexos --chainNumber 2 --chainLength 2] [2024-05-15 21:52:03] INFO Starting cert-csi; ver. 0.8.1 [2024-05-15 21:52:03] INFO Using EVENT observer type [2024-05-15 21:52:03] INFO Using config from /root/.kube/config [2024-05-15 21:52:03] INFO Successfully loaded config. Host: https://10.247.103.251:6443 [2024-05-15 21:52:03] INFO Created new KubeClient [2024-05-15 21:52:03] INFO Running 1 iteration(s) [2024-05-15 21:52:03] INFO *** ITERATION NUMBER 1 *** [2024-05-15 21:52:03] INFO Starting VolumeIoSuite with op-e2e-vxflexos storage class [2024-05-15 21:52:03] INFO Successfully created namespace volumeio-test-4af2b8d8 [2024-05-15 21:52:03] INFO Using default number of volumes [2024-05-15 21:52:03] INFO Creating IO pod [2024-05-15 21:52:03] INFO Waiting for pod iowriter-test-94pd2 to be READY [2024-05-15 21:52:03] INFO Waiting for pod iowriter-test-k9scx to be READY [2024-05-15 21:53:39] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:53:39] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:53:41] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:53:41] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:53:48] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:53:48] INFO VolumeAttachment deleted [2024-05-15 21:53:48] INFO Waiting for pod iowriter-test-x2h6p to be READY [2024-05-15 21:53:50] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:53:50] INFO VolumeAttachment deleted [2024-05-15 21:53:50] INFO Waiting for pod iowriter-test-r5mws to be READY [2024-05-15 21:53:56] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512] [2024-05-15 21:53:58] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512] [2024-05-15 21:53:58] INFO Hashes match [2024-05-15 21:53:58] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:53:59] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:54:02] INFO Hashes match [2024-05-15 21:54:02] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:54:02] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:54:02] INFO VolumeAttachment deleted [2024-05-15 21:54:03] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:54:06] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:54:06] INFO VolumeAttachment deleted [2024-05-15 21:54:06] INFO Deleting all resources in namespace volumeio-test-4af2b8d8 [2024-05-15 21:54:18] INFO Namespace volumeio-test-4af2b8d8 was deleted in 12.015406783s [2024-05-15 21:54:19] INFO SUCCESS: VolumeIoSuite in 2m15.417295236s [2024-05-15 21:54:19] INFO Started generating reports... Collecting metrics 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/sCollecting metrics [2024-05-15 21:54:19] INFO Started generating reports... Generating plots 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------] 100.00% 211 p/s[2024-05-15 21:54:19] WARN No ResourceUsageMetrics provided [2024-05-15 21:54:19] ERROR no ResourceUsageMetrics provided report-test-run-fd465f33: Name: test-run-fd465f33 Host: https://10.247.103.251:6443 StorageClass: op-e2e-vxflexos Minimum and Maximum EntityOverTime charts: /root/.cert-csi/reports/test-run-fd465f33/PodsCreatingOverTime.png /root/.cert-csi/reports/test-run-fd465f33/PodsReadyOverTime.png /root/.cert-csi/reports/test-run-fd465f33/PodsTerminatingOverTime.png /root/.cert-csi/reports/test-run-fd465f33/PvcsCreatingOverTime.png /root/.cert-csi/reports/test-run-fd465f33/PvcsBoundOverTime.png Tests: -------------------------------------------------------------- 1. TestCase: VolumeIoSuite Started: 2024-05-15 21:52:03.67127787 -0400 -0400 Ended: 2024-05-15 21:54:19.097896816 -0400 -0400 Result: SUCCESS Stage metrics: PVCAttachment: Avg: 464.132515ms Min: 449.38483ms Max: 478.8802ms Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCAttachment.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCAttachment_boxplot.png PVCBind: Avg: 1m22.79835052s Min: 1m22.782330732s Max: 1m22.814370309s Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCBind.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCBind_boxplot.png PVCCreation: Avg: 1m23.511649057s Min: 1m23.50108966s Max: 1m23.522208455s Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCCreation.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCCreation_boxplot.png PVCDeletion: Avg: 13.190385ms Min: 10.080169ms Max: 16.300602ms Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCDeletion_boxplot.png PVCUnattachment: Avg: 425.240106ms Min: 412.278557ms Max: 438.201655ms Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCUnattachment.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PVCUnattachment_boxplot.png PodCreation: Avg: 51.211090277s Min: 5.84238894s Max: 1m35.466752723s Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PodCreation.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PodCreation_boxplot.png PodDeletion: Avg: 3.747993474s Min: 1.437965853s Max: 6.028372306s Histogram: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PodDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/PodDeletion_boxplot.png EntityNumberOverTime: /root/.cert-csi/reports/test-run-fd465f33/VolumeIoSuite6/EntityNumberOverTime.png [2024-05-15 21:54:19] INFO Avg time of a run: 122.77s [2024-05-15 21:54:19] INFO Avg time of a del: 12.02s [2024-05-15 21:54:19] INFO Avg time of all: 135.41s [2024-05-15 21:54:19] INFO During this run 100.0% of suites succeeded STEP: Executing Upgrade from custom resource [2] to [3] @ 05/15/24 21:54:19.593 STEP: Executing Validate custom resource [3] @ 05/15/24 21:54:19.619 STEP: Executing Validate [powerflex] driver from CR [3] is installed @ 05/15/24 21:54:19.624 STEP: Executing Validate [authorization] module from CR [3] is installed @ 05/15/24 21:54:19.629 map[com.dell.karavi-authorization-proxy:true deployment.kubernetes.io/revision:1] map[com.dell.karavi-authorization-proxy:true deprecated.daemonset.template.generation:1] STEP: Executing Run custom test @ 05/15/24 21:54:19.658 May 15 21:54:19.658: INFO: Running ./cert-csi [test vio --sc op-e2e-vxflexos --chainNumber 2 --chainLength 2] [2024-05-15 21:54:19] INFO Starting cert-csi; ver. 0.8.1 [2024-05-15 21:54:19] INFO Using EVENT observer type [2024-05-15 21:54:19] INFO Using config from /root/.kube/config [2024-05-15 21:54:19] INFO Successfully loaded config. Host: https://10.247.103.251:6443 [2024-05-15 21:54:19] INFO Created new KubeClient [2024-05-15 21:54:19] INFO Running 1 iteration(s) [2024-05-15 21:54:19] INFO *** ITERATION NUMBER 1 *** [2024-05-15 21:54:19] INFO Starting VolumeIoSuite with op-e2e-vxflexos storage class [2024-05-15 21:54:19] INFO Successfully created namespace volumeio-test-a9962c8e [2024-05-15 21:54:19] INFO Using default number of volumes [2024-05-15 21:54:19] INFO Creating IO pod [2024-05-15 21:54:19] INFO Waiting for pod iowriter-test-96plz to be READY [2024-05-15 21:54:19] INFO Waiting for pod iowriter-test-w7kjh to be READY [2024-05-15 21:54:55] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:54:56] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:54:58] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:54:59] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:55:03] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:55:03] INFO VolumeAttachment deleted [2024-05-15 21:55:03] INFO Waiting for pod iowriter-test-2f52r to be READY [2024-05-15 21:55:03] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:55:03] INFO VolumeAttachment deleted [2024-05-15 21:55:03] INFO Waiting for pod iowriter-test-b8c5x to be READY [2024-05-15 21:55:13] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512] [2024-05-15 21:55:13] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512] [2024-05-15 21:55:16] INFO Hashes match [2024-05-15 21:55:16] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-05-15 21:55:16] INFO Hashes match [2024-05-15 21:55:16] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-05-15 21:55:17] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-05-15 21:55:20] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-05-15 21:55:20] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:55:20] INFO VolumeAttachment deleted [2024-05-15 21:55:23] INFO Waiting until no Volume Attachments with PV left [2024-05-15 21:55:23] INFO VolumeAttachment deleted [2024-05-15 21:55:23] INFO Deleting all resources in namespace volumeio-test-a9962c8e [2024-05-15 21:55:35] INFO Namespace volumeio-test-a9962c8e was deleted in 12.011089004s Collecting metrics [2024-05-15 21:55:40] INFO SUCCESS: VolumeIoSuite in 1m20.206515629s [2024-05-15 21:55:40] INFO Started generating reports... 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/s[2024-05-15 21:55:40] INFO Started generating reports... Collecting metrics Generating plots 1 / 1 [------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------] 100.00% 139 p/s[2024-05-15 21:55:40] WARN No ResourceUsageMetrics provided [2024-05-15 21:55:40] ERROR no ResourceUsageMetrics provided report-test-run-02d6cd69: Name: test-run-02d6cd69 Host: https://10.247.103.251:6443 StorageClass: op-e2e-vxflexos Minimum and Maximum EntityOverTime charts: /root/.cert-csi/reports/test-run-02d6cd69/PodsCreatingOverTime.png /root/.cert-csi/reports/test-run-02d6cd69/PodsReadyOverTime.png /root/.cert-csi/reports/test-run-02d6cd69/PodsTerminatingOverTime.png /root/.cert-csi/reports/test-run-02d6cd69/PvcsCreatingOverTime.png /root/.cert-csi/reports/test-run-02d6cd69/PvcsBoundOverTime.png Tests: -------------------------------------------------------------- 1. TestCase: VolumeIoSuite Started: 2024-05-15 21:54:19.78384468 -0400 -0400 Ended: 2024-05-15 21:55:39.997881894 -0400 -0400 Result: SUCCESS Stage metrics: PVCAttachment: Avg: 724.477976ms Min: 566.568252ms Max: 882.387701ms Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCAttachment.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCAttachment_boxplot.png PVCBind: Avg: 27.944524061s Min: 27.933614677s Max: 27.955433445s Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCBind.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCBind_boxplot.png PVCCreation: Avg: 28.769260295s Min: 28.614624732s Max: 28.923895858s Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCCreation.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCCreation_boxplot.png PVCDeletion: Avg: 11.91705ms Min: 8.863885ms Max: 14.970215ms Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCDeletion_boxplot.png PVCUnattachment: Avg: 420.542368ms Min: 383.803223ms Max: 457.281514ms Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCUnattachment.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PVCUnattachment_boxplot.png PodCreation: Avg: 22.9668187s Min: 9.527167182s Max: 37.335440768s Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PodCreation.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PodCreation_boxplot.png PodDeletion: Avg: 3.218158127s Min: 1.868362767s Max: 5.559040846s Histogram: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PodDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/PodDeletion_boxplot.png EntityNumberOverTime: /root/.cert-csi/reports/test-run-02d6cd69/VolumeIoSuite7/EntityNumberOverTime.png [2024-05-15 21:55:40] INFO Avg time of a run: 63.22s [2024-05-15 21:55:40] INFO Avg time of a del: 12.01s [2024-05-15 21:55:40] INFO Avg time of all: 80.20s [2024-05-15 21:55:40] INFO During this run 100.0% of suites succeeded STEP: Executing Enable forceRemoveDriver on CR [3] @ 05/15/24 21:55:40.491 STEP: Executing Delete custom resource [3] @ 05/15/24 21:55:40.509 STEP: Executing Delete custom resource [1] @ 05/15/24 21:55:40.532 STEP: Executing Restore template [testfiles/powerflex-templates/csm-authorization-config.json] for [pflexAuthSidecar] @ 05/15/24 21:55:40.546 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflexAuth] @ 05/15/24 21:55:40.653 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 05/15/24 21:55:40.735 STEP: Ending: Install PowerFlex Driver (With Authorization), Upgrade driver and authorization sidecar @ 05/15/24 21:55:40.852 • [685.055 seconds] ------------------------------ Ran 1 of 1 Specs in 685.062 seconds SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped PASS Ginkgo ran 1 suite in 12m13.952961309s Test Suite Passed [root@master-1-cKPzXSI7IkXr5 e2e]#