Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detach volume groups before deleting machine #460

Merged
merged 5 commits into from
Jul 26, 2024
Merged

Conversation

thunderboltsid
Copy link
Contributor

@thunderboltsid thunderboltsid commented Jul 25, 2024

This commit is a workaround to a known problem with hypervisor attached volumes that are created by CSI 3.0. Hypervisor attached volumes result in a volume group disk being attached to the VM on which the pod consuming the PV is running. When a cluster is deleted before deleting the PVC, the prism task fails to delete the VM because an existing volume group is attached to the VM. Since this problem is restricted to CSI 3.0 which relies on a minimum PC version of 2024.1 and the solution requires the volume group detach v4 API which is also available starting 2024.1, we make a v3 get prism central info call to fetch the PC version. If PC version is 2024 or greater we create a v4 client for the cluster as well. Before a machine is deleted, we check if the VM backing the machine has any disks backed by volume groups. If it does, we detach those volume groups before deleting the machine.

How has this been tested?

$ LABEL_FILTERS="csi3" make test-e2e-cilium
CNI="/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/cni/cilium/cilium.yaml" GIT_COMMIT="db33004f38fcc273f13b56d3f37b2413408e0570" GINKGO_SKIP= make test-e2e
make[1]: Entering directory '/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix'
echo "Git commit hash: db33004f38fcc273f13b56d3f37b2413408e0570"
Git commit hash: db33004f38fcc273f13b56d3f37b2413408e0570
KO_DOCKER_REPO=ko.local GOFLAGS="-ldflags=-X=main.gitCommitHash=db33004f38fcc273f13b56d3f37b2413408e0570" ko build -B --platform=linux/amd64 -t e2e-db33004f38fcc273f13b56d3f37b2413408e0570 .
2024/07/26 15:39:49 Using base cgr.dev/chainguard/static:latest@sha256:d7518504f59dacbc90852349c0878871199cefd4bed7952d2eeb7cc3ddbe69e5 for github.com/nutanix-cloud-native/cluster-api-provider-nutanix
2024/07/26 15:39:50 Building github.com/nutanix-cloud-native/cluster-api-provider-nutanix for linux/amd64
2024/07/26 15:39:55 Loading ko.local/cluster-api-provider-nutanix:3596bbfb8401072ee5f2657ed6877b43cad58c20b487a3e7b4e07856f7af2198
2024/07/26 15:39:56 Loaded ko.local/cluster-api-provider-nutanix:3596bbfb8401072ee5f2657ed6877b43cad58c20b487a3e7b4e07856f7af2198
2024/07/26 15:39:56 Adding tag e2e-db33004f38fcc273f13b56d3f37b2413408e0570
2024/07/26 15:39:56 Added tag e2e-db33004f38fcc273f13b56d3f37b2413408e0570
ko.local/cluster-api-provider-nutanix:3596bbfb8401072ee5f2657ed6877b43cad58c20b487a3e7b4e07856f7af2198
docker tag ko.local/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570 harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-secret --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-secret.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-nutanix-cluster --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-nutanix-cluster.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-additional-categories --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-additional-categories.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-nmt --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-no-nmt.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-project --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-project.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-upgrades --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-upgrades.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-md-remediation --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-md-remediation.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-kcp-remediation --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-kcp-remediation.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-kcp-scale-in --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-kcp-scale-in.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-csi --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-csi.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-csi3 --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-csi3.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-failure-domains --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-failure-domains.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-clusterclass --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-clusterclass.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-clusterclass --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/clusterclass-nutanix-quick-start.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-topology --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1beta1/cluster-template-topology.yaml
kustomize build /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1.3.5/cluster-template --load-restrictor LoadRestrictionsNone > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/infrastructure-nutanix/v1.3.5/cluster-template.yaml
kustomize build templates/base > templates/cluster-template.yaml
kustomize build templates/csi > templates/cluster-template-csi.yaml
kustomize build templates/csi3 > templates/cluster-template-csi3.yaml
kustomize build templates/clusterclass > templates/cluster-template-clusterclass.yaml
kustomize build templates/topology > templates/cluster-template-topology.yaml
echo "Image tag for E2E test is e2e-db33004f38fcc273f13b56d3f37b2413408e0570"
Image tag for E2E test is e2e-db33004f38fcc273f13b56d3f37b2413408e0570
LOCAL_PROVIDER_VERSION=v1.4.99 \
	MANAGER_IMAGE=harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570 \
	envsubst < /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/config/nutanix.yaml > /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/config/nutanix.yaml.tmp
docker tag ko.local/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570 harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570
docker push harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570
The push refers to repository [harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix]
1a62ce8decff: Pushed
ffe56a1c5f38: Layer already exists
935a6850a620: Layer already exists
e2e-db33004f38fcc273f13b56d3f37b2413408e0570: digest: sha256:d4814708b197768d1d88ec7ca26da5fb48e8672432230b5461aab9e65d175fe0 size: 946
mkdir -p /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/_artifacts
NUTANIX_LOG_LEVEL=debug ginkgo -v \
	--trace \
	--tags=e2e \
	--label-filter="!only-for-validation && csi3" \
	--skip="" \
	--focus="" \
	--nodes=1 \
	--no-color=false \
	--output-dir="/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/_artifacts" \
	--junit-report="junit.e2e_suite.1.xml" \
	--timeout="24h" \
	 \
	./test/e2e -- \
	-e2e.artifacts-folder="/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/_artifacts" \
	-e2e.config="/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/config/nutanix.yaml.tmp" \
	-e2e.skip-resource-cleanup=false \
	-e2e.use-existing-cluster=false
Running Suite: capx-e2e - /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e
========================================================================================================================
Random Seed: 1722001375

Will run 2 of 101 specs
------------------------------
[SynchronizedBeforeSuite]
/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/e2e_suite_test.go:69
  STEP: Initializing a runtime.Scheme with all the GVK relevant for this test @ 07/26/24 15:43:04.011
  STEP: Loading the e2e test configuration from "/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/config/nutanix.yaml.tmp" @ 07/26/24 15:43:04.011
  STEP: Creating a clusterctl local repository into "/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/_artifacts" @ 07/26/24 15:43:04.012
  STEP: Reading the ClusterResourceSet manifest /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/data/cni/cilium/cilium.yaml @ 07/26/24 15:43:04.012
  STEP: Setting up the bootstrap cluster @ 07/26/24 15:43:09.541
  STEP: Creating the bootstrap cluster @ 07/26/24 15:43:09.541
  INFO: Creating a kind cluster with name "test-ydj8nx"
Creating cluster "test-ydj8nx" ...
 • Ensuring node image (kindest/node:v1.30.2) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.30.2) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
  INFO: The kubeconfig file for the kind cluster is /var/folders/g3/lb827bt96z10xz_m_c2xn93w0000gp/T/e2e-kind1693377008
  INFO: Loading image: "harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570"
  INFO: Image harbor.eng.nutanix.com/ncn-ci/cluster-api-provider-nutanix:e2e-db33004f38fcc273f13b56d3f37b2413408e0570 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/cluster-api-controller:v1.6.2"
  INFO: Image registry.k8s.io/cluster-api/cluster-api-controller:v1.6.2 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.6.2"
  INFO: Image registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.6.2 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/kubeadm-control-plane-controller:v1.6.2"
  INFO: Image registry.k8s.io/cluster-api/kubeadm-control-plane-controller:v1.6.2 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/cluster-api-controller:v1.7.3"
  INFO: Image registry.k8s.io/cluster-api/cluster-api-controller:v1.7.3 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.7.3"
  INFO: Image registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.7.3 is present in local container image cache
  INFO: Loading image: "registry.k8s.io/cluster-api/kubeadm-control-plane-controller:v1.7.3"
  INFO: Image registry.k8s.io/cluster-api/kubeadm-control-plane-controller:v1.7.3 is present in local container image cache
  STEP: Overriding coreDNS resolver @ 07/26/24 15:43:28.22
  STEP: Initializing the bootstrap cluster @ 07/26/24 15:43:28.241
  INFO: clusterctl init --config /Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/_artifacts/repository/clusterctl-config.yaml --kubeconfig /var/folders/g3/lb827bt96z10xz_m_c2xn93w0000gp/T/e2e-kind1693377008 --wait-providers --core cluster-api --bootstrap kubeadm --control-plane kubeadm --infrastructure nutanix
  INFO: Waiting for provider controllers to be running
  STEP: Waiting for deployment capi-kubeadm-bootstrap-system/capi-kubeadm-bootstrap-controller-manager to be available @ 07/26/24 15:44:00.432
  INFO: Creating log watcher for controller capi-kubeadm-bootstrap-system/capi-kubeadm-bootstrap-controller-manager, pod capi-kubeadm-bootstrap-controller-manager-659b5fb778-n2rtl, container manager
  STEP: Waiting for deployment capi-kubeadm-control-plane-system/capi-kubeadm-control-plane-controller-manager to be available @ 07/26/24 15:44:00.559
  INFO: Creating log watcher for controller capi-kubeadm-control-plane-system/capi-kubeadm-control-plane-controller-manager, pod capi-kubeadm-control-plane-controller-manager-696879d594-9c5h9, container manager
  STEP: Waiting for deployment capi-system/capi-controller-manager to be available @ 07/26/24 15:44:00.571
  INFO: Creating log watcher for controller capi-system/capi-controller-manager, pod capi-controller-manager-6f69847fd8-6trkw, container manager
  STEP: Waiting for deployment capx-system/capx-controller-manager to be available @ 07/26/24 15:44:00.58
  INFO: Creating log watcher for controller capx-system/capx-controller-manager, pod capx-controller-manager-5995c78987-bt8g7, container kube-rbac-proxy
  INFO: Creating log watcher for controller capx-system/capx-controller-manager, pod capx-controller-manager-5995c78987-bt8g7, container manager
[SynchronizedBeforeSuite] PASSED [56.839 seconds]
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nutanix flavor CSI Create a cluster with Nutanix CSI 3.0 and use Nutanix Volumes to create PV [capx-feature-test, csi, csi3]
/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/csi_test.go:224
  STEP: Creating a namespace for hosting the "cluster-csi" test spec @ 07/26/24 15:44:00.851
  INFO: Creating namespace cluster-csi-q3gzwq
  INFO: Creating event watcher for namespace "cluster-csi-q3gzwq"
  STEP: Creating a workload cluster @ 07/26/24 15:44:00.861
  INFO: Creating the workload cluster with name "cluster-csi-l63lnd" using the "csi3" template (Kubernetes v1.29.2, 1 control-plane machines, 1 worker machines)
  INFO: Getting the cluster template yaml
  INFO: clusterctl config cluster cluster-csi-l63lnd --infrastructure (default) --kubernetes-version v1.29.2 --control-plane-machine-count 1 --worker-machine-count 1 --flavor csi3
  INFO: Creating the workload cluster with name "cluster-csi-l63lnd" from the provided yaml
  INFO: Applying the cluster template yaml of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd
Running kubectl apply --kubeconfig /var/folders/g3/lb827bt96z10xz_m_c2xn93w0000gp/T/e2e-kind1693377008 -f -
stdout:
configmap/cluster-csi-l63lnd-pc-trusted-ca-bundle created
configmap/nutanix-ccm created
configmap/nutanix-csi created
configmap/cni-cluster-csi-l63lnd-crs-cni created
secret/cluster-csi-l63lnd created
secret/nutanix-ccm-secret created
secret/nutanix-csi-secret created
clusterresourceset.addons.cluster.x-k8s.io/nutanix-ccm-crs created
clusterresourceset.addons.cluster.x-k8s.io/nutanix-csi-crs created
clusterresourceset.addons.cluster.x-k8s.io/cluster-csi-l63lnd-crs-cni created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/cluster-csi-l63lnd-kcfg-0 created
cluster.cluster.x-k8s.io/cluster-csi-l63lnd created
machinedeployment.cluster.x-k8s.io/cluster-csi-l63lnd-wmd created
machinehealthcheck.cluster.x-k8s.io/cluster-csi-l63lnd-mhc created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cluster-csi-l63lnd-kcp created
nutanixcluster.infrastructure.cluster.x-k8s.io/cluster-csi-l63lnd created
nutanixmachinetemplate.infrastructure.cluster.x-k8s.io/cluster-csi-l63lnd-mt-0 created

  INFO: Waiting for the cluster infrastructure of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be provisioned
  STEP: Waiting for cluster to enter the provisioned phase @ 07/26/24 15:44:01.607
  INFO: Waiting for control plane of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be initialized
  INFO: Waiting for the first control plane machine managed by cluster-csi-q3gzwq/cluster-csi-l63lnd-kcp to be provisioned
  STEP: Waiting for one control plane node to exist @ 07/26/24 15:44:11.624
  INFO: Waiting for control plane of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be ready
  INFO: Waiting for control plane cluster-csi-q3gzwq/cluster-csi-l63lnd-kcp to be ready (implies underlying nodes to be ready as well)
  STEP: Waiting for the control plane to be ready @ 07/26/24 15:46:01.732
  STEP: Checking all the control plane machines are in the expected failure domains @ 07/26/24 15:46:21.778
  INFO: Waiting for the machine deployments of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be provisioned
  STEP: Waiting for the workload nodes to exist @ 07/26/24 15:46:21.795
  STEP: Checking all the machines controlled by cluster-csi-l63lnd-wmd are in the "<None>" failure domain @ 07/26/24 15:47:01.863
  INFO: Waiting for the machine pools of cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be provisioned
  STEP: Fetching workload client @ 07/26/24 15:47:01.879
  STEP: Checking if CSI namespace exists @ 07/26/24 15:47:01.894
  STEP: Checking if CSI deployment exists @ 07/26/24 15:47:02.661
  STEP: Creating CSI Storage class @ 07/26/24 15:47:03.091
  STEP: Creating CSI PVC @ 07/26/24 15:47:03.47
  STEP: Creating a pod using the PVC to ensure VG is attached to the VM @ 07/26/24 15:47:03.659
  STEP: Checking if the pod is running @ 07/26/24 15:47:03.852
  STEP: Checking CSI PVC status is bound @ 07/26/24 15:47:03.852
  STEP: Deleting the pod @ 07/26/24 15:47:34.613
  STEP: Deleting CSI PVC @ 07/26/24 15:47:34.815
  STEP: Dumping logs from the "cluster-csi-l63lnd" workload cluster @ 07/26/24 15:47:35.004
Failed to get logs for Machine cluster-csi-l63lnd-kcp-6tm22, Cluster cluster-csi-q3gzwq/cluster-csi-l63lnd: [error creating container exec: Error response from daemon: No such container: cluster-csi-l63lnd-kcp-6tm22, : error creating container exec: Error response from daemon: No such container: cluster-csi-l63lnd-kcp-6tm22]
Failed to get logs for Machine cluster-csi-l63lnd-wmd-fkskc-x6t6l, Cluster cluster-csi-q3gzwq/cluster-csi-l63lnd: [error creating container exec: Error response from daemon: No such container: cluster-csi-l63lnd-wmd-fkskc-x6t6l, : error creating container exec: Error response from daemon: No such container: cluster-csi-l63lnd-wmd-fkskc-x6t6l]
Failed to get infrastructure logs for Cluster cluster-csi-q3gzwq/cluster-csi-l63lnd: failed to inspect container "cluster-csi-l63lnd-lb": Error response from daemon: No such container: cluster-csi-l63lnd-lb
  STEP: Dumping all the Cluster API resources in the "cluster-csi-q3gzwq" namespace @ 07/26/24 15:47:35.052
  STEP: Deleting cluster cluster-csi-q3gzwq/cluster-csi-l63lnd @ 07/26/24 15:47:35.189
  STEP: Deleting cluster cluster-csi-q3gzwq/cluster-csi-l63lnd @ 07/26/24 15:47:35.192
  INFO: Waiting for the Cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be deleted
  STEP: Waiting for cluster cluster-csi-q3gzwq/cluster-csi-l63lnd to be deleted @ 07/26/24 15:47:35.198
  STEP: Deleting namespace used for hosting the "cluster-csi" test spec @ 07/26/24 15:48:55.261
  INFO: Deleting namespace cluster-csi-q3gzwq
• [294.418 seconds]
------------------------------
Nutanix flavor CSI Create a cluster with Nutanix CSI 3.0 and use Nutanix Volumes to create PV and delete cluster without deleting PVC [capx-feature-test, csi, csi3]
/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/csi_test.go:384
  STEP: Creating a namespace for hosting the "cluster-csi" test spec @ 07/26/24 15:48:55.27
  INFO: Creating namespace cluster-csi-1ldvl7
  INFO: Creating event watcher for namespace "cluster-csi-1ldvl7"
  STEP: Creating a workload cluster @ 07/26/24 15:48:55.28
  INFO: Creating the workload cluster with name "cluster-csi-czg4ev" using the "csi3" template (Kubernetes v1.29.2, 1 control-plane machines, 1 worker machines)
  INFO: Getting the cluster template yaml
  INFO: clusterctl config cluster cluster-csi-czg4ev --infrastructure (default) --kubernetes-version v1.29.2 --control-plane-machine-count 1 --worker-machine-count 1 --flavor csi3
  INFO: Creating the workload cluster with name "cluster-csi-czg4ev" from the provided yaml
  INFO: Applying the cluster template yaml of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev
Running kubectl apply --kubeconfig /var/folders/g3/lb827bt96z10xz_m_c2xn93w0000gp/T/e2e-kind1693377008 -f -
stdout:
configmap/cluster-csi-czg4ev-pc-trusted-ca-bundle created
configmap/nutanix-ccm created
configmap/nutanix-csi created
configmap/cni-cluster-csi-czg4ev-crs-cni created
secret/cluster-csi-czg4ev created
secret/nutanix-ccm-secret created
secret/nutanix-csi-secret created
clusterresourceset.addons.cluster.x-k8s.io/nutanix-ccm-crs created
clusterresourceset.addons.cluster.x-k8s.io/nutanix-csi-crs created
clusterresourceset.addons.cluster.x-k8s.io/cluster-csi-czg4ev-crs-cni created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/cluster-csi-czg4ev-kcfg-0 created
cluster.cluster.x-k8s.io/cluster-csi-czg4ev created
machinedeployment.cluster.x-k8s.io/cluster-csi-czg4ev-wmd created
machinehealthcheck.cluster.x-k8s.io/cluster-csi-czg4ev-mhc created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cluster-csi-czg4ev-kcp created
nutanixcluster.infrastructure.cluster.x-k8s.io/cluster-csi-czg4ev created
nutanixmachinetemplate.infrastructure.cluster.x-k8s.io/cluster-csi-czg4ev-mt-0 created

  INFO: Waiting for the cluster infrastructure of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be provisioned
  STEP: Waiting for cluster to enter the provisioned phase @ 07/26/24 15:48:55.67
  INFO: Waiting for control plane of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be initialized
  INFO: Waiting for the first control plane machine managed by cluster-csi-1ldvl7/cluster-csi-czg4ev-kcp to be provisioned
  STEP: Waiting for one control plane node to exist @ 07/26/24 15:49:35.706
  INFO: Waiting for control plane of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be ready
  INFO: Waiting for control plane cluster-csi-1ldvl7/cluster-csi-czg4ev-kcp to be ready (implies underlying nodes to be ready as well)
  STEP: Waiting for the control plane to be ready @ 07/26/24 15:51:45.832
  STEP: Checking all the control plane machines are in the expected failure domains @ 07/26/24 15:51:55.851
  INFO: Waiting for the machine deployments of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be provisioned
  STEP: Waiting for the workload nodes to exist @ 07/26/24 15:51:55.867
  STEP: Checking all the machines controlled by cluster-csi-czg4ev-wmd are in the "<None>" failure domain @ 07/26/24 15:52:45.944
  INFO: Waiting for the machine pools of cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be provisioned
  STEP: Fetching workload client @ 07/26/24 15:52:45.962
  STEP: Checking if CSI namespace exists @ 07/26/24 15:52:45.976
  STEP: Checking if CSI deployment exists @ 07/26/24 15:52:46.745
  STEP: Creating CSI Storage class @ 07/26/24 15:52:47.171
  STEP: Creating CSI PVC @ 07/26/24 15:52:47.548
  STEP: Creating a pod using the PVC to ensure VG is attached to the VM @ 07/26/24 15:52:47.737
  STEP: Checking if the pod is running @ 07/26/24 15:52:47.928
  STEP: Checking CSI PVC status is bound @ 07/26/24 15:52:47.929
  STEP: Dumping logs from the "cluster-csi-czg4ev" workload cluster @ 07/26/24 15:53:28.897
Failed to get logs for Machine cluster-csi-czg4ev-kcp-5rqwg, Cluster cluster-csi-1ldvl7/cluster-csi-czg4ev: [error creating container exec: Error response from daemon: No such container: cluster-csi-czg4ev-kcp-5rqwg, : error creating container exec: Error response from daemon: No such container: cluster-csi-czg4ev-kcp-5rqwg]
Failed to get logs for Machine cluster-csi-czg4ev-wmd-vzw29-jlt8j, Cluster cluster-csi-1ldvl7/cluster-csi-czg4ev: [error creating container exec: Error response from daemon: No such container: cluster-csi-czg4ev-wmd-vzw29-jlt8j, : error creating container exec: Error response from daemon: No such container: cluster-csi-czg4ev-wmd-vzw29-jlt8j]
Failed to get infrastructure logs for Cluster cluster-csi-1ldvl7/cluster-csi-czg4ev: failed to inspect container "cluster-csi-czg4ev-lb": Error response from daemon: No such container: cluster-csi-czg4ev-lb
  STEP: Dumping all the Cluster API resources in the "cluster-csi-1ldvl7" namespace @ 07/26/24 15:53:28.939
  STEP: Deleting cluster cluster-csi-1ldvl7/cluster-csi-czg4ev @ 07/26/24 15:53:29.057
  STEP: Deleting cluster cluster-csi-1ldvl7/cluster-csi-czg4ev @ 07/26/24 15:53:29.059
  INFO: Waiting for the Cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be deleted
  STEP: Waiting for cluster cluster-csi-1ldvl7/cluster-csi-czg4ev to be deleted @ 07/26/24 15:53:29.064
  STEP: Deleting namespace used for hosting the "cluster-csi" test spec @ 07/26/24 15:54:09.097
  INFO: Deleting namespace cluster-csi-1ldvl7
• [313.843 seconds]
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[SynchronizedAfterSuite]
/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix/test/e2e/e2e_suite_test.go:157
  STEP: Dumping logs from the bootstrap cluster @ 07/26/24 15:54:09.114
  STEP: Tearing down the management cluster @ 07/26/24 15:54:09.234
[SynchronizedAfterSuite] PASSED [0.581 seconds]
------------------------------
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
[ReportAfterSuite] PASSED [0.013 seconds]
------------------------------

Ran 2 of 101 Specs in 665.683 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 99 Skipped
PASS

Ginkgo ran 1 suite in 11m13.843363042s
Test Suite Passed
make[1]: Leaving directory '/Users/sid.shukla/go/src/github.com/nutanix-cloud-native/cluster-api-provider-nutanix'

This commit is a workaround to a known problem with hypervisor attached
volumes that are created by CSI 3.0. Hypervisor attached volumes result
in a volume group disk being attached to the VM on which the pod consuming
the PV is running. When a cluster is deleted before deleting the PVC, the
prism task fails to delete the VM because an existing volume group is attached
to the VM. Since this problem is restricted to CSI 3.0 which relies on a
minimum PC version of 2024.1 and the solution requires the volume group
detach v4 API which is also available starting 2024.1, we make a v3
get prism central info call to fetch the PC version. If PC version is
2024 or greater we create a v4 client for the cluster as well. Before a machine
is deleted, we check if the VM backing the machine has any disks backed by
volume groups. If it does, we detach those volume groups before deleting the
machine.
Copy link

codecov bot commented Jul 25, 2024

Codecov Report

Attention: Patch coverage is 1.63934% with 120 lines in your changes missing coverage. Please review.

Project coverage is 29.35%. Comparing base (9ebba60) to head (81fbace).

Files Patch % Lines
controllers/helpers.go 2.32% 84 Missing ⚠️
controllers/nutanixmachine_controller.go 0.00% 26 Missing ⚠️
controllers/nutanixcluster_controller.go 0.00% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #460      +/-   ##
==========================================
- Coverage   31.69%   29.35%   -2.35%     
==========================================
  Files          14       14              
  Lines        1366     1475     +109     
==========================================
  Hits          433      433              
- Misses        933     1042     +109     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@deepakm-ntnx
Copy link
Contributor

as discussed, lets update https://github.com/nutanix-cloud-native/cluster-api-provider-nutanix/blob/main/scripts/csi_nutanix_update.sh as well or add a csi 3.0 specific one

@deepakm-ntnx
Copy link
Contributor

Also, as discussed, lets post the results of using a user with role "Kubernetes Infrastructure Provision" and try create/delete the cluster to verify if any permissions are required if possible

controllers/nutanixcluster_controller.go Show resolved Hide resolved
controllers/nutanixmachine_controller.go Outdated Show resolved Hide resolved
controllers/nutanixmachine_controller.go Outdated Show resolved Hide resolved
controllers/helpers.go Outdated Show resolved Hide resolved
test/e2e/csi_test.go Show resolved Hide resolved
test/e2e/csi_test.go Show resolved Hide resolved
test/e2e/csi_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@dkoshkin dkoshkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@deepakm-ntnx
Copy link
Contributor

/retest

@deepakm-ntnx deepakm-ntnx merged commit 412d9d4 into main Jul 26, 2024
4 of 9 checks passed
@deepakm-ntnx deepakm-ntnx deleted the jira/NCN-101558 branch July 26, 2024 17:21
@thunderboltsid thunderboltsid added the enhancement New feature or request label Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants