E2E Updates

tests/README.md

@@ -1,64 +1,94 @@
-# Test for CSM Operator
+# Testing for the CSM Operator
 
-This is the test directory for CSM operator.
+This directory contains the testing infrastructure and E2E test implementation for the csm-operator. There are two kinds of tests for the operator: [unit tests](#unit-tests) and [end-to-end (E2E)](#e2e-tests) tests.
 
-`config` directory includes yaml files consumed by test cases. For example `driverconfig/powerscale/v2.x.y/node.yaml` is consumed by `pkg/drivers/commonconfig_test.go`.
+## Table of Contents
 
-`shared/clientgoclient` implements kubernetes client from client-go package. It has a getter function for each API version like `AppsV1Interface` or `CoreV1Interface`. `AppsV1Interface` is the one that we need as it has getter function for `daemonsetInterface`. The `daemonsetInterface` has all `Create`, `Apply`, `Delete` etc. methods that we will be using to manipulate Kubernetes runtime objects.
+* [Unit Tests](#unit-tests)
+* [E2E Tests](#e2e-tests)
+  * [Prerequisites](#prerequisites)
+    * [Application Mobility Prerequisites](#application-mobility-prerequisites)
+  * [Run](#run)
+    * [Scenarios File](#scenarios-file)
+  * [Developing E2E Tests](#developing-e2e-tests)
+* [Directory Layout](#directory-layout)
 
-`shared/crclient` implements kubernetes client from controller runtime. It has very similar functionalities as the one above except that it can't do apply. These two clients share the same memory to store runtime objects.
+# Unit Tests
 
-## Unit Test
+The unit tests are quick, easy-to-run tests that verify the functinality of the methods in different packages. The only major requirement for unit tests is that Go is installed. The unit tests are automatically run by GitHub actions as part of any PR targeting the main branch, and must pass with sufficient coverage for the PR to be merged.
 
-To run unit test, go to the root directory of this project and run `make unit-test`. It will output a report of tests being run.
+To run unit tests, go to the root directory of this project and run `make <component>-unit-test`. Components include `controller` (controllers package), `module` (modules package), and `driver` (drivers package).
 
-## E2E Test
+# E2E Tests
 
-The E2E tests test the installation of Dell CSM Drivers and Modules.
+The end-to-end tests test the functionality of the operator as a whole by installing different combinations of drivers and modules, enabling and disabling components, and verifying the installed functionality (using e.g. cert-csi). The E2E tests need to be run from the master node of a Kubernetes cluster. All test scenarios are specified in `tests/e2e/testfiles/scenarios.yaml` and are tagged by which test suite(s) they are a part of -- test suites include a test suite for each driver and module, as well as a `sanity` suite.
 
-### Prerequisites
+Any time changes made to the operator are being checked into the main branch, sanity tests should be run (they should take 20-30 minutes to complete, the very first run may take a few minutes more). In addition, if you have made any driver- or module-specific changes, (any changes in `pkg/drivers`, `pkg/modules`, `operatorconfig/driverconfig`, `operatorconfig/moduleconfig`, etc), please run the E2E tests specific to these components as well.
 
-- A supported environment where the Dell Container Storage Modules Operator is running and a storageclass is installed.
-- All prerequisites for a specific driver and modules to test. For documentation, please visit [Container Storage Modules documentation](https://dell.github.io/csm-docs/)
-- For tests that configure secret/storageclasses; The following namespaces need to be created beforehand:
-  - isilon
-  - dell
-  - test-vxflexos
-- Ginkgo v1.16.5 is installed. To install, go to `tests/e2e` and run the following commands:
+## Prerequisites
 
+- A supported environment where the Dell Container Storage Modules Operator.
+- Fill in the `array-info.sh` environment variables (more info below).
+- The following namespaces need to be created beforehand:
+  - `dell`
+  - `karavi`
+  - `authorization`
+  - `proxy-ns`
+  - (if running sanity, powerflex, or modules suites) `test-vxflexos`
+  - (if running sanity, powerscale, or modules suites) `isilon`
+- For auth: edit your `/etc/hosts` file to include the following line: `<master node IP> csm-authorization.com`
+- In addition, for drivers that do not use the secret and storageclass creation steps, any required secrets, storageclasses, etc. will need to be created beforehand as well as required namespaces.
+- Ginkgo v2 is installed. To install, go to `tests/e2e` and run the following commands:
 ```bash
 go install github.com/onsi/ginkgo/v2/ginkgo
 go get github.com/onsi/gomega/...
-```    
-#### Application Mobility Prerequisites
-If running the Application Mobility e2e tests, further setup must be done, you must:
-- have a MinIO object storage setup, with default credentials 
-   - At least 2 buckets setup, if instance only has one bucket, set ALT_BUCKET_NAME = BUCKET_NAME
+```
+### Array Information
+For PowerFlex, PowerScale, Authorization, and Application-Mobility, system-specific information (array login credentials, system IDs, endpoints, etc.) need to be provided so that all the required resources (secrets, storageclasses, etc.) can be created by the tests. Example values have been inserted; please replace these with values from your system. Refer to [CSM documentation](https://dell.github.io/csm-docs/docs/) for any further questions about driver or module pre-requisites. 
+
+Please note that, if tests are stopped in the middle of a run, some files in `testfiles/*-templates` folders may remain in a partially modified state and break subsequent test runs. To undo these changes, you can run `git checkout -- <template file>`.
+
+### Application Mobility Prerequisites
+If running the Application Mobility e2e tests, (the sanity suite includes a few simple app-mobility tests), further setup must be done:
+- have a MinIO object storage setup, with default credentials
+   - At least 2 buckets set up, if instance only has one bucket, set ALT_BUCKET_NAME = BUCKET_NAME
 - have all required licenses installed in your testing environment
-- have the latest Application Mobility controller and plugin images 
+- have the latest Application Mobility controller and plugin images
+The application-mobility repo has information on all of these pre-requisites up, including a script to install minio.
 
-### Run
+## Run
 
-To run e2e test, go through the following steps:
+The tests are run by the `run-e2e-test.sh` script in the `tests/e2e` directory.
 
 - Ensure you meet all [prerequisites](https://github.com/dell/csm-operator/blob/main/tests/README.md#prerequisites).
 - Change to the `tests/e2e` directory.
-- Set your environment variables in the file `env-e2e-test.sh`. You MUST set `CERT-CSI` to point to a cert-csi executable.
-- If you want to test any modules, uncomment their environment variables in `env-e2e-test.sh`.
-- Run the e2e test by executing the commands below:
+- Set your array information in the `array-info.sh` file.
+- If you do not have `cert-csi`, `karavictl`, and (for app-mobility) `dellctl` accessible through your `PATH` variable, pass the path to each executable to the script, like so, `run-e2e-test.sh --cert-csi=/path/to/cert-csi --karavictl=/path/to/karavictl`, and they will be added to `/usr/local/bin`
+- Decide on the test suites you want to run, based on the changes made. Available test suites can be seen by running `run-e2e-test.sh -h` If multiple suites are specified, the union (not intersection) of those suites will be run.
+- Run the e2e tests by executing the `run-e2e-test.sh` script with desired options. Three examples are provided:
 
+You have made changes to `controllers/csm_controller.go` and `pkg/drivers/powerflex.go`, and need run sanity and powerflex test suites. Additionally, you do not have cert-csi or karavictl executables in your PATH:
 ```bash
-./run-e2e-test.sh
+run-e2e-test.sh --cert-csi=/path/to/cert-csi --karavictl=/path/to/karavictl --sanity --powerflex
 ```
+You made some changes to `controllers/csm_controller.go`, and need to run sanity tests. cert-csi and karavictl are already in your PATH:
+```bash
+run-e2e-test.sh --sanity
+```
+You made some changes to `pkg/modules/observability.go`, and need to run observability tests. cert-csi and karavictl are already in your path, but you need to update the karavictl binary with a newer one:
+```bash
+run-e2e-test.sh --obs --karavictl=/path/to/karavictl
+```
+### Scenarios File
 
-#### Values File
-
-An e2e test values file is a yaml file that defines all e2e tests to be ran. An excerpt of the file is shown below:
+An e2e test scenarios file is a yaml file that defines all e2e test scenarios to be run. An excerpt of the file is shown below:
 
 ```yaml
 - scenario: "Install PowerScale Driver(Standalone)"
   path: "testfiles/storage_csm_powerscale.yaml"
-  modules:
+  tags:
+    - powerscale
+    - sanity
   steps:
     - "Given an environment with k8s or openshift, and CSM operator installed"
     - "Apply custom resources"
@@ -76,19 +106,21 @@ An e2e test values file is a yaml file that defines all e2e tests to be ran. An
     # Example:
     #   ./hello_world.sh
     #   cert-csi test vio --sc <storage class> --chainNumber 2 --chainLength 2
-    run: ./cert-csi test vio --sc isilon --chainNumber 2 --chainLength 2
+    run:
+      - cert-csi test vio --sc isilon --chainNumber 2 --chainLength 2
 ```
 
 Each test has:
 
 - `scenario`: The name of the test to run
 - `path`: The path to the custom resources yaml file that has the specific configuration you want to test.
+- `tags`: Each test can belong to one or more groups of tests, specified by tags. To see a list of currently available tags, run `./run-e2e-test.sh -h`.
 - `steps`: Steps to take for the specific scenearios. Please note that all steps above and the ones in this sample file `tests/e2e/testfiles/values.yaml` already have a backend implementation. If you desire to use a different step, see [Develop](#develop) for how to add new E2E Test
 - `customTest`: An entrypoint for users to run custom test against their environment. You must have `"Run custom test"` as part of your `steps` above for this custom test to run. This object has the following parameter.
   - `name`: Name of your custom test
-  - `run`: A command line argument that will be run by the e2e test. To ensure the command is accessible from e2e test repo, use absolute paths if you are running a script.
+  - `run`: A list of command line arguments that will be run by the e2e test.
 
-### Develop
+## Developing E2E Tests
 
 Most steps to cover common use cases already have their respective backend implementations. Sometimes we run into a situation where we may need to add a new step. For the sake of illustration, please follow the constraints and steps below to add a new test scenario called `"Install PowerHello Driver(With a module called World)"` to excerpt of yaml file shown above.
 
@@ -189,3 +221,11 @@ Most steps to cover common use cases already have their respective backend imple
             ```
 
         3. [Run your E2E](#run). If you get this error `no method for step: <you step>`, it means you either haven't implemented it or there's a problem with your regex.
+
+# Directory Layout
+
+`config` directory includes yaml files consumed by test cases. For example `driverconfig/powerscale/v2.x.y/node.yaml` is consumed by `pkg/drivers/commonconfig_test.go`.
+
+`shared/clientgoclient` implements kubernetes client from client-go package. It has a getter function for each API version like `AppsV1Interface` or `CoreV1Interface`. `AppsV1Interface` is the one that we need as it has getter function for `daemonsetInterface`. The `daemonsetInterface` has all `Create`, `Apply`, `Delete` etc. methods that we will be using to manipulate Kubernetes runtime objects.
+
+`shared/crclient` implements kubernetes client from controller runtime. It has very similar functionalities as the one above except that it can't do apply. These two clients share the same memory to store runtime objects.

tests/e2e/array-info.sh

@@ -0,0 +1,57 @@
+# Copyright © 2022-2024 Dell Inc. or its subsidiaries. All Rights Reserved.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#      http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# USER MODIFICATION REQUIRED: must supply address of Authorization Proxy Server
+# Since this e2e exposes the Proxy Server via NodePort, you can use a cluster node IP
+export PROXY_HOST="csm-authorization.com"
+
+# The following are Authorization Proxy Server specific:
+# Must supply storage array details
+# Storage type examples - powerscale, powerflex, powermax
+export STORAGE_TYPE="powerscale"
+export END_POINT="1.1.1.1:8080"
+export SYSTEM_ID="Isilon-System-Name"
+export STORAGE_USER="username"
+export STORAGE_PASSWORD="password"
+export STORAGE_POOL="/ifs/data/csi"
+# Must specify and manually create driver namespace
+export DRIVER_NAMESPACE="isilon"
+
+# The following are for creating PFlex secret/storage class
+# do not include "https://" in the endpoint
+export PFLEX_USER="username"
+export PFLEX_PASS="password"
+export PFLEX_SYSTEMID="00990099ddcc"
+export PFLEX_ENDPOINT="10.1.1.1"
+export PFLEX_MDM="10.0.0.1,10.0.0.2"
+export PFLEX_AUTH_ENDPOINT="localhost:9401"  
+export PFLEX_POOL="pool1"
+
+# The following are for creating PScale secret/storage class
+# do not include "https://" in the endpoint
+export PSCALE_CLUSTER="Isilon-System-Name"
+export PSCALE_USER="username"
+export PSCALE_PASS="password"
+export PSCALE_ENDPOINT="1.1.1.1"
+export PSCALE_AUTH_ENDPOINT="localhost"
+export PSCALE_AUTH_PORT="9400"
+
+# The following are for testing AM
+export VOL_NS=wordpress
+export RES_NS=res-wordpress
+export AM_NS=test-vxflexos
+export BACKEND_STORAGE_URL="10.0.0.4:32000"
+export BUCKET_NAME="my-bucket"
+export ALT_BUCKET_NAME="alt-bucket" 
+# Be sure to escape / with \
+export AM_CONTROLLER_IMAGE="dellemc/csm-application-mobility-controller:nightly"
+export AM_PLUGIN_IMAGE="dellemc/csm-application-mobility-velero-plugin:nightly"

tests/e2e/e2e_test.go

@@ -34,14 +34,14 @@ import (
 const (
 	timeout          = time.Minute * 10
 	interval         = time.Second * 10
-	valuesFileEnvVar = "E2E_VALUES_FILE"
+	valuesFileEnvVar = "E2E_SCENARIOS_FILE"
 )
 
 var (
-	testResources    []step.Resource
-	installedModules []string
-	stepRunner       *step.Runner
-	beautify         string
+	testResources []step.Resource
+	tagsSpecified []string
+	stepRunner    *step.Runner
+	beautify      string
 )
 
 func Contains(slice []string, str string) bool {
@@ -53,19 +53,15 @@ func Contains(slice []string, str string) bool {
 	return false
 }
 
-func ContainsModules(modulesRequired []string, modulesInstalled []string) bool {
-	if len(modulesRequired) == 0 && len(modulesInstalled) == 0 {
-		return true
-	}
-
-	for _, moduleName := range modulesRequired {
-		// check to see if we have modules required
-		if Contains(modulesInstalled, moduleName) == false {
-			By(fmt.Sprintf("Required module not installed: %s ", moduleName))
-			return false
+func ContainsTag(scenarioTags []string, tagsSpecified []string) bool {
+	for _, tag := range tagsSpecified {
+		// Check to see if scenario has required tag
+		if Contains(scenarioTags, tag) {
+			return true
 		}
 	}
-	return true
+	By(fmt.Sprintf("No matching tags for scenario"))
+	return false
 }
 
 // TestE2E -
@@ -80,19 +76,19 @@ func TestE2E(t *testing.T) {
 }
 
 var _ = BeforeSuite(func() {
-	moduleEnvVars := []string{"AUTHORIZATION", "REPLICATION", "OBSERVABILITY", "AUTHORIZATIONPROXYSERVER", "RESILIENCY", "APPLICATIONMOBILITY"}
+	tagEnvVars := []string{"AUTHORIZATION", "REPLICATION", "OBSERVABILITY", "AUTHORIZATIONPROXYSERVER", "RESILIENCY", "APPLICATIONMOBILITY", "POWERFLEX", "POWERSCALE", "POWERMAX", "POWERSTORE", "UNITY", "SANITY"}
 	By("Getting test environment variables")
 	valuesFile := os.Getenv(valuesFileEnvVar)
-	Expect(valuesFile).NotTo(BeEmpty(), "Missing environment variable required for tests. E2E_VALUES_FILE must be set.")
+	Expect(valuesFile).NotTo(BeEmpty(), "Missing environment variable required for tests. E2E_SCENARIOS_FILE must be set.")
 
-	for _, moduleEnvVar := range moduleEnvVars {
-		enabled := os.Getenv(moduleEnvVar)
+	for _, tagEnvVar := range tagEnvVars {
+		enabled := os.Getenv(tagEnvVar)
 		if enabled == "true" {
-			installedModules = append(installedModules, strings.ToLower(moduleEnvVar))
+			tagsSpecified = append(tagsSpecified, strings.ToLower(tagEnvVar))
 		}
 	}
 
-	By(fmt.Sprint(installedModules))
+	By(fmt.Sprint(tagsSpecified))
 	By("Reading values file")
 	res, err := step.GetTestResources(valuesFile)
 	if err != nil {
@@ -123,8 +119,8 @@ var _ = Describe("[run-e2e-test] E2E Testing", func() {
 	It("Running all test Given Test Scenarios", func() {
 		for _, test := range testResources {
 			By(fmt.Sprintf("Starting: %s ", test.Scenario.Scenario))
-			if ContainsModules(test.Scenario.Modules, installedModules) == false {
-				By(fmt.Sprintf("Required module %s not installed, skipping", test.Scenario.Modules))
+			if ContainsTag(test.Scenario.Tags, tagsSpecified) == false {
+				By(fmt.Sprintf("Not tagged for this test run, skipping"))
 				By(fmt.Sprintf("Ending: %s\n", test.Scenario.Scenario))
 				continue
 			}