Calico integration Tests #1906

haouc · 2022-03-04T17:47:16Z

What type of PR is this?
This PR is fixing a failing weekly Calico integration tests and replacing it with helm installation with demoed STAR network policy tests suite.

bug
Which issue does this PR fix:
Fixes #1874

What does this PR do / Why do we need it:
We have deprecated support of Calico charts in this repo, but we need do routine integration tests with certain/latest calico version with our VPC CNI. This PR contains the following changes/additions:

Helm installation of Calico operator with a given version of Calico
STAR tests with various network policy settings
Calico tests are no longer using bash scripts. Ginkgo suite is used to implement the tests
New ENVs to config tests: RUN_LATEST_CALICO_VERSION (boolean) that indicates if fetching Calico repo to use the latest Calico released version; CALICO_VERSION (boolean) that sets Calico version to test if the RUN_LATEST_CALICO_VERSION == false
If DEPROVISION == false, EC2 instances will be terminated to avoid impact for future tests in the cluster
Reorganized VPC CNI tests on current image with Calico tests to avoid interference
Due to installing AMD arch based Calico images and STAR resources, during the tests ARM arch nodes will be unschedulable. They will be restored to schedulable state after Calico tests are finished
To utilize the recently added metrics infra, Calico tests also add a new metrics calico_test_status
If the ENV RUN_CALICO_TEST_WITH_PD NOT set, the script will also run PD with Calico by default

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Testing done on this change:

Using specified Calico version:

$ RUN_CALICO_TEST=true CALICO_VERSION=3.21.0 ./scripts/run-integration-tests.sh
...
2022-03-04 07:49:42 [✔]  EKS cluster "cni-test-16533" in "us-west-2" region is ready
ok.
TIMELINE: Upping test cluster took 1079 seconds.
...
=== RUN   TestIntegration
Mar  4 07:49:44.618: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
Running Suite: Amazon VPC CNI Integration Tests
...
Ran 2 of 2 Specs in 31.384 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestIntegration (31.38s)
...
TIMELINE: Current image integration tests took 21 seconds.
Starting Helm installing Tigera operator and running Calico STAR tests
~/environment/go/src/amazon-vpc-cni-k8s/test ~/environment/go/src/amazon-vpc-cni-k8s
Using Calico version 3.21.0 to test   <-----
Running Suite: Calico with VPC CNI e2e Test Suite
=================================================
Random Seed: 1646380248
Will run 22 of 22 specs
...
STEP: Restore ARM64 Nodes Schedulability
 
Ran 22 of 22 Specs in 379.131 seconds
SUCCESS! -- 22 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
 
Ginkgo ran 1 suite in 6m24.072209201s
Test Suite Passed
~/environment/go/src/amazon-vpc-cni-k8s
Deleting cluster  (this may take ~10 mins) ... ok.
TIMELINE: Down processes took 420 seconds.

Using latest Calico released version (Automatically fetching from Calico)

% RUN_CALICO_TEST=true RUN_LATEST_CALICO_VERSION=true ./scripts/run-integration-tests.sh
...
TIMELINE: Current image integration tests took 39 seconds.
Starting Helm installing Tigera operator and running Calico STAR tests
~/go/src/amazon-vpc-cni-k8s/test ~/go/src/amazon-vpc-cni-k8s
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19996    0 19996    0     0  43094      0 --:--:-- --:--:-- --:--:-- 43094
Using Calico version v3.22.1 to test   <-----
Running Suite: Calico with VPC CNI e2e Test Suite
=================================================
Random Seed: 1646377066
Will run 22 of 22 specs

...
STEP: Restore ARM64 Nodes Schedulability
 
Ran 22 of 22 Specs in 373.879 seconds
SUCCESS! -- 22 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
 
Ginkgo ran 1 suite in 6m24.177285169s
Test Suite Passed
~/go/src/amazon-vpc-cni-k8s
Deleting cluster  (this may take ~10 mins) ... ok.
TIMELINE: Down processes took 442 seconds.

Prefix Delegation tests

STEP: Restore ARM64 Nodes Schedulability

Ran 22 of 22 Specs in 378.191 seconds
SUCCESS! -- 22 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 6m24.845380702s
Test Suite Passed
~/go/src/amazon-vpc-cni-k8s
Run Calico tests with Prefix Delegation enabled
daemonset.apps/aws-node env updated
{
    "TerminatingInstances": [
        {
            "CurrentState": {
                "Code": 32,
                "Name": "shutting-down"
            },
            "InstanceId": "i-0341454b54c510205",
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}
Waiting 15 minutes for new nodes being ready

Starting Helm installing Tigera operator and running Calico STAR tests
~/go/src/amazon-vpc-cni-k8s/test ~/go/src/amazon-vpc-cni-k8s
Using Calico version 3.22.0 to test
Running Suite: Calico with VPC CNI e2e Test Suite
=================================================
Random Seed: 1646727202
Will run 22 of 22 specs
...
STEP: Restore ARM64 Nodes Schedulability

Ran 22 of 22 Specs in 370.725 seconds
SUCCESS! -- 22 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 6m18.087117999s
Test Suite Passed

Automation added to e2e:

yes
Will this PR introduce any new dependencies?:

no
Will this break upgrades or downgrades. Has updating a running cluster been tested?:
no

Does this change require updates to the CNI daemonset config files to work?:

no
Does this PR introduce any user-facing change?:

no

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

vikasmb · 2022-03-04T19:05:50Z

scripts/run-integration-tests.sh

+  ginkgo -v e2e/calico -- --cluster-kubeconfig=$KUBECONFIG --cluster-name=$CLUSTER_NAME --aws-region=$AWS_DEFAULT_REGION --aws-vpc-id=$VPC_ID --calico-version=$calico_version
+  popd
+
+  if [[ "$DEPROVISION" == false ]]; then


I think DEPROVISION is set to false only when we want to login to the cluster nodes and debug. As such, automated termination of nodes may not be needed here. Thoughts ?

Are we planning to recycle the cluster? @jayanthvn

I feel we should keep this in case the Calico tests interfere with any tests without awareness in the future.

We don't need to reuse the cluster since this is run only weekly.

vikasmb · 2022-03-04T19:06:52Z

test/framework/controller/installation_manager.go

@@ -24,6 +24,9 @@ import (
 type InstallationManager interface {
 	InstallCNIMetricsHelper(image string, tag string) error
 	UnInstallCNIMetricsHelper() error
+	InstallTigeraOperator(version string) error
+	UninstallTigeraOperator() error
+	//AddAndUpdateRepository(entry repo.Entry)


Can we remove this commented code ?

Thanks for catching this. Will remove it.

jayanthvn · 2022-03-07T18:35:52Z

test/e2e/calico/calico_suite_test.go

+	Expect(err).ToNot(HaveOccurred())
+
+	By("Patching ARM64 node unschedulable")
+	err = updateNodesSchedulability("beta.kubernetes.io/arch", "arm64", true)


Can we instead use "kubernetes.io/arch" ? I know we do tag with beta still but better to use "kubernetes.io/arch".

make sense. I will change this to not use beta.

jayanthvn · 2022-03-07T18:36:31Z

test/e2e/calico/calico_suite_test.go

+	err := f.InstallationManager.InstallTigeraOperator(tigeraVersion)
+	Expect(err).ToNot(HaveOccurred())
+
+	By("Patching ARM64 node unschedulable")


Is there a reason we mark it unschedulable?

Oh is it because star policy is still not supported for ARM instance types? Maybe we can check this - projectcalico/calico#3717 (comment)

Cordon these arm nodes to avoid Calico tests pods deployed to them. For now the provided tests containers are for AMD.

jayanthvn · 2022-03-07T18:37:06Z

test/e2e/calico/calico_suite_test.go

+		Container(uiContainer).
+		Replicas(1).
+		PodLabel("role", "management-ui").
+		NodeSelector("beta.kubernetes.io/arch", "amd64").


Same as above.

jayanthvn · 2022-03-07T18:43:24Z

Can we also add these tests with PD enabled? We can run the test with secondary IP mode, enable PD, ASG can be reduced to 0 and again back to few nodes and then run the test again.

… tests

haouc · 2022-03-08T16:11:07Z

Can we also add these tests with PD enabled? We can run the test with secondary IP mode, enable PD, ASG can be reduced to 0 and again back to few nodes and then run the test again.

PD tests were added.

jayanthvn

lgtm

vikasmb

lgtm

* Adding Tigera operator installation, Stars resource installations and tests * Calico test pods can only work on amd64 nodes. * Need cordon the arm64 nodes to test amd version calico and stars * we should run new image CNI test and then calico tests * Enable metrics for calico test * Fix make format * updates for node label and adding PD tests

haouc requested review from jayanthvn and vikasmb March 4, 2022 17:47

haouc requested a review from a team as a code owner March 4, 2022 17:47

haouc changed the title ~~Calico integration~~ Calico integration Tests Mar 4, 2022

vikasmb reviewed Mar 4, 2022

View reviewed changes

jayanthvn reviewed Mar 7, 2022

View reviewed changes

haouc added 7 commits March 7, 2022 18:51

Adding Tigera operator installation, Stars resource installations and…

4051182

… tests

Calico test pods can only work on amd64 nodes.

933032f

Need cordon the arm64 nodes to test amd version calico and stars

3ca9faa

we should run new image CNI test and then calico tests

f0b36ef

Enable metrics for calico test

3cad380

Fix make format

d743107

updates for node label and adding PD tests

6d3ef43

haouc force-pushed the calico-integration branch from bbb0eaf to 6d3ef43 Compare March 8, 2022 16:06

haouc requested a review from jayanthvn March 8, 2022 16:11

jayanthvn approved these changes Mar 8, 2022

View reviewed changes

vikasmb approved these changes Mar 8, 2022

View reviewed changes

haouc merged commit 1de5da4 into aws:master Mar 8, 2022

haouc deleted the calico-integration branch March 8, 2022 22:00

jayanthvn mentioned this pull request Mar 31, 2022

e2e testcase to automatically test k8s nodeport feature #159

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calico integration Tests #1906

Calico integration Tests #1906

haouc commented Mar 4, 2022 •

edited by jayanthvn

Loading

vikasmb Mar 4, 2022

haouc Mar 4, 2022

haouc Mar 4, 2022

jayanthvn Mar 7, 2022

vikasmb Mar 4, 2022

haouc Mar 4, 2022

jayanthvn Mar 7, 2022

haouc Mar 7, 2022

jayanthvn Mar 7, 2022

jayanthvn Mar 7, 2022 •

edited

Loading

haouc Mar 7, 2022

jayanthvn Mar 7, 2022

jayanthvn commented Mar 7, 2022

haouc commented Mar 8, 2022

jayanthvn left a comment

vikasmb left a comment

Calico integration Tests #1906

Calico integration Tests #1906

Conversation

haouc commented Mar 4, 2022 • edited by jayanthvn Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayanthvn Mar 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayanthvn commented Mar 7, 2022

haouc commented Mar 8, 2022

jayanthvn left a comment

Choose a reason for hiding this comment

vikasmb left a comment

Choose a reason for hiding this comment

haouc commented Mar 4, 2022 •

edited by jayanthvn

Loading

jayanthvn Mar 7, 2022 •

edited

Loading