🌱 E2E Upgrade test: Add more BMO versions to upgrade from #1523

mquhuy · 2024-01-19T09:01:40Z

This PR adds more BMO config versions to the E2E upgrade tests, so that we can test from either of release-0.3, release-0.4 and release-0.5 to main (with both ironic and fixture config).
It also changes the default "from" version from release-0.4 to release-0.5.

e.g. To run upgrade test from release-0.3 to main with ironic config:

export GINKGO_FOCUS="upgrade"
export UPGRADE_BMO_KUSTOMIZATION_FROM=config/overlays/e2e-release-0.3
./hack/ci-e2e.sh

Edit: The upgrade-from version can now be seen from the test output:

 STEP: Installing BMO from ../../config/overlays/e2e-release-0.5 on the upgrade cluster

And from the logs

test/e2e/_artifacts/logs/baremetal-operator-system-upgrade/
├── bmo-e2e
│   └── baremetal-operator-controller-manager
│       ├── baremetal-operator-controller-manager-696c96dbdc-ng2qh
│       │   ├── kube-rbac-proxy-log-metadata.json
│       │   ├── kube-rbac-proxy.log
│       │   ├── manager-log-metadata.json
│       │   └── manager.log
│       └── baremetal-operator-controller-manager-7dd8b7c4b4-9mlhg
│           ├── kube-rbac-proxy-log-metadata.json
│           ├── kube-rbac-proxy.log
│           ├── manager-log-metadata.json
│           └── manager.log
├── bmo-e2e-release-0.5
│   └── baremetal-operator-controller-manager
│       ├── baremetal-operator-controller-manager-696c96dbdc-ng2qh
│       │   ├── kube-rbac-proxy-log-metadata.json
│       │   ├── kube-rbac-proxy.log
│       │   ├── manager-log-metadata.json
│       │   └── manager.log
│       └── baremetal-operator-controller-manager-7dd8b7c4b4-9mlhg
│           ├── kube-rbac-proxy-log-metadata.json
│           ├── kube-rbac-proxy.log
│           ├── manager-log-metadata.json
│           └── manager.log
└── ironic
    └── ironic-754996654b-ftkxq
        ├── ironic-httpd-log-metadata.json
        ├── ironic-httpd.log
        ├── ironic-inspector-log-metadata.json
        ├── ironic-inspector.log
        ├── ironic-log-metadata.json
        ├── ironic-log-watch-log-metadata.json
        ├── ironic-log-watch.log
        └── ironic.log

mquhuy · 2024-01-19T09:19:36Z

/metal3-bmo-e2e-test

mquhuy · 2024-01-19T09:45:42Z

/metal3-bmo-e2e-optional-test

lentzi90

This looks good already, but I have one request: would it be possible to make it visible in the logs what version we test?
Maybe just printing the exact overlay that is used? We could also print the image tag or something like that.

mquhuy · 2024-01-31T11:50:12Z

This looks good already, but I have one request: would it be possible to make it visible in the logs what version we test? Maybe just printing the exact overlay that is used? We could also print the image tag or something like that.

Thanks Lennart. Sorry it took a bit long, I almost forgot about it.
I've updated the PR with reference to the version in logs. Details in PR description. Please let me know if something else is still missing xD

mquhuy · 2024-01-31T11:53:08Z

/test metal3-bmo-e2e-optional-test

metal3-io-bot · 2024-01-31T11:53:15Z

@mquhuy: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test generate
/test gofmt
/test golint
/test gomod
/test gosec
/test manifestlint
/test markdownlint
/test metal3-bmo-e2e-test-pull
/test shellcheck
/test unit

The following commands are available to trigger optional jobs:

/test metal3-bmo-e2e-test-optional-pull

Use /test all to run the following jobs that were automatically triggered:

generate
gofmt
golint
gomod
gosec
manifestlint
shellcheck
unit

In response to this:

/test metal3-bmo-e2e-optional-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mquhuy · 2024-01-31T11:57:36Z

/test metal3-bmo-e2e-test-optional-pull

config/overlays/e2e-release-0.3/kustomization.yaml

lentzi90 · 2024-02-02T09:43:59Z

test/e2e/upgrade_test.go

+			bmoKustomization := e2eConfig.GetVariable("UPGRADE_BMO_KUSTOMIZATION_FROM")
+			// Get the index of the last "release" word, which typically indicates the start of the BMO version name
+			lastReleaseWordIndex := strings.LastIndex(bmoKustomization, "release")
+			BmoFromVer := bmoKustomization[lastReleaseWordIndex:]
+			By(fmt.Sprintf("Installing BMO with version: %s on the upgrade cluster", BmoFromVer))


This basically adds a requirement on the kustomization folder name. I would rather have something simpler that doesn't care what you name them. Perhaps just print the path?

Suggested change

bmoKustomization := e2eConfig.GetVariable("UPGRADE_BMO_KUSTOMIZATION_FROM")

// Get the index of the last "release" word, which typically indicates the start of the BMO version name

lastReleaseWordIndex := strings.LastIndex(bmoKustomization, "release")

BmoFromVer := bmoKustomization[lastReleaseWordIndex:]

By(fmt.Sprintf("Installing BMO with version: %s on the upgrade cluster", BmoFromVer))

bmoKustomization := e2eConfig.GetVariable("UPGRADE_BMO_KUSTOMIZATION_FROM")

By(fmt.Sprintf("Installing BMO from %s on the upgrade cluster", bmoKustomization))

Hmm. Okay. How about using the Base path instead? Having a full path in the log is fine, but I also want to have a nice way to notice it in the artifacts.

Oh I didn't notice it was used for the artifacts. Thanks for pointing it out!
I think the base path sounds good for artifacts, in the log I would say the more we include the better :D

Sounds good. I'll update it real quick

Signed-off-by: Huy Mai <huy.mai@est.tech>

tuminoid · 2024-02-06T06:07:57Z

/test metal3-bmo-e2e-test-pull
/test-centos-e2e-integration-main

lentzi90 · 2024-02-06T07:48:46Z

/test metal3-bmo-e2e-test-optional-pull

lentzi90

/lgtm
Thanks for this!
I think that it would be nice eventually to refactor it so that we can run the upgrade test for all different versions, one after the other. But let's do that in a follow up.
The reason I think it would be nice is to reduce the number of jobs so it is easier to keep track of them and also to understand what we test.

mquhuy · 2024-02-07T06:43:46Z

/lgtm Thanks for this! I think that it would be nice eventually to refactor it so that we can run the upgrade test for all different versions, one after the other. But let's do that in a follow up. The reason I think it would be nice is to reduce the number of jobs so it is easier to keep track of them and also to understand what we test.

Thank you. Yes, I was thinking about that idea the other day. Having a separate job for every upgrade test is too much to handle, but the "problem" with running them in sequence is that to run any upgrade test (including ironic upgrade), the cluster should be cleaned up to its initial state (no ironic nor bmo), so we need to test if cleaning up a cluster is reliable (i.e. if the cluster is really "clean" afterwards). I guess it needs more investigating, and I might start doing it together with the ironic upgrade test.

mquhuy · 2024-02-07T06:44:06Z

/cc @kashifest

mquhuy · 2024-02-08T07:43:18Z

/cc @tuminoid @Rozzii

tuminoid

Does the inspector config here work with the #1544 ?

dtantsur · 2024-02-08T10:13:32Z

config/overlays/e2e-release-0.3/ironic.env

+CACHEURL=http://192.168.222.199/images
+IRONIC_FAST_TRACK=true
+IRONIC_KERNEL_PARAMS=console=ttyS0
+IRONIC_INSPECTOR_VLAN_INTERFACES=all


Will this consume ironic-image from main? If so, you need to set USE_IRONIC_INSPECTOR=true.

Yes, we use the same ironic image for all jobs currently and that is the latest. Please add it @mquhuy

Wait a second, this is for BMO. In #1544 you did not touch those, do we need to?

BMO: https://github.com/metal3-io/baremetal-operator/blob/main/config/overlays/e2e/ironic.env

Ironic: https://github.com/metal3-io/baremetal-operator/blob/main/ironic-deployment/overlays/e2e/ironic_bmo_configmap.env

Does BMO need to have the variable or is it only relevant for Ironic?

BMO requires Inspector before 0.5.0. Does it answer your question?

dtantsur · 2024-02-08T10:13:51Z

config/overlays/e2e-release-0.5/ironic.env

+CACHEURL=http://192.168.222.199/images
+IRONIC_FAST_TRACK=true
+IRONIC_KERNEL_PARAMS=console=ttyS0
+IRONIC_INSPECTOR_VLAN_INTERFACES=all


same re INSPECTOR

mquhuy · 2024-02-08T12:48:06Z

@tuminoid I cannot say for sure, but if inspector removal doesn't work with these kustomization, then it will fail on e2e as well. I think we can merge this PR, and fix everything later when/if it fails.
@dtantsur Thanks. May I ask does it mean BMO cannot work without ironic inspector, or it means the ironic image from main needs to adapt?

kashifest · 2024-02-12T11:53:35Z

/approve

metal3-io-bot · 2024-02-12T11:53:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kashifest

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kashifest]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 19, 2024

lentzi90 reviewed Jan 19, 2024

View reviewed changes

mquhuy mentioned this pull request Jan 23, 2024

🌱 Enable parallel testing in E2E #1527

Merged

mquhuy force-pushed the mquhuy/add-e2e-upgrade-versions branch from 4f3d2fb to 16e4c8d Compare January 23, 2024 07:57

mquhuy force-pushed the mquhuy/add-e2e-upgrade-versions branch 2 times, most recently from b426eec to cd03d66 Compare January 31, 2024 11:48

tuminoid reviewed Feb 2, 2024

View reviewed changes

config/overlays/e2e-release-0.3/kustomization.yaml Outdated Show resolved Hide resolved

lentzi90 reviewed Feb 2, 2024

View reviewed changes

mquhuy requested review from lentzi90 and tuminoid February 5, 2024 07:38

mquhuy force-pushed the mquhuy/add-e2e-upgrade-versions branch 2 times, most recently from 14b9aad to f2aed3e Compare February 5, 2024 12:21

E2E test: Add more BMO versions for upgrade tests

4c94b2b

Signed-off-by: Huy Mai <huy.mai@est.tech>

mquhuy force-pushed the mquhuy/add-e2e-upgrade-versions branch from f2aed3e to 4c94b2b Compare February 5, 2024 14:28

lentzi90 reviewed Feb 6, 2024

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Feb 6, 2024

metal3-io-bot requested a review from kashifest February 7, 2024 06:44

metal3-io-bot requested a review from Rozzii February 8, 2024 07:43

tuminoid reviewed Feb 8, 2024

View reviewed changes

dtantsur reviewed Feb 8, 2024

View reviewed changes

metal3-io-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 12, 2024

metal3-io-bot merged commit 52b3c0c into metal3-io:main Feb 12, 2024
21 checks passed

metal3-io-bot deleted the mquhuy/add-e2e-upgrade-versions branch February 12, 2024 11:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 E2E Upgrade test: Add more BMO versions to upgrade from #1523

🌱 E2E Upgrade test: Add more BMO versions to upgrade from #1523

mquhuy commented Jan 19, 2024 •

edited

Loading

mquhuy commented Jan 19, 2024

mquhuy commented Jan 19, 2024

lentzi90 left a comment

mquhuy commented Jan 31, 2024 •

edited

Loading

mquhuy commented Jan 31, 2024

metal3-io-bot commented Jan 31, 2024

mquhuy commented Jan 31, 2024

lentzi90 Feb 2, 2024

mquhuy Feb 3, 2024

lentzi90 Feb 5, 2024

mquhuy Feb 5, 2024

tuminoid commented Feb 6, 2024

lentzi90 commented Feb 6, 2024

lentzi90 left a comment

mquhuy commented Feb 7, 2024

mquhuy commented Feb 7, 2024

mquhuy commented Feb 8, 2024

tuminoid left a comment

dtantsur Feb 8, 2024

lentzi90 Feb 9, 2024

lentzi90 Feb 9, 2024

dtantsur Feb 20, 2024

dtantsur Feb 8, 2024

mquhuy commented Feb 8, 2024

kashifest commented Feb 12, 2024

metal3-io-bot commented Feb 12, 2024

🌱 E2E Upgrade test: Add more BMO versions to upgrade from #1523

🌱 E2E Upgrade test: Add more BMO versions to upgrade from #1523

Conversation

mquhuy commented Jan 19, 2024 • edited Loading

mquhuy commented Jan 19, 2024

mquhuy commented Jan 19, 2024

lentzi90 left a comment

Choose a reason for hiding this comment

mquhuy commented Jan 31, 2024 • edited Loading

mquhuy commented Jan 31, 2024

metal3-io-bot commented Jan 31, 2024

mquhuy commented Jan 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tuminoid commented Feb 6, 2024

lentzi90 commented Feb 6, 2024

lentzi90 left a comment

Choose a reason for hiding this comment

mquhuy commented Feb 7, 2024

mquhuy commented Feb 7, 2024

mquhuy commented Feb 8, 2024

tuminoid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mquhuy commented Feb 8, 2024

kashifest commented Feb 12, 2024

metal3-io-bot commented Feb 12, 2024

mquhuy commented Jan 19, 2024 •

edited

Loading

mquhuy commented Jan 31, 2024 •

edited

Loading