[TEST] Rancher v2.9.0 alpha integration test of Harvester charts for K8s v1.29

[TachunLin]

In order to test the K8s v1.29 support for Harvester-cloud-provider and Harvester-csi-driver of the following tickets:

• k8s 1.29 support for harvester-cloud-provider rancher/rancher#45470
• k8s 1.29 support for Harvester-csi-driver rancher/rancher#45471

We need to perform Rancher v2.9.0 integration test with Harvester v1.3.1

Test environment

• Rancher 2.9.0-alpha5
• Harvseter v1.3.1
• K8s: v1.29.5
• Type: RKE2, RKE1

Test plans

• 1. RKE2: Necessary checks for cloud provider and csi driver working on cluster.
• 2. Delete RKE2 cluster
• 3. Execute Rancher integration test backend automation test
• 4. RKE1: Install, uninstall of cloud-provider and csi-driver
• 5. Delete RKE1 cluster
• 6. Create RKE2 guest cluster using terraform,
• 7. Scale down and scale up of guest clusters
• 8. RKE2: upgrade v1.28 to v1.29
• 9. Upgrade Rancher from v2.8.5 to v2.9.0-alpha5 with RKE2 guest cluster
• 10. Create RKE2 guest cluster using custom cluster
• 11. Create RKE2 guest cluster using import cluster

[TachunLin]

Verifying on Rancher v2.9.0-alpha5 with Harvester v1.3.1 for K8s v1.29.5

Result

$\color{green}{\textsf{PASS}}$ RKE2: Necessary checks for cloud provider and csi driver working on cluster. $~~$

• Can provision v1.29.5 RKE2 cluster well

  $\color{green}{\textsf{PASS}}$: RKE2 running well $~~$

  

• Check the default cloud provider version:

  $\color{orange}{\textsf{Not ready}}$: default cloud provider version is 0.2.3 not 0.2.4 $~~$

  

• Check the default csi-driver version:

  $\color{green}{\textsf{PASS}}$: default csi driver version is 0.1.1700 $~~$

  

• Check the default storage class set to Harvester

  $\color{green}{\textsf{PASS}}$: set to Harvester 0.1.1700 $~~$

  

• Check the application created pvc and pv

  $\color{green}{\textsf{PASS}}$: Can create nginx application pvc $~~$

  

  $\color{green}{\textsf{PASS}}$: Can create nginx application pv $~~$

  

  $\color{green}{\textsf{PASS}}$: nginx pvc created one Harvester $~~$

  

• Check to create the dhcp and pool type load balancer for Harvester cloud provider

  $\color{green}{\textsf{PASS}}$: Can create dhcp loadbalancer and route correctly $~~$

  

  $\color{green}{\textsf{PASS}}$: Can create pool loadbalancer and route correctly $~~$

  

$\color{green}{\textsf{PASS}}$ Delete RKE2 guest cluster on Rancher v2.9.0-alpha5 $~~$

• Test automation result

  $\color{green}{\textsf{PASS}}$: Can correctly delete RKE2 v1.29.5 cluster $~~$

  

  

$\color{green}{\textsf{PASS}}$ Execute Rancher integration test backend automation test $~~$

• Test automation result

  $\color{green}{\textsf{PASS}}$: all Rancher integration backend test $~~$

  • PASS all backend automation Rancher integration test result
    
    

  • PASS the following Rancher integration test cases

    • test_import_harvester
    • test_add_project_owner_user
    • TestRKE2::test_create_rke2[1]
    • TestRKE2::test_create_pvc[1]
    • TestRKE2::test_csi_deployment[1]
    • TestRKE2::test_delete_deployment[1]
    • TestRKE2::test_deploy_nginx[1]
    • TestRKE2::test_load_balancer_service[1-dhcp]
    • TestRKE2::test_load_balancer_service[1-pool]
    • TestRKE2::test_delete_rke2[1]
    • TestRKE1::test_create_rke1[1]
    • TestRKE1::test_create_pvc[1]
    • TestRKE1::test_cloud_provider_chart[1]
    • TestRKE1::test_deploy_nginx[1]
    • TestRKE1::test_load_balancer_service[1-dhcp]
    • TestRKE1::test_load_balancer_service[1-pool]
    • TestRKE1::test_csi_driver_chart[1]
    • TestRKE1::test_csi_deployment[1]
    • TestRKE1::test_delete_deployment[1]
    • TestRKE1::test_delete_rke1[1]

$\color{green}{\textsf{PASS}}$ RKE1: Provision, install, uninstall of cloud-provider and csi-driver $~~$

• Provision RKE1 cluster

  $\color{green}{\textsf{PASS}}$: Can correctly provision v1.29.5 RKE1 cluster $~~$

  

• Install cloud-provider and csi-driver

  $\color{green}{\textsf{PASS}}$: Correctly install cloud-provider and csi-driver $~~$

  

• Check cloud-provider version on RKE1 cluster

  $\color{green}{\textsf{PASS}}$: harvester-cloud-provider:104.0.1+up0.2.4 $~~$

  

• Check csi-driver version on RKE1 cluster

  $\color{green}{\textsf{PASS}}$: harvester-csi-driver:104.0.0+up0.1.17 $~~$

  

• Uninstall csi-driver

  $\color{green}{\textsf{PASS}}$: Can correctly uninstall csi-driver $~~$

  

• Uninstall cloud-provider

  $\color{green}{\textsf{PASS}}$: Can correctly uninstall cloud-provider $~~$

  

$\color{green}{\textsf{PASS}}$ Delete RKE1 guest cluster on Rancher v2.9.0-alpha5 $~~$

• Test automation result

  $\color{green}{\textsf{PASS}}$: Can correctly delete RKE1 v1.29.5 cluster $~~$

  
  

$\color{red}{\textsf{FAIL}}$ Create RKE2 guest cluster using terraform $~~$

• Check can provision v1.29.5 RKE2 cluster well (by automation test)

  $\color{red}{\textsf{FAIL}}$: RKE2 cluster stuck in "waiting for cluster agent to connect" $~~$

  
  

• Check can provision v1.29.5 RKE2 cluster well (manual test)

  $\color{red}{\textsf{FAIL}}$: RKE2 cluster stuck in "waiting for cluster agent to connect" $~~$

  
  

$\color{green}{\textsf{PASS}}$ Scale down and scale up of guest clusters $~~$

• Check can scale up the RKE2 cluster from 1 to 2 nodes

  $\color{green}{\textsf{PASS}}$: Can scale up the RKE2 cluster from 1 to 2 nodes $~~$

  

• Check can scale up the RKE2 cluster from 2 to 3 nodes

  $\color{green}{\textsf{PASS}}$: Can scale up the RKE2 cluster from 2 to 3 nodes $~~$

  

• Check can scale down the RKE2 cluster from 3 to 2 nodes

  $\color{green}{\textsf{PASS}}$: Can scale up the RKE2 cluster from 2 to 3 nodes $~~$

  

  

  

• Check can scale down the RKE2 cluster from 2 to 1 nodes

  $\color{green}{\textsf{PASS}}$: Can scale up the RKE2 cluster from 2 to 3 nodes $~~$

  

  
  

$\color{green}{\textsf{PASS}}$ RKE2: upgrade v1.28.10 to v1.29.5 $~~$

• Check can upgrade RKE2 v1.28.10 cluster to v1.29.5

  $\color{green}{\textsf{PASS}}$: Can correctly upgrade RKE2 cluster from v1.28.10 to v1.29.5 $~~$

  

$\color{green}{\textsf{PASS}}$ RKE2: Upgrade Rancher from v2.8.5 -> v2.9.0-alpha5 with RKE2 guest cluster $~~$

• Check upgrade Rancher from v2.8.5 to v2.9.0-alpha5

  $\color{green}{\textsf{PASS}}$: Can upgrade Rancher from v2.8.5 to v2.9.0-alpha5 $~~$

  

$\color{red}{\textsf{FAIL}}$ Create v1.29.5 RKE2 guest cluster using import cluster $~~$

• Create v1.29.5 RKE2 custom cluster

  $\color{green}{\textsf{PASS}}$: Can import v1.29.5 RKE2 guest cluster in running state $~~$

  

• Check can install cloud provider from chart store

  $\color{green}{\textsf{PASS}}$: Can install the latest cloud provider 0.2.4 $~~$

  

  

• Check the bundled csi driver version

  $\color{green}{\textsf{PASS}}$: Can install the latest csi driver 0.1.17 $~~$

  

  

• Check the basic functionality of csi driver

  $\color{green}{\textsf{PASS}}$: Harvester storage class, pvc and pv works well on application deployment $~~$

  

  

  

  

• Check the basic functionality of cloud provider

  $\color{red}{\textsf{FAIL}}$: The dhcp load balancer stuck in Pending state $~~$

  

  The harvester-cloud-provider keep in updating
  

  With the Crashloopbackoff failure
  

$\color{green}{\textsf{PASS}}$ Create v1.29.5 RKE2 guest cluster using custom cluster $~~$

• Create v1.29.5 RKE2 custom cluster

  $\color{green}{\textsf{PASS}}$: Can create v1.29.5 RKE2 custom cluster in running state $~~$

  

• Check the bundled cloud provider version

  $\color{green}{\textsf{PASS}}$: The bundled cloud provider version for RKE2 v1.29.5 is 0.2.300 $~~$

  

• Check the bundled csi driver version

  $\color{green}{\textsf{PASS}}$: The bundled csi driver version for RKE2 v1.29.5 is 0.1.1700 $~~$

  

• Check the basic functionality of csi driver

  $\color{green}{\textsf{PASS}}$: Harvester storage class, pvc and pv works well on application deployment $~~$

  

  

  

  

  

• Check the basic functionality of cloud provider

  $\color{green}{\textsf{PASS}}$: Check the load balancer service can running well $~~$

  

Test Information

• Test Environment: Single nodes harvester on baremetal machine
• Harvester version: v1.3.1
• Rancher version: v2.9.0-alpha5
• K8s version: v1.29.5

Verify Steps

Necessary checks for cloud provider and csi driver

[TachunLin]

After complete the full Rancher integration on Rancher v2.9.0-alpha5 with Harvester v1.3.1

Most of the provision of RKE2, RKE1 cluster, bundle chart of Harvester cloud-provider and csi-driver are working as expected.

As the above test result stated:
#1337 (comment)

We can pass the following test scenarios

• RKE2: Necessary checks for cloud provider and csi driver working on cluster.
• Delete RKE2 guest cluster on Rancher v2.9.0-alpha5
• Execute Rancher integration test backend automation test
• RKE1: Provision, install, uninstall of cloud-provider and csi-driver
• Delete RKE1 guest cluster on Rancher v2.9.0-alpha5
• Scale down and scale up of guest clusters
• RKE2: upgrade RKE2 from v1.28.10 to v1.29.5
• RKE2: Upgrade Rancher from v2.8.5 -> v2.9.0-alpha5 with RKE2 guest cluster

While we found the harvester-cloud-provider chart bundled with RKE2 v1.29.5+rke2r1 is not yet updated to the latest version v0.2.4

According to the investigation result in comment rancher/rancher#45470 (comment), we need to wait next release for v1.29.6+rke2r1 to have v0.2.4 cloud provider bundled on it.

And we encounter the following failure while executing the following cases

1. Create RKE2 guest cluster using terraform-provider-rancher2

• Use both automation test or manually execution, both the RKE2 cluster stuck in "waiting for cluster agent to connect"
• We have created following issue to track the failure
  • [BUG] Unable to use terraform-provider-rancher2 to provision RKE2 cluster well on Rancher v2.9.0-alpha5 harvester#6098

2. Create v1.29.5 RKE2 guest cluster using import cluster

• When we have RKE2 cluster imported to Rancher and try to install csi-driver and cloud-provider, the cloud-provider can't fully deployed which cause failed to create the load balancer service
• We have create following issue to track the failure
  • [BUG] Unable to install cloud-provider in fully running with csi-driver on the imported cluster in Rancher v2.9.0-alpha5  harvester#6099

[TachunLin]

We also encounter issue when access the virtualization management in Rancher

Access Harvester in Rancher virtualization management, most of the UI text are displayed with the incorrect %variable-reference%

We also created the following issue for a track:
rancher/dashboard#11349
rancher/dashboard#11350

[TachunLin]

Close this since we have finished all task activities.