Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update log plugin to support more than one test case runs #437

Merged
merged 15 commits into from
Mar 28, 2023

Conversation

keirazhang
Copy link
Contributor

@keirazhang keirazhang commented Mar 16, 2023

Changes in this fix will print test cases in order in results.log with a case index in front of them, so that it can distinguish different runs for same test case. And test case log folder will have same case index in front of folder name.

If there are less than 10 test cases, case index will start from "1".
If there are 10 or more test cases, case index will starts from "01".

Testbed information:
+------------------------------------------------------------------------------------------+
| Product | Version | Build    | Hostname or IP | Server Model                             |
+------------------------------------------------------------------------------------------+
| vCenter | 8.0.0   | 20519528 | x.x.x.x  |                                          |
+------------------------------------------------------------------------------------------+
| ESXi    | 8.0.0   | 20513097 | x.x.x.x | VMware, Inc. VMware7,1                   |
|         |         |          |                | Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz |
+------------------------------------------------------------------------------------------+


VM information:
+----------------------------------------------------------------------+
| Name                      | test_ubuntu22.10_ova-3                   |
+----------------------------------------------------------------------+
| Guest OS Distribution     | Ubuntu 22.10 x86_64                      |
+----------------------------------------------------------------------+
| IP                        | 10.186.131.184                           |
+----------------------------------------------------------------------+
| GUI Installed             | False                                    |
+----------------------------------------------------------------------+
| CloudInit Version         |                                          |
+----------------------------------------------------------------------+
| Hardware Version          | vmx-20                                   |
+----------------------------------------------------------------------+
| VMTools Version           | 12.3.0 (build-21507661)                  |
+----------------------------------------------------------------------+
| Config Guest Id           | ubuntu64Guest                            |
+----------------------------------------------------------------------+
| GuestInfo Guest Id        | ubuntu64Guest                            |
+----------------------------------------------------------------------+
| GuestInfo Guest Full Name | Ubuntu Linux (64-bit)                    |
+----------------------------------------------------------------------+
| GuestInfo Guest Family    | linuxGuest                               |
+----------------------------------------------------------------------+
| GuestInfo Detailed Data   | architecture='X86'                       |
|                           | bitness='64'                             |
|                           | distroAddlVersion='22.10 (Kinetic Kudu)' |
|                           | distroName='Ubuntu'                      |
|                           | distroVersion='22.10'                    |
|                           | familyName='Linux'                       |
|                           | kernelVersion='5.19.0-21-generic'        |
|                           | prettyName='Ubuntu 22.10'                |
+----------------------------------------------------------------------+


Test Results (Total: 4, Passed: 4, Elapsed Time: 00:30:48)
+----------------------------------------------+
| ID | Name               | Status | Exec Time |
+----------------------------------------------+
|  1 | deploy_vm_ova      | Passed | 00:07:11  |
|  2 | check_os_fullname  | Passed | 00:01:12  |
|  3 | ovt_verify_install | Passed | 00:20:57  |
|  4 | check_os_fullname  | Passed | 00:01:10  |
+----------------------------------------------+


VM information:
+------------------------------------------------------------------------------------------+
| Name                      | test_windows_server_ltsc_vnext                               |
+------------------------------------------------------------------------------------------+
| Guest OS Distribution     | Microsoft Windows Server 2022 Datacenter 10.0.25314.0 64-bit |
+------------------------------------------------------------------------------------------+
| IP                        | x.x.x.x                                      |
+------------------------------------------------------------------------------------------+
| Hardware Version          | vmx-20                                                       |
+------------------------------------------------------------------------------------------+
| VMTools Version           | x.x.x                              |
+------------------------------------------------------------------------------------------+
| Config Guest Id           | windows2019srvNext_64Guest                                   |
+------------------------------------------------------------------------------------------+
| GuestInfo Guest Id        | windows2022srvNext_64Guest                                   |
+------------------------------------------------------------------------------------------+
| GuestInfo Guest Full Name | Microsoft Windows Server 2025 (64-bit)                       |
+------------------------------------------------------------------------------------------+
| GuestInfo Guest Family    | windowsGuest                                                 |
+------------------------------------------------------------------------------------------+
| GuestInfo Detailed Data   | architecture='X86'                                           |
|                           | bitness='64'                                                 |
|                           | buildNumber='25314'                                          |
|                           | distroVersion='Windows'                                      |
|                           | distroAddlVersion='10.0'                                     |
|                           | familyName='Windows'                                         |
|                           | kernelVersion='25314.1000'                                   |
|                           | prettyName='Windows Server 2025, 64-bit (Build 25314.1000)'  |
+------------------------------------------------------------------------------------------+


Test Results (Total: 27, Passed: 27, Elapsed Time: 02:23:50)
+------------------------------------------------------------+
| ID | Name                             | Status | Exec Time |
+------------------------------------------------------------+
| 01 | deploy_vm_efi_nvme_e1000e        | Passed | 00:13:52  |
| 02 | check_inbox_driver               | Passed | 00:03:30  |
| 03 | secureboot_enable_disable        | Passed | 00:05:43  |
| 04 | wintools_complete_install_verify | Passed | 00:10:47  |
| 05 | check_efi_firmware               | Passed | 00:00:58  |
| 06 | check_ip_address                 | Passed | 00:01:07  |
| 07 | check_os_fullname                | Passed | 00:00:49  |
| 08 | mouse_driver_vmtools             | Passed | 00:00:40  |
| 09 | vgauth_check_service             | Passed | 00:00:49  |
| 10 | stat_balloon                     | Passed | 00:00:50  |
| 11 | stat_hosttime                    | Passed | 00:00:58  |
| 12 | paravirtual_vhba_device_ops      | Passed | 00:05:25  |
| 13 | lsilogicsas_vhba_device_ops      | Passed | 00:06:00  |
| 14 | sata_vhba_device_ops             | Passed | 00:05:36  |
| 15 | nvme_vhba_device_ops             | Passed | 00:07:18  |
| 16 | nvme_vhba_device_ops_spec13      | Passed | 00:07:35  |
| 17 | nvme_disk_hot_extend_spec13      | Passed | 00:05:19  |
| 18 | nvdimm_cold_add_remove           | Passed | 00:09:42  |
| 19 | e1000e_network_device_ops        | Passed | 00:05:45  |
| 20 | vmxnet3_network_device_ops       | Passed | 00:04:58  |
| 21 | memory_hot_add_basic             | Passed | 00:05:15  |
| 22 | check_quiesce_snapshot           | Passed | 00:02:32  |
| 23 | cpu_multicores_per_socket        | Passed | 00:10:24  |
| 24 | gosc_sanity_staticip             | Passed | 00:06:56  |
| 25 | gosc_sanity_dhcp                 | Passed | 00:09:08  |
| 26 | wintools_uninstall_verify        | Passed | 00:06:25  |
| 27 | cpu_hot_add_basic                | Passed | 00:04:00  |
+------------------------------------------------------------+

Test cases log folders will be like

01_deploy_vm
02_check_inbox_driver
07_check_os_fullname
11_check_quiesce_snapshot_custom_script
20_gosc_perl_dhcp
21_gosc_perl_staticip
22_gosc_cloudinit_dhcp
23_gosc_cloudinit_staticip
29_nvdimm_cold_add_remove

The play name in log files will also has the case index:

2023-03-26 15:51:14,026 | Failed at Play [29_nvdimm_cold_add_remove] *****************

2023-03-26 15:51:14,026 | TASK [29_nvdimm_cold_add_remove][Check the new PMem device is recognized by guest OS after cold add] 
task path: /home/worker/workspace/Ansible_Regression_Ubuntu_20.04_Server_OVA/ansible-vsphere-gos-validation/linux/nvdimm_cold_add_remove/cold_add_nvdimm_test.yml:117
fatal: [localhost]: FAILED! => {
    "assertion": "(guest_nvdimm_list_after_add | difference(guest_nvdimm_list_before_add)) | length == 1",
    "changed": false,
    "evaluated_to": false,
    "msg": "Guest PMem device list before cold add is [], after cold add is [], the new added PMem device is not recognized."
}
error message:
Guest PMem device list before cold add is [], after cold add is [], the new added PMem device is not recognized.

Deploy_vm failure would block the rest of test cases:

Test Results (Total: 4, Failed: 1, Blocked: 3, Elapsed Time: 01:16:00)
+---------------------------------------------------------------+
| ID | Name                             |   Status  | Exec Time |
+---------------------------------------------------------------+
|  1 | deploy_vm_efi_lsilogicsas_e1000e | * Failed  | 01:14:59  |
|  2 | wintools_complete_install_verify | * Blocked | 00:00:00  |
|  3 | guest_os_inplace_upgrade         | * Blocked | 00:00:00  |
|  4 | wintools_complete_install_verify | * Blocked | 00:00:00  |
+---------------------------------------------------------------+

@keirazhang keirazhang added this to the v2.3 milestone Mar 16, 2023
@keirazhang keirazhang changed the title Update log plugin to support more than one test case runs [WIP] Update log plugin to support more than one test case runs Mar 16, 2023
@keirazhang keirazhang force-pushed the update_log_plugin branch 6 times, most recently from 5670dcf to 07d6c6a Compare March 20, 2023 11:50
@keirazhang keirazhang changed the title [WIP] Update log plugin to support more than one test case runs Update log plugin to support more than one test case runs Mar 21, 2023
@keirazhang keirazhang force-pushed the update_log_plugin branch 6 times, most recently from 1e687f9 to e07f14c Compare March 28, 2023 00:23
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
Signed-off-by: Qi Zhang <qiz@vmware.com>
common/set_current_testcase_facts.yml Outdated Show resolved Hide resolved
common/set_current_testcase_facts.yml Outdated Show resolved Hide resolved
common/set_current_testcase_facts.yml Outdated Show resolved Hide resolved
Signed-off-by: Qi Zhang <qiz@vmware.com>
Copy link
Member

@Tomorrow9 Tomorrow9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants