Health check of the vulnerability detector tests after the new development added in 4.3

[jmv74211]

After the merge of the following developments wazuh/wazuh#7749 and wazuh/wazuh#8727 corresponding to the vulnerability detector module for 4.3, it was necessary to update the integration tests.

These tests were updated in the following PR: #1243

The following is requested:

Tasks

• Update the dev-cves-alerts-inventory branch with all changes from the master branch. Conflicts should be fixed.
• Test each vulnerability detector integration test separately (5 rounds) and report the results obtained.

The final objective is to evaluate the general state of the vulnerability detector tests applying the new changes introduced in 4.3.

Note: The environment to be used for testing can be deployed using the qa-ctl tool. To speed up the process, it is recommended to deploy 5-8 Wazuh environments in parallel and launch each of the test rounds.

[jmv74211]

Testing report

Testing parameters

QA branch	Wazuh version	Package URL
dev-cves-alerts-inventory	v4.3.0-rc2	wazuh-manager-4.3.0-qa.2451.vuln.detect.x86_64

Testing status

Path	Test	R1	R2	R3	R4	R5	Status
test_general_settings
	test_general_settings_enabled.py	🟢	🟢	🟢	🟢	🟢	2p 2s 5w
	test_general_settings_min_full_scan_interval.py	🔵	🔵	🔵	🔵	🔵	4s
	test_general_settings_run_on_start.py	🟢	🟢	🟢	🟢	🟢	2p
	test_general_settings_interval.py	🟢	🟢	🟢	🟢	🟢	12p
	test_general_settings_retry_interval.py	🟢	🟢	🟢	🟢	🟢	3p
test_providers
	test_providers_enabled.py	🔵	🔵	🔵	🔵	🔵	16s
	test_providers_multiple_providers.py	🔴	🔴	🔴	🔴	🔴	2f 12s
	test_providers_no_os.py	🔵	🔵	🔵	🔵	🔵	7s
	test_providers_os.py	🔵	🔵	🔵	🔵	🔵	13s
	test_providers_update_from_year.py	🔵	🔵	🔵	🔵	🔵	14s
	test_providers_update_interval.py	🟢	🟢	🟢	🟢	🟢	28p
test_scan_results
	test_alas_inventory_alas_feed.py	🟢	🟢	🟢	🟢	🟢	2p
	test_alert_vulnerability_removal.py	🔵	🔵	🔵	🔵	🔵	1s
	test_archlinux_inventory_archlinux_feed.py	🔴	🔴	🔴	🔴	🔴	2f
	test_debian_inventory_debian_feed.py	🔵	🔵	🔵	🔵	🔵	2s
	test_macos_inventory.py	🔵	🔵	🔵	🔵	🔵	2s
	test_msu_inventory_msu_feed.py	🔵	🔵	🔵	🔵	🔵	3s
	test_redhat_duplicate_vulns.py	🔵	🔵	🔵	🔵	🔵	3s
	test_redhat_inventory_redhat_feed.py	🔵	🔵	🔵	🔵	🔵	4s
	test_scan_different_cves.py	🔵	🔵	🔵	🔵	🔵	12s
	test_scan_nvd_feed.py	🔵	🔵	🔵	🔵	🔵	16s
	test_scan_providers_and_nvd_feed.py	🔵	🔵	🔵	🔵	🔵	12s
	test_ubuntu_inventory_canonical_feed.py	🔵	🔵	🔵	🔵	🔵	4s
test_scan_types
	test_baseline_scan_type.py	🔵	🔵	🔵	🔵	🔵	1s
	test_full_scan_type.py	🟢	🟢	🟢	🟢	🟢	1p
	test_partial_scan_type.py	🔵	🔵	🔵	🔵	🔵	1s
test_windows
	test_cpe_indexing.py	🔴	🔴	🔴	🔴	🔴	29 e
test_feeds
	test_download_feeds.py	🔵	🔵	🔵	🔵	🔵	16s
	test_invalid_type_custom_feeds.py	🔵	🔵	🔵	🔵	🔵	118s
	test_invalid_type_url_feeds.py	🔵	🔵	🔵	🔵	🔵	118s
	test_validate_feed_content.py	🟢	🟢	🟢	🟢	🟢	38p
test_feeds/alas
	test_extra_tags_alas_feed.py	🟢	🟢	🟢	🟢	🟢	244p
	test_invalid_syntax_alas_feed.py	🔴	🟢	🟢	🟢	🟢	1e 110p
	test_invalid_values_alas_feed.py	🟢	🟢	🟢	🟢	🔴	1e 122p
	test_missing_tags_alas_feed.py	🔴	🟢	🟢	🔴	🟢	1e 14p
test_feeds/archlinux
	test_extra_tags_archlinux_feed.py	🟢	🟢	🟢	🟢	🟢	122p
	test_invalid_syntax_archlinux_feed.py	🟢	🟢	🟢	🟢	🟢	55p
	test_invalid_values_archlinux_feed.py	🟢	🟢	🟢	🟢	🟢	21p
	test_missing_tags_archlinux_feed.py	🟢	🟢	🟢	🟢	🟢	11p
test_feeds/canonical
	test_extra_tags_canonical_feed.py	🔵	🔵	🔵	🔵	🔵	1p 121s
	test_invalid_syntax_canonical_feed.py	🟢	🟢	🟢	🟢	🟢	83p
	test_invalid_values_canonical_feed.py	🟢	🟢	🟢	🟢	🟢	161p 10xf
	test_missing_tags_canonical_feed.py	🟢	🔴	🔴	🔴	🟢	1f 27p 5xf
test_feeds/debian
	test_extra_tags_debian_feed.py	🟢	🟢	🟢	🟢	🟢	122p
	test_invalid_syntax_debian_feed.py	🟢	🟢	🟢	🟢	🟢	83p
	test_invalid_values_debian_feed.py	🟢	🟢	🟢	🟢	🟢	81p
	test_missing_tags_debian_feed.py	🟢	🟢	🟢	🟢	🟢	34p 4xf
test_feeds/redhat
	test_extra_fields_redhat_feed.py	🟢	🟢	🟢	🔴	🔴	1f 121p
	test_invalid_syntax_redhat_feed.py	🟢	🟢	🟢	🟢	🟢	82p
	test_invalid_values_redhat_feed.py	🟢	🟢	🔴	🟢	🟢	80f 80p
	test_missing_fields_redhat_feed.py	🟢	🟢	🟢	🟢	🟢	24p 3xf
test_feeds/msu
	test_extra_fields_msu_feed.py	🟢	🟢	🟢	🟢	🟢	122p
	test_invalid_syntax_msu_feed.py	🟢	🟢	🟢	🟢	🟢	57p
	test_invalid_values_msu_feed.py	🟢	🟢	🟢	🟢	🟢	61p
	test_missing_fields_msu_feed.py	🟢	🟢	🟢	🟢	🟢	7p

🟢	Round passed
🔴	Round failed
🔵	All skipped. Disabled test
⚫	Pending to launch

p	Passed
s	Skipped
w	Warning
e	Error
f	Failed
xf	Espected fail

[jmv74211]

Looking at the results obtained, there are tests that are already disabled because they have been deprecated or malfunctioned. However, there are others that have failed in some executions, causing false positives, or fail always.

Also, it has been concluded that feed tests can be unstable, depending on which tests have been previously run on that machine. Therefore, it has been decided to disable all of them, for future research and refactoring.

The tests that must be disabled in order to have the initial "full green" are shown below.

• test_providers/test_providers_multiple_providers.py
• test_scan_results/test_archlinux_inventory_archlinux_feed.py
• test_windows/test_cpe_indexing.py
• test_feeds/alas/*
• test_feeds/archlinux/*
• test_feeds/canonical/*
• test_feeds/debian/*
• test_feeds/redhat/*
• test_feeds/msu/*