CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation #1243

DProvinciani · 2021-04-20T21:39:58Z

Related issue
#1242

Description

This pull request includes all the necessary test cases to cover the new Wazuh DB commands and Vulnerability Detector alerting mechanisms created as part of the epic wazuh/wazuh#7749. For details, check the description of the issue #1242 and #1261.

Vulnerability Detector:

test_scan_results: Includes tests that check vulnerabilities insertion and removal in:
- Log file
- Alerts file
- Vulnerabilities inventory database.
test_scan_types: Includes tests for Baseline, Partial and Full scans
test_general_settings: Includes new settings min_full_scan_interval and retry_interval

Wazuh DB:

The set of tests implemented verify the right behavior of all the new Wazuh DB commands implemented as part of the epic CVEs alerts inventory for Vulnerability Detector wazuh#7749.

Dod

test_scan_types

test_general_settings

test_scan_results

test_windows

tests_wazuh_db

Tests

Proven that tests pass when they have to pass.
Proven that tests fail when they have to fail.

Python codebase satisfies PEP-8 style style guide. pycodestyle --max-line-length=120 --show-source --show-pep8 file.py.
Python codebase is documented following the Google Style for Python docstrings.
The test is documented in wazuh-qa/docs.

DProvinciani · 2021-06-24T18:55:16Z

Jenkins execution

https://ci.wazuh.info/job/Test_integration/6716/

juliamagan

It is needed to change the quotes in the yaml and the test following the next format:

x = f"{var}string"
x = 'raw string'
x = "this is my raw 'string' with single quotation marks"

Some of them were requested, but there are more.

tests/integration/test_wazuh_db/data/agent_messages.yaml

tests/integration/test_wazuh_db/test_wazuh_db.py

juliamagan · 2021-09-13T16:52:33Z

After talking with @snaow we have considered that new development or changes into existing tests should follow new repository standards about Wazuh daemons. For now, it is required to use fixtures to start only required daemons, and turn them off when the test is over. That means, that the default status of Wazuh would be off and each test should be responsible for daemons handling.
In case of the need of all Wazuh daemon, the test should disable every module not necessary for the testing in ossec.conf.

The reasons for this change are:

Faster testing
More encapsulated tests

Also, we need to implement the common fixtures for auto-configure local internal options and file monitoring.

pereyra-m · 2021-11-03T01:06:21Z

Validation

After implementing the changes, the proper tests were run.
Jenkins: https://ci.wazuh.info/job/Test_integration/14836/console

Test Executions	Date	By	Status
report1.zip	2021-11-02	Matias	🟢
report2.zip	2021-11-02	Matias	🟢
report3.zip	2021-11-02	Matias	🟢

DProvinciani · 2021-11-11T21:30:30Z

Validation

We run three times the Vulnerability Detector tests in tier 0.

Test Executions	Date	Status
test_vuldet_1.zip	2021-11-11	🟢
test_vuldet_2.zip	2021-11-11	🟢
test_vuldet_3.zip	2021-11-11	🟢

We run three times the Vulnerability Detector tests in tier 1.

Test Executions	Date	Status
vuln_det_tier_1_round1.zip	2021-11-11	🟢
vuln_det_tier_1_round2.zip	2021-11-11	🟢
vuln_det_tier_1_round3.zip	2021-11-11	🟢

Notes:

During the execution of the tests, some random errors were observed due to the known issue tracked here (The database got locked because of simultaneous access). We followed the same approach and added the tests using this access to the list of skipped tests. The following logfile shows this known error.
We also got some failures telling that some alerts were not in the logs. But after reviewing the logs, the alerts were right there. This is a typical case in which the FileMonitor doesn't achieve the log verification before the timeout.

Failure

Event in the logs

ossec_database_locked_test_general_settings_min_full_scan_interval_120m.py.zip

The successful executions were done discarding the last commit in this pull request as evidence that the tests are working as expected besides the known error of the database locked. That commit skips some test cases that present some instability and intermittent failures.

pereyra-m · 2021-11-11T21:39:32Z

Validation

The Wazuh-DB tests were run three times

Test Executions	Date	By	Status
wazuh_db_1.zip	2021-11-11	Matias	🟢
wazuh_db_2.zip	2021-11-11	Matias	🟢
wazuh_db_3.zip	2021-11-11	Matias	🟢

palaciosjeremias · 2021-11-12T13:53:07Z

Validation

Successful execution of the tests in Jenkins (contains skipped tests)
https://ci.wazuh.info/job/Test_integration/15817/
https://ci.wazuh.info/job/Test_integration/15818/

juliamagan

We will need to improve these tests in the future moving the callbacks to common libraries, using common fixtures, and fixing the style of strings or imports. However, it is important to fix test_wazuh_db.py.

juliamagan · 2021-11-12T16:00:22Z

docs/tests/integration/test_vulnerability_detector/index.md

-the different feeds (Redhat, Canonical, Debian, Amazon Linux and NVD), import them, and check if the confirmation message appears
-in the logs.
+- **[test_download_feeds](test_feeds/test_download_feed.md)**: The tests download
+the different feeds (Redhat, Canonical, Debian, and NVD), import them, and check if the confirmation message appears


Amazon Linux was deleted.

Done in commit 0ba0725

juliamagan · 2021-11-12T16:14:13Z

mkdocs.yml

+            - Test baseline scan type: tests/integration/test_vulnerability_detector/test_scan_types/test_baseline_scan_type.md
+            - Test partial scan type: tests/integration/test_vulnerability_detector/test_scan_types/test_partial_scan_type.md


We need to add Test full scan type

Done in commit 0ba0725

tests/integration/test_wazuh_db/test_wazuh_db.py

If we stop wazuh-db, we can't connect to the socket

juliamagan · 2021-11-15T16:28:21Z

15/11/2021

We have removed the skip from the next tests to execute them locally, but they will be merged skipped:

tests/integration/test_vulnerability_detector/test_general_settings/test_general_settings_min_full_scan_interval.py
tests/integration/test_vulnerability_detector/test_scan_results/test_alert_vulnerability_removal.py
tests/integration/test_vulnerability_detector/test_scan_types/test_baseline_scan_type.py
tests/integration/test_vulnerability_detector/test_scan_types/test_partial_scan_type.py

Package

Version	Revision	Link
4.3.0	40301	https://packages-dev.wazuh.com/warehouse/pullrequests/4.3/rpm/var/wazuh-manager-4.3.0-0.commit3799ec1.x86_64.rpm

Testing

tests/integration/test_vulnerability_detector --tier 0

OS	Local	Jenkins	Notes
PS1	🔴	🟡	Failed due to known issue wazuh/wazuh#9309
PS2	🔴	🟡	Failed due to known issue wazuh/wazuh#9309
PS3	🔴	🟡	Failed due to known issue wazuh/wazuh#9309

tests/integration/test_vulnerability_detector --tier 1

OS	Local	Jenkins
PS1	🟢	🟡
PS2	🟢	🟡
PS3	🟢	🟡

tests/integration/test_wazuh_db

OS	Local	Jenkins
PS1	🟢	🟢
PS2	🟢	🟢
PS3	🟢	🟢

🟢: All pass
🟡: Some warnings
🔴: Some errors/fails
🔵: In progress

…ts-inventory

DProvinciani requested a review from BraulioV April 20, 2021 21:42

DProvinciani assigned DProvinciani, palaciosjeremias, MiguelazoDS and pereyra-m Apr 20, 2021

DProvinciani added component/wazuhDB labels Apr 20, 2021

DProvinciani linked an issue Apr 20, 2021 that may be closed by this pull request

CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests #1242

Closed

BraulioV previously approved these changes Apr 27, 2021

View reviewed changes

This was referenced May 17, 2021

Improving wazuh-db integration tests documentation and comments #1346

Closed

Documentation for new tests and minor comment fixes #1347

Merged

spothound mentioned this pull request May 19, 2021

CVEs alerts inventory for Vulnerability Detector - Vulnerability Detector Integration Tests #1261

Closed

DProvinciani dismissed BraulioV’s stale review via 89dd05f May 21, 2021 16:31

jmv74211 removed component/wazuh-db labels Jun 22, 2021

DProvinciani requested a review from jmv74211 June 24, 2021 18:53

juliamagan requested changes Aug 31, 2021

View reviewed changes

juliamagan requested changes Sep 2, 2021

View reviewed changes

tests/integration/test_wazuh_db/test_wazuh_db.py Outdated Show resolved Hide resolved

tests/integration/test_wazuh_db/test_wazuh_db.py Outdated Show resolved Hide resolved

pereyra-m force-pushed the dev-cves-alerts-inventory branch 3 times, most recently from ab1a940 to 62f66b3 Compare November 1, 2021 21:33

MiguelazoDS added 6 commits November 9, 2021 12:00

Revised and implemented new test cases for Wazuh DB commands

8f651b8

Cleaning table at the end of the test cases.

9d2e925

Fix according to review

ac905c5

Added new test case for set triaged command

371f985

Changes according to review

591183b

Wazuh-db socket timeout validation

b3cdbc0

DProvinciani changed the title ~~CVEs alerts inventory for Vulnerability Detector - Wazuh DB Integration Tests implementation~~ CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation Nov 9, 2021

DProvinciani and others added 3 commits November 9, 2021 17:32

Fixing error in expected output of Wazuh DB integration test

08dc7f7

Restoring tests to skipped state

8d07c54

Setting a low retry_interval for scan_types tests

e093ae5

pereyra-m force-pushed the dev-cves-alerts-inventory branch from 46c2a25 to e093ae5 Compare November 10, 2021 15:19

Skip flaky tests using direct VulDet queries

d1966af

juliamagan requested changes Nov 12, 2021

View reviewed changes

juliamagan added 3 commits November 15, 2021 16:19

fix: Remove stop wazuh-db

5cf5fe5

If we stop wazuh-db, we can't connect to the socket

fix: Change mock_agent to mock_agent_module

f55d8e4

fix: Fix documentation

0ba0725

juliamagan mentioned this pull request Nov 15, 2021

Wazuh DB chunks test is failing due to socket size #2217

Closed

Skip test_wazuh_db_chunks due to #2217

55fdb68

This was referenced Nov 17, 2021

Fix syscollector deltas IT after CVEs alerts inventory feature #2228

Merged

Syscollector deltas IT #2146

Merged

This was referenced Dec 23, 2021

WazuhDB IT: Fix for 4.3 #2399

Closed

WazuhDB IT: Fix for 4.3 #2400

Merged

fedepacher mentioned this pull request Jan 17, 2022

Research on the new changes introduced in vulnerability detector for 4.3 #2448

Closed

jmv74211 mentioned this pull request Jan 18, 2022

Health check of the vulnerability detector tests after the new development added in 4.3 #2451

Closed

2 tasks

Merge branch 'master' of github.com:wazuh/wazuh-qa into dev-cves-aler…

5cf18aa

…ts-inventory

jmv74211 force-pushed the dev-cves-alerts-inventory branch from 50aae5a to 5cf18aa Compare January 18, 2022 10:33

jmv74211 force-pushed the dev-cves-alerts-inventory branch from 8f5fbbe to 5cf18aa Compare January 26, 2022 15:34

jmv74211 added the subteam/qa-thunder label Jan 26, 2022

jmv74211 closed this Jan 26, 2022

jmv74211 reopened this Jan 26, 2022

snaow merged commit e07e04e into master Jan 27, 2022

snaow deleted the dev-cves-alerts-inventory branch January 27, 2022 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation #1243

CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation #1243

DProvinciani commented Apr 20, 2021 •

edited

Loading

DProvinciani commented Jun 24, 2021

juliamagan left a comment

juliamagan commented Sep 13, 2021

pereyra-m commented Nov 3, 2021

DProvinciani commented Nov 11, 2021 •

edited by MiguelazoDS

Loading

pereyra-m commented Nov 11, 2021

palaciosjeremias commented Nov 12, 2021

juliamagan left a comment

juliamagan Nov 12, 2021

juliamagan Nov 15, 2021

juliamagan Nov 12, 2021

juliamagan Nov 15, 2021

juliamagan commented Nov 15, 2021 •

edited

Loading

		- Test baseline scan type: tests/integration/test_vulnerability_detector/test_scan_types/test_baseline_scan_type.md
		- Test partial scan type: tests/integration/test_vulnerability_detector/test_scan_types/test_partial_scan_type.md

CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation #1243

CVEs alerts inventory for Vulnerability Detector - VDT and WDB Integration Tests implementation #1243

Conversation

DProvinciani commented Apr 20, 2021 • edited Loading

Description

Dod

Tests

DProvinciani commented Jun 24, 2021

Jenkins execution

juliamagan left a comment

Choose a reason for hiding this comment

juliamagan commented Sep 13, 2021

pereyra-m commented Nov 3, 2021

Validation

DProvinciani commented Nov 11, 2021 • edited by MiguelazoDS Loading

Validation

pereyra-m commented Nov 11, 2021

Validation

palaciosjeremias commented Nov 12, 2021

Validation

juliamagan left a comment

Choose a reason for hiding this comment

juliamagan Nov 12, 2021

Choose a reason for hiding this comment

juliamagan Nov 15, 2021

Choose a reason for hiding this comment

juliamagan Nov 12, 2021

Choose a reason for hiding this comment

juliamagan Nov 15, 2021

Choose a reason for hiding this comment

juliamagan commented Nov 15, 2021 • edited Loading

15/11/2021

Package

Testing

tests/integration/test_vulnerability_detector --tier 0

tests/integration/test_vulnerability_detector --tier 1

tests/integration/test_wazuh_db

DProvinciani commented Apr 20, 2021 •

edited

Loading

DProvinciani commented Nov 11, 2021 •

edited by MiguelazoDS

Loading

juliamagan commented Nov 15, 2021 •

edited

Loading