Umbrella integration tests

[nico-stefani]

Related issue
#3347

Description

This PR adds tests for AWS Umbrella integration

Added

Tier 0

• Default configuration tests
• remove_from_bucket parameter tests
• only_logs_after parameter tests
• path parameter tests
• discard_regex parameter tests

Tier 1

• only_logs_after parameter tests

---

Testing performed

Tester	Test path	Jenkins	Local	OS	Commit	Notes
@nico-stefani (Developer)	tests/integration/test_aws	⚫⚫⚫	🟢 🔴	Ubuntu 22.04		1
@Reviewer (Reviewer)		⚫⚫⚫				Nothing to highlight

Tier 0 results

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected

test_basic.py .                                                          [ 12%]
test_discard_regex.py .                                                  [ 25%]
test_only_logs_after.py ..                                               [ 50%]
test_path.py ...                                                         [ 87%]
test_remove_from_bucket.py .                                             [100%]

=============================== warnings summary ===============================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========== 8 passed, 16 deselected, 2 warnings in 174.76s (0:02:54) ===========

Tier 1 results

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini
plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0
collected 24 items / 23 deselected / 1 selected

test_only_logs_after.py F                                                [100%]

=================================== FAILURES ===================================
______ test_multiple_calls[cisco_umbrella_only_logs_after_multiple_calls] ______
------------------------------ Captured log call -------------------------------
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-umbrella-integration-tests', '--type', 'cisco_umbrella', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--trail_prefix', 'dnslogs']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Table does not exist; create
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: dnslogs/2023-01-20
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ No logs to process in bucket: None/None
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-umbrella-integration-tests', '--type', 'cisco_umbrella', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--trail_prefix', 'dnslogs', '--only_logs_after', '2022-NOV-20']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: dnslogs/2022-11-20
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: dnslogs/2022-11-21/2022-11-21-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: dnslogs/2022-11-23/2022-11-23-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Found new log: dnslogs/2022-11-26/2022-11-26-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
DEBUG    wazuh_testing:cli_utils.py:23 Calling AWS module with: '[PosixPath('/var/ossec/wodles/aws/aws-s3'), '--bucket', 'wazuh-umbrella-integration-tests', '--type', 'cisco_umbrella', '--regions', 'us-east-1', '--aws_profile', 'qa', '--debug', '2', '--trail_prefix', 'dnslogs', '--only_logs_after', '2022-NOV-20']'
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Debug mode on - Level: 2
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ Marker: dnslogs/2022-11-20
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: dnslogs/2022-11-21/2022-11-21-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: dnslogs/2022-11-23/2022-11-23-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: ++ Skipping previously processed file: dnslogs/2022-11-26/2022-11-26-00-00-ioxa.csv
DEBUG    wazuh_testing:cli_utils.py:52 DEBUG: +++ DB Maintenance
ERROR    wazuh_testing:cli_utils.py:62 Some logs may were processed
ERROR    wazuh_testing:cli_utils.py:63 Results found: 0
ERROR    wazuh_testing:cli_utils.py:64 Results expected: 1
=============================== warnings summary ===============================
../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:387
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:387: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/lastfailed
    config.cache.set("cache/lastfailed", self.lastfailed)

../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52
  /usr/local/lib/python3.10/dist-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /home/vagrant/qa/tests/integration/.pytest_cache/v/cache/stepwise
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test_only_logs_after.py::test_multiple_calls[cisco_umbrella_only_logs_after_multiple_calls]
================ 1 failed, 23 deselected, 3 warnings in 26.22s =================

Footnotes

1. Because the module is not showing the message about no logs to process, the server_access_only_logs_after_multiple_calls test is failing. This should be fixed in his respective issue. ↩

[fdalmaup]

The Tier 0 test_discard_regex[cisco_umbrella_discard_regex] fails with the following output:

root@ubuntu-focal:/home/vagrant/wazuh-qa/tests/integration/test_aws# pytest -vv -k 'umbrella' --tier 0 
====================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-5.0.0, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.8.10', 'Platform': 'Linux-5.4.0-137-generic-x86_64-with-glibc2.29', 'Packages': {'pytest': '5.0.0', 'pluggy': '0.13.1'}, 'Plugins': {'html': '2.0.1', 'metadata': '2.0.4', 'testinfra': '5.0.0'}}
rootdir: /home/vagrant/wazuh-qa/tests/integration, inifile: pytest.ini
plugins: html-2.0.1, metadata-2.0.4, testinfra-5.0.0
collected 24 items / 16 deselected / 8 selected                                                                                                                 

test_basic.py::test_defaults[cisco_umbrella_defaults] PASSED                                                                                              [ 12%]
test_discard_regex.py::test_discard_regex[cisco_umbrella_discard_regex] FAILED                                                                            [ 25%]
test_only_logs_after.py::test_without_only_logs_after[cisco_umbrella_without_only_logs_after] PASSED                                                      [ 37%]
test_only_logs_after.py::test_with_only_logs_after[cisco_umbrella_with_only_logs_after] PASSED                                                            [ 50%]
test_path.py::test_path[cisco_umbrella_path_with_data] PASSED                                                                                             [ 62%]
test_path.py::test_path[cisco_umbrella_path_without_data] PASSED                                                                                          [ 75%]
test_path.py::test_path[cisco_umbrella_inexistent_path] PASSED                                                                                            [ 87%]
test_remove_from_bucket.py::test_remove_from_bucket[cisco_umbrella_remove_from_bucket] PASSED                                                             [100%]

=========================================================================== FAILURES ============================================================================
_______________________________________________________ test_discard_regex[cisco_umbrella_discard_regex] ________________________________________________________

configuration = {'metadata': {'bucket_name': 'wazuh-umbrella-integration-tests', 'bucket_type': 'cisco_umbrella', 'description': 'Clou...elements': [{'disabled': {'value': 'no'}}, {'bucket': {'attributes': [...], 'elements': [...]}}], 'section': 'wodle'}]}
metadata = {'bucket_name': 'wazuh-umbrella-integration-tests', 'bucket_type': 'cisco_umbrella', 'description': 'CloudTrail discard regex configurations', 'discard_field': 'action', ...}
load_wazuh_basic_configuration = None, set_wazuh_configuration = None, clean_s3_cloudtrail_db = None, configure_local_internal_options_function = None
truncate_monitored_files = None, restart_wazuh_function = None, wazuh_log_monitor = <wazuh_testing.tools.monitoring.FileMonitor object at 0x7f1bd56e5bb0>

    @pytest.mark.tier(level=0)
    @pytest.mark.parametrize('configuration, metadata', zip(configurations, configuration_metadata), ids=case_ids)
    def test_discard_regex(
        configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db,
        configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, wazuh_log_monitor,
    ):
        """
        description: Fetch logs excluding the ones that match with the regex.
        test_phases:
            - setup:
                - Load Wazuh light configuration.
                - Apply ossec.conf configuration changes according to the configuration template and use case.
                - Apply custom settings in local_internal_options.conf.
                - Truncate wazuh logs.
                - Restart wazuh-manager service to apply configuration changes.
            - test:
                - Check in the ossec.log that a line has appeared calling the module with correct parameters.
                - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket and skips
                  the ones that match with regex.
                - Check the database was created and updated accordingly.
            - teardown:
                - Truncate wazuh logs.
                - Restore initial configuration, both ossec.conf and local_internal_options.conf.
                - Delete the uploaded file
        wazuh_min_version: 4.5.0
        parameters:
            - configuration:
                type: dict
                brief: Get configurations from the module.
            - metadata:
                type: dict
                brief: Get metadata from the module.
            - load_wazuh_basic_configuration:
                type: fixture
                brief: Load basic wazuh configuration.
            - set_wazuh_configuration:
                type: fixture
                brief: Apply changes to the ossec.conf configuration.
            - clean_s3_cloudtrail_db:
                type: fixture
                brief: Delete the DB file before and after the test execution.
            - configure_local_internal_options_function:
                type: fixture
                brief: Apply changes to the local_internal_options.conf configuration.
            - truncate_monitored_files:
                type: fixture
                brief: Truncate wazuh logs.
            - restart_wazuh_daemon_function:
                type: fixture
                brief: Restart the wazuh service.
            - wazuh_log_monitor:
                type: fixture
                brief: Return a `ossec.log` monitor.
        assertions:
            - Check in the log that the module was called with correct parameters.
            - Check the expected number of events were forwarded to analysisd.
            - Check the database was created and updated accordingly.
        input_description:
            - The `configuration_discard_regex` file provides the module configuration for this test.
            - The `cases_discard_regex` file provides the test cases.
        """
        bucket_name = metadata['bucket_name']
        bucket_type = metadata['bucket_type']
        only_logs_after = metadata['only_logs_after']
        discard_field = metadata['discard_field']
        discard_regex = metadata['discard_regex']
        found_logs = metadata['found_logs']
        skipped_logs = metadata['skipped_logs']
        path = metadata.get('path')
    
        pattern = fr'.*The "{discard_regex}" regex found a match in the "{discard_field}" field. The event will be skipped.'
    
        parameters = [
            'wodles/aws/aws-s3',
            '--bucket', bucket_name,
            '--aws_profile', 'qa',
            '--only_logs_after', only_logs_after,
            '--discard-field', discard_field,
            '--discard-regex', discard_regex,
            '--type', bucket_type,
            '--debug', '2'
        ]
    
        if path is not None:
            parameters.insert(5, path)
            parameters.insert(5, '--trail_prefix')
    
        # Check AWS module started
        wazuh_log_monitor.start(
            timeout=global_parameters.default_timeout,
            callback=event_monitor.callback_detect_aws_module_start,
            error_message='The AWS module did not start as expected',
        ).result()
    
        # Check command was called correctly
        wazuh_log_monitor.start(
            timeout=global_parameters.default_timeout,
            callback=event_monitor.callback_detect_aws_module_called(parameters),
            error_message='The AWS module was not called with the correct parameters',
        ).result()
    
>       wazuh_log_monitor.start(
            timeout=T_20,
            callback=event_monitor.callback_detect_event_processed_or_skipped(pattern),
            error_message=(
                'The AWS module did not show correct message about discard regex or ',
                'did not process the expected amout of logs'
            ),
            accum_results=found_logs + skipped_logs
        ).result()

test_discard_regex.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/wazuh_testing/tools/monitoring.py:201: in start
    self._result = monitor.start(timeout=timeout, callback=callback, accum_results=accum_results,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <wazuh_testing.tools.monitoring.QueueMonitor object at 0x7f1bd56e5160>, timeout = 20
callback = <function callback_detect_event_processed_or_skipped.<locals>.<lambda> at 0x7f1bd56e3ee0>, accum_results = 6, update_position = True
timeout_extra = 0, error_message = ('The AWS module did not show correct message about discard regex or ', 'did not process the expected amout of logs')

    def start(self, timeout=-1, callback=_callback_default, accum_results=1, update_position=True, timeout_extra=0,
              error_message=''):
        """Start the queue monitoring until the stop method is called."""
        if not self._continue:
            self._continue = True
            self._abort = False
            result = None
    
            while self._continue:
                if self._abort:
                    self.stop()
                    if error_message:
                        logger.error(error_message)
                        logger.error(f"Results accumulated: "
                                     f"{len(result) if isinstance(result, list) else 0}")
                        logger.error(f"Results expected: {accum_results}")
>                   raise TimeoutError(error_message)
E                   TimeoutError: ('The AWS module did not show correct message about discard regex or ', 'did not process the expected amout of logs')

/usr/local/lib/python3.8/dist-packages/wazuh_testing/tools/monitoring.py:469: TimeoutError
--------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------
2023-02-02 18:26:03,927 - wazuh_testing - DEBUG - Set local_internal_option to {'wazuh_modules.debug': '2', 'monitord.rotate_log': '0'}
---------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------
DEBUG    wazuh_testing:conftest.py:619 Set local_internal_option to {'wazuh_modules.debug': '2', 'monitord.rotate_log': '0'}
--------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------
2023-02-02 18:26:37,889 - wazuh_testing - ERROR - ('The AWS module did not show correct message about discard regex or ', 'did not process the expected amout of logs')
2023-02-02 18:26:37,890 - wazuh_testing - ERROR - Results accumulated: 4
2023-02-02 18:26:37,891 - wazuh_testing - ERROR - Results expected: 6
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
ERROR    wazuh_testing:monitoring.py:465 ('The AWS module did not show correct message about discard regex or ', 'did not process the expected amout of logs')
ERROR    wazuh_testing:monitoring.py:466 Results accumulated: 4
ERROR    wazuh_testing:monitoring.py:468 Results expected: 6
------------------------------------------------------------------- Captured stderr teardown --------------------------------------------------------------------
2023-02-02 18:26:42,009 - wazuh_testing - DEBUG - Restore local_internal_option to {'syscheck.debug': '2\n', 'monitord.rotate_log': '0\n'}
--------------------------------------------------------------------- Captured log teardown ---------------------------------------------------------------------
DEBUG    wazuh_testing:conftest.py:624 Restore local_internal_option to {'syscheck.debug': '2\n', 'monitord.rotate_log': '0\n'}
====================================================== 1 failed, 7 passed, 16 deselected in 188.24 seconds ======================================================

[nico-stefani]

The Tier 0 test_discard_regex[cisco_umbrella_discard_regex] fails with the following output:

Fixed in 81a3e51

[fdalmaup]

LGTM