Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate the error in the agent_files_deletion test #2282

Closed
yanazaeva opened this issue Nov 26, 2021 · 3 comments · Fixed by #2296
Closed

Investigate the error in the agent_files_deletion test #2282

yanazaeva opened this issue Nov 26, 2021 · 3 comments · Fixed by #2296
Assignees

Comments

@yanazaeva
Copy link
Contributor

While running the system test for the 4.3 release, it was noticed that this test was failing half of the time it was runned:

python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -xvvs
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                                       

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion FAILED

=============================================================================================== FAILURES ===============================================================================================
______________________________________________________________________________________ test_agent_files_deletion _______________________________________________________________________________________

register_agent = None

    def test_agent_files_deletion(register_agent):
        """Check that when an agent is deleted, all its related files in managers are also removed."""
        # Get the current ID and name of the agent that is reporting to worker_host.
        master_token = host_manager.get_api_token(master_host)
        response = host_manager.make_api_call(host=master_host, method='GET', token=master_token,
                                              endpoint=f'/agents?select=id,name&q=manager={worker_host}')
        assert response['status'] == 200, f'Failed when trying to obtain agent ID: {response}'
        try:
            agent_id = response['json']['data']['affected_items'][0]['id']
            agent_name = response['json']['data']['affected_items'][0]['name']
        except IndexError as e:
            pytest.fail(f"Could not find any agent reporting to {worker_host}: {response['json']}")
    
        # Check that expected files exist in each node before removing the agent.
        for file in files:
            for host in file['hosts']:
                result = host_manager.run_shell(
                    host, f'test -e {file["path"].format(id=agent_id, name=agent_name)} && echo "exists"'
                )
                assert result, f'This file should exist in {host} but could not be found: ' \
                               f'{file["path"].format(id=agent_id, name=agent_name)}'
    
        # Check that agent information exists in global.db
        for host in managers_hosts:
            for query in db_queries:
                result = host_manager.run_command(
                    host,
                    f'sqlite3 {join(WAZUH_PATH, "queue", "db", "global.db")} "{query.format(id=agent_id, name=agent_name)}"'
                )
>               assert result, f'This db query should have returned something in {host}, but it did not: ' \
                               f'{query.format(id=agent_id, name=agent_name)}'
E               AssertionError: This db query should have returned something in wazuh-master, but it did not: select * from agent where id=013
E               assert ''

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py:66: AssertionError
------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmpc68p2rja -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes' wazuh-master", exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "433",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Fri, 26 Nov 2021 12:29:39 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM3OTI5Nzc5LCJleHAiOjE2Mzc5MzA2NzksInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AJpmYLgT0j_uvgH3aAvF2d0Wy7FxD-6MHxLtRT0w1KcyZQy_JUamUXNPZS5-YMaU1w-aJ7lm54OAb-vRfIlJRb67ABpiwQlc86JtrwR0NSZQNIj_NI8DKXLFG_Rrt7m2_QO31u7Ygh0F9lUODLjubyuvgXAIThvdwsNdMJQ4jJTnZJeZ"\n        },\n        "error": 0\n    },\n    "msg": "OK (433 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/security/user/authenticate",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '433',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Fri, 26 Nov 2021 12:29:39 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'token': 'eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM3OTI5Nzc5LCJleHAiOjE2Mzc5MzA2NzksInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AJpmYLgT0j_uvgH3aAvF2d0Wy7FxD-6MHxLtRT0w1KcyZQy_JUamUXNPZS5-YMaU1w-aJ7lm54OAb-vRfIlJRb67ABpiwQlc86JtrwR0NSZQNIj_NI8DKXLFG_Rrt7m2_QO31u7Ygh0F9lUODLjubyuvgXAIThvdwsNdMJQ4jJTnZJeZ'},
          'error': 0},
 'msg': 'OK (433 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/security/user/authenticate',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpo1k_qcfo -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args \'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'"\'"\'Authorization\'"\'"\': \'"\'"\'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM3OTI5Nzc5LCJleHAiOjE2Mzc5MzA2NzksInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AJpmYLgT0j_uvgH3aAvF2d0Wy7FxD-6MHxLtRT0w1KcyZQy_JUamUXNPZS5-YMaU1w-aJ7lm54OAb-vRfIlJRb67ABpiwQlc86JtrwR0NSZQNIj_NI8DKXLFG_Rrt7m2_QO31u7Ygh0F9lUODLjubyuvgXAIThvdwsNdMJQ4jJTnZJeZ\'"\'"\'}"  validate_certs=no\' wazuh-master', exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "212",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Fri, 26 Nov 2021 12:29:40 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "affected_items": [\n                {\n                    "id": "013",\n                    "name": "wazuh-agent3"\n                }\n            ],\n            "failed_items": [],\n            "total_affected_items": 1,\n            "total_failed_items": 0\n        },\n        "error": 0,\n        "message": "All selected agents information was returned"\n    },\n    "msg": "OK (212 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'Authorization\': \'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM3OTI5Nzc5LCJleHAiOjE2Mzc5MzA2NzksInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AJpmYLgT0j_uvgH3aAvF2d0Wy7FxD-6MHxLtRT0w1KcyZQy_JUamUXNPZS5-YMaU1w-aJ7lm54OAb-vRfIlJRb67ABpiwQlc86JtrwR0NSZQNIj_NI8DKXLFG_Rrt7m2_QO31u7Ygh0F9lUODLjubyuvgXAIThvdwsNdMJQ4jJTnZJeZ\'}"  validate_certs=no', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '212',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Fri, 26 Nov 2021 12:29:40 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'affected_items': [{'id': '013', 'name': 'wazuh-agent3'}],
                   'failed_items': [],
                   'total_affected_items': 1,
                   'total_failed_items': 0},
          'error': 0,
          'message': 'All selected agents information was returned'},
 'msg': 'OK (212 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpufy6nt80 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/rids/013 && echo "exists"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/rids/013 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/rids/013 && echo "exists"',
 'delta': '0:00:00.071357',
 'end': '2021-11-26 13:29:41.009426',
 'rc': 0,
 'start': '2021-11-26 13:29:40.938069',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpccuuzyuy -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/rids/013 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/rids/013 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/rids/013 && echo "exists"',
 'delta': '0:00:00.068539',
 'end': '2021-11-26 13:29:41.734327',
 'rc': 0,
 'start': '2021-11-26 13:29:41.665788',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpwzk8l1ip -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/agent-groups/013 && echo "exists"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/agent-groups/013 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/agent-groups/013 && echo "exists"',
 'delta': '0:00:00.071960',
 'end': '2021-11-26 13:29:42.504928',
 'rc': 0,
 'start': '2021-11-26 13:29:42.432968',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpxc5omytd -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/agent-groups/013 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/agent-groups/013 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/agent-groups/013 && echo "exists"',
 'delta': '0:00:00.070932',
 'end': '2021-11-26 13:29:43.263617',
 'rc': 0,
 'start': '2021-11-26 13:29:43.192685',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpoaoo4926 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"',
 'delta': '0:00:00.069628',
 'end': '2021-11-26 13:29:44.001697',
 'rc': 0,
 'start': '2021-11-26 13:29:43.932069',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp7uegt2f2 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/db/013.db && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/db/013.db && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/db/013.db && echo "exists"',
 'delta': '0:00:00.071757',
 'end': '2021-11-26 13:29:44.714212',
 'rc': 0,
 'start': '2021-11-26 13:29:44.642455',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpxex93_mv -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m command --args \'sqlite3 /var/ossec/queue/db/global.db "select * from agent where id=013"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\n\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('command', 'sqlite3 /var/ossec/queue/db/global.db "select * from agent where id=013"', {'check': False, 'become': False}): {'changed': True,
 'cmd': ['sqlite3',
         '/var/ossec/queue/db/global.db',
         'select * from agent where id=013'],
 'delta': '0:00:00.070934',
 'end': '2021-11-26 13:29:45.463859',
 'rc': 0,
 'start': '2021-11-26 13:29:45.392925',
 'stderr': '',
 'stderr_lines': [],
 'stdout': '',
 'stdout_lines': []}
---------------------------------------------------------------------------------------- Captured log teardown -----------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmpad7cxpf6 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m command --args 'service wazuh-agent restart' wazuh-agent3", exit_status=0, stdout=b'wazuh-agent3 | CHANGED | rc=0 >>\nKilling wazuh-modulesd... \nKilling wazuh-logcollector... \nKilling wazuh-syscheckd... \nKilling wazuh-agentd... \nKilling wazuh-execd... \nWazuh v4.3.0 Stopped\nStarting Wazuh v4.3.0...\nStarted wazuh-execd...\nStarted wazuh-agentd...\nStarted wazuh-syscheckd...\nStarted wazuh-logcollector...\nStarted wazuh-modulesd...\nCompleted.\n', stderr=b"[WARNING]: Consider using the service module rather than running 'service'.  If\nyou need to use command because service is insufficient you can add 'warn:\nfalse' to this command task or set 'command_warnings=False' in ansible.cfg to\nget rid of this message.\n")
INFO     testinfra:ansible.py:71 RUN Ansible('command', 'service wazuh-agent restart', {'check': False, 'become': False}): {'changed': True,
 'cmd': ['service', 'wazuh-agent', 'restart'],
 'delta': '0:00:07.663619',
 'end': '2021-11-26 13:29:53.828732',
 'rc': 0,
 'start': '2021-11-26 13:29:46.165113',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'Killing wazuh-modulesd... \n'
           'Killing wazuh-logcollector... \n'
           'Killing wazuh-syscheckd... \n'
           'Killing wazuh-agentd... \n'
           'Killing wazuh-execd... \n'
           'Wazuh v4.3.0 Stopped\n'
           'Starting Wazuh v4.3.0...\n'
           'Started wazuh-execd...\n'
           'Started wazuh-agentd...\n'
           'Started wazuh-syscheckd...\n'
           'Started wazuh-logcollector...\n'
           'Started wazuh-modulesd...\n'
           'Completed.',
 'stdout_lines': ['Killing wazuh-modulesd... ',
                  'Killing wazuh-logcollector... ',
                  'Killing wazuh-syscheckd... ',
                  'Killing wazuh-agentd... ',
                  'Killing wazuh-execd... ',
                  'Wazuh v4.3.0 Stopped',
                  'Starting Wazuh v4.3.0...',
                  'Started wazuh-execd...',
                  'Started wazuh-agentd...',
                  'Started wazuh-syscheckd...',
                  'Started wazuh-logcollector...',
                  'Started wazuh-modulesd...',
                  'Completed.'],
 'warnings': ['Consider using the service module rather than running '
              "'service'.  If you need to use command because service is "
              "insufficient you can add 'warn: false' to this command task or "
              "set 'command_warnings=False' in ansible.cfg to get rid of this "
              'message.']}
======================================================================================= short test summary info ========================================================================================
FAILED ../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion - AssertionError: This db query should have returned something in wazuh-master, but it di...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================== 1 failed in 15.76s ==========================================================================================

It seems that it is failing when checking whether there is any information about a particular agent in the global.db so this is probably because the manager did not had time enough to register it into the database.

@yanazaeva
Copy link
Contributor Author

yanazaeva commented Dec 2, 2021

Issue update

While looking for the root cause of this issue, it was seen that several problems take place in this test. In the first run, we can see that everything goes smooth:

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

================================================================================ 1 passed in 85.14s (0:01:25) ================================================================================

Before this first execution, we were able to check that the worker2, which is the one being used for the test has a connected agent:

ID   NAME          IP          STATUS  VERSION       NODE NAME      
000  wazuh-master  127.0.0.1   active  Wazuh v4.3.0  wazuh-master   
001  wazuh-agent1  172.17.0.5  active  Wazuh v4.3.0  wazuh-master   
002  wazuh-agent2  172.17.0.6  active  Wazuh v4.3.0  wazuh-worker1  
010  wazuh-agent3  172.17.0.7  active  Wazuh v4.3.0  wazuh-worker2 

After this first execution, if we run the test again we will face the following trouble:

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion FAILED                                                                            [100%]

========================================================================================== FAILURES ==========================================================================================
_________________________________________________________________________________ test_agent_files_deletion __________________________________________________________________________________

register_agent = None

    def test_agent_files_deletion(register_agent):
        """Check that when an agent is deleted, all its related files in managers are also removed."""
        # Get the current ID and name of the agent that is reporting to worker_host.
        master_token = host_manager.get_api_token(master_host)
        response = host_manager.make_api_call(host=master_host, method='GET', token=master_token,
                                              endpoint=f'/agents?select=id,name&q=manager={worker_host}')
        assert response['status'] == 200, f'Failed when trying to obtain agent ID: {response}'
        try:
            agent_id = response['json']['data']['affected_items'][0]['id']
            agent_name = response['json']['data']['affected_items'][0]['name']
        except IndexError as e:
            pytest.fail(f"Could not find any agent reporting to {worker_host}: {response['json']}")
    
        # Check that expected files exist in each node before removing the agent.
        for file in files:
            for host in file['hosts']:
                result = host_manager.run_shell(
                    host, f'test -e {file["path"].format(id=agent_id, name=agent_name)} && echo "exists"'
                )
                assert result, f'This file should exist in {host} but could not be found: ' \
                               f'{file["path"].format(id=agent_id, name=agent_name)}'
    
        # Check that agent information exists in global.db
        for host in managers_hosts:
            for query in db_queries:
                result = host_manager.run_command(
                    host,
                    f'sqlite3 {join(WAZUH_PATH, "queue", "db", "global.db")} "{query.format(id=agent_id, name=agent_name)}"'
                )
>               assert result, f'This db query should have returned something in {host}, but it did not: ' \
                               f'{query.format(id=agent_id, name=agent_name)}'
E               AssertionError: This db query should have returned something in wazuh-master, but it did not: select * from agent where id=010
E               assert ''

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py:66: AssertionError
------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmpbssu2hs3 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes' wazuh-master", exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "433",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Thu, 02 Dec 2021 10:04:10 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDM5NDUwLCJleHAiOjE2Mzg0NDAzNTAsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AHSK9gDYOv-mQy1nl8gnGqzycCHBbx6CInakGlBXFimXKttsTt8riKXfxpwY9Ni2XouWpDFShfGTbINaB-XIlUlBAOo5SFy0ObKNX8qU1uZPlIcXdt0sr2A3efvNuZ9mB9yIQGXJofpeGW8g5K3CHHHt45D9r__AbLlLzv6FtBMuMJLB"\n        },\n        "error": 0\n    },\n    "msg": "OK (433 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/security/user/authenticate",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '433',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Thu, 02 Dec 2021 10:04:10 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'token': 'eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDM5NDUwLCJleHAiOjE2Mzg0NDAzNTAsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AHSK9gDYOv-mQy1nl8gnGqzycCHBbx6CInakGlBXFimXKttsTt8riKXfxpwY9Ni2XouWpDFShfGTbINaB-XIlUlBAOo5SFy0ObKNX8qU1uZPlIcXdt0sr2A3efvNuZ9mB9yIQGXJofpeGW8g5K3CHHHt45D9r__AbLlLzv6FtBMuMJLB'},
          'error': 0},
 'msg': 'OK (433 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/security/user/authenticate',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp23anidu6 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args \'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'"\'"\'Authorization\'"\'"\': \'"\'"\'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDM5NDUwLCJleHAiOjE2Mzg0NDAzNTAsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AHSK9gDYOv-mQy1nl8gnGqzycCHBbx6CInakGlBXFimXKttsTt8riKXfxpwY9Ni2XouWpDFShfGTbINaB-XIlUlBAOo5SFy0ObKNX8qU1uZPlIcXdt0sr2A3efvNuZ9mB9yIQGXJofpeGW8g5K3CHHHt45D9r__AbLlLzv6FtBMuMJLB\'"\'"\'}"  validate_certs=no\' wazuh-master', exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "212",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Thu, 02 Dec 2021 10:04:11 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "affected_items": [\n                {\n                    "id": "010",\n                    "name": "wazuh-agent3"\n                }\n            ],\n            "failed_items": [],\n            "total_affected_items": 1,\n            "total_failed_items": 0\n        },\n        "error": 0,\n        "message": "All selected agents information was returned"\n    },\n    "msg": "OK (212 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'Authorization\': \'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDM5NDUwLCJleHAiOjE2Mzg0NDAzNTAsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.AHSK9gDYOv-mQy1nl8gnGqzycCHBbx6CInakGlBXFimXKttsTt8riKXfxpwY9Ni2XouWpDFShfGTbINaB-XIlUlBAOo5SFy0ObKNX8qU1uZPlIcXdt0sr2A3efvNuZ9mB9yIQGXJofpeGW8g5K3CHHHt45D9r__AbLlLzv6FtBMuMJLB\'}"  validate_certs=no', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '212',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Thu, 02 Dec 2021 10:04:11 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'affected_items': [{'id': '010', 'name': 'wazuh-agent3'}],
                   'failed_items': [],
                   'total_affected_items': 1,
                   'total_failed_items': 0},
          'error': 0,
          'message': 'All selected agents information was returned'},
 'msg': 'OK (212 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpxjkobhbm -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/rids/010 && echo "exists"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/rids/010 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/rids/010 && echo "exists"',
 'delta': '0:00:00.073252',
 'end': '2021-12-02 11:04:12.363360',
 'rc': 0,
 'start': '2021-12-02 11:04:12.290108',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmph1y6fbfc -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/rids/010 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/rids/010 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/rids/010 && echo "exists"',
 'delta': '0:00:00.073352',
 'end': '2021-12-02 11:04:13.053511',
 'rc': 0,
 'start': '2021-12-02 11:04:12.980159',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpuhyxa2nv -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/agent-groups/010 && echo "exists"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/agent-groups/010 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/agent-groups/010 && echo "exists"',
 'delta': '0:00:00.072503',
 'end': '2021-12-02 11:04:13.713015',
 'rc': 0,
 'start': '2021-12-02 11:04:13.640512',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp7fjbv_7j -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/agent-groups/010 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/agent-groups/010 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/agent-groups/010 && echo "exists"',
 'delta': '0:00:00.068872',
 'end': '2021-12-02 11:04:14.394664',
 'rc': 0,
 'start': '2021-12-02 11:04:14.325792',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp1qkd3jgo -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/diff/wazuh-agent3 && echo "exists"',
 'delta': '0:00:00.072623',
 'end': '2021-12-02 11:04:15.065508',
 'rc': 0,
 'start': '2021-12-02 11:04:14.992885',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp3rlggcv4 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m shell --args \'test -e /var/ossec/queue/db/010.db && echo "exists"\' wazuh-worker2', exit_status=0, stdout=b'wazuh-worker2 | CHANGED | rc=0 >>\nexists\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('shell', 'test -e /var/ossec/queue/db/010.db && echo "exists"', {'check': False, 'become': False}): {'changed': True,
 'cmd': 'test -e /var/ossec/queue/db/010.db && echo "exists"',
 'delta': '0:00:00.069652',
 'end': '2021-12-02 11:04:15.718115',
 'rc': 0,
 'start': '2021-12-02 11:04:15.648463',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'exists',
 'stdout_lines': ['exists']}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmp_hkl5ix8 -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m command --args \'sqlite3 /var/ossec/queue/db/global.db "select * from agent where id=010"\' wazuh-master', exit_status=0, stdout=b'wazuh-master | CHANGED | rc=0 >>\n\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('command', 'sqlite3 /var/ossec/queue/db/global.db "select * from agent where id=010"', {'check': False, 'become': False}): {'changed': True,
 'cmd': ['sqlite3',
         '/var/ossec/queue/db/global.db',
         'select * from agent where id=010'],
 'delta': '0:00:00.068996',
 'end': '2021-12-02 11:04:16.390266',
 'rc': 0,
 'start': '2021-12-02 11:04:16.321270',
 'stderr': '',
 'stderr_lines': [],
 'stdout': '',
 'stdout_lines': []}
----------------------------------------------------------------------------------- Captured log teardown ------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmpakwyghcg -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m command --args 'service wazuh-agent restart' wazuh-agent3", exit_status=0, stdout=b'wazuh-agent3 | CHANGED | rc=0 >>\nKilling wazuh-modulesd... \nKilling wazuh-logcollector... \nKilling wazuh-syscheckd... \nKilling wazuh-agentd... \nKilling wazuh-execd... \nWazuh v4.3.0 Stopped\nStarting Wazuh v4.3.0...\nStarted wazuh-execd...\nStarted wazuh-agentd...\nStarted wazuh-syscheckd...\nStarted wazuh-logcollector...\nStarted wazuh-modulesd...\nCompleted.\n', stderr=b"[WARNING]: Consider using the service module rather than running 'service'.  If\nyou need to use command because service is insufficient you can add 'warn:\nfalse' to this command task or set 'command_warnings=False' in ansible.cfg to\nget rid of this message.\n")
INFO     testinfra:ansible.py:71 RUN Ansible('command', 'service wazuh-agent restart', {'check': False, 'become': False}): {'changed': True,
 'cmd': ['service', 'wazuh-agent', 'restart'],
 'delta': '0:00:07.749476',
 'end': '2021-12-02 11:04:24.769609',
 'rc': 0,
 'start': '2021-12-02 11:04:17.020133',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'Killing wazuh-modulesd... \n'
           'Killing wazuh-logcollector... \n'
           'Killing wazuh-syscheckd... \n'
           'Killing wazuh-agentd... \n'
           'Killing wazuh-execd... \n'
           'Wazuh v4.3.0 Stopped\n'
           'Starting Wazuh v4.3.0...\n'
           'Started wazuh-execd...\n'
           'Started wazuh-agentd...\n'
           'Started wazuh-syscheckd...\n'
           'Started wazuh-logcollector...\n'
           'Started wazuh-modulesd...\n'
           'Completed.',
 'stdout_lines': ['Killing wazuh-modulesd... ',
                  'Killing wazuh-logcollector... ',
                  'Killing wazuh-syscheckd... ',
                  'Killing wazuh-agentd... ',
                  'Killing wazuh-execd... ',
                  'Wazuh v4.3.0 Stopped',
                  'Starting Wazuh v4.3.0...',
                  'Started wazuh-execd...',
                  'Started wazuh-agentd...',
                  'Started wazuh-syscheckd...',
                  'Started wazuh-logcollector...',
                  'Started wazuh-modulesd...',
                  'Completed.'],
 'warnings': ['Consider using the service module rather than running '
              "'service'.  If you need to use command because service is "
              "insufficient you can add 'warn: false' to this command task or "
              "set 'command_warnings=False' in ansible.cfg to get rid of this "
              'message.']}
================================================================================== short test summary info ===================================================================================
FAILED ../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion - AssertionError: This db query should have returned something in wazuh-master,...
===================================================================================== 1 failed in 15.33s =====================================================================================

We will solve this problem by adding a sleep, to make sure that the information got time enough to be synchronized. With this little change, the test passed five out of five times.

However, another problem was encountered here. The error shown below seems to be appearing whenever we run the test and do not have any agent reporting to the worker2:

========================================================================================== FAILURES ==========================================================================================
_________________________________________________________________________________ test_agent_files_deletion __________________________________________________________________________________

register_agent = None

    def test_agent_files_deletion(register_agent):
        """Check that when an agent is deleted, all its related files in managers are also removed."""
        # Get the current ID and name of the agent that is reporting to worker_host.
        master_token = host_manager.get_api_token(master_host)
        response = host_manager.make_api_call(host=master_host, method='GET', token=master_token,
                                              endpoint=f'/agents?select=id,name&q=manager={worker_host}')
        assert response['status'] == 200, f'Failed when trying to obtain agent ID: {response}'
        try:
>           agent_id = response['json']['data']['affected_items'][0]['id']
E           IndexError: list index out of range

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py:45: IndexError

During handling of the above exception, another exception occurred:

register_agent = None

    def test_agent_files_deletion(register_agent):
        """Check that when an agent is deleted, all its related files in managers are also removed."""
        # Get the current ID and name of the agent that is reporting to worker_host.
        master_token = host_manager.get_api_token(master_host)
        response = host_manager.make_api_call(host=master_host, method='GET', token=master_token,
                                              endpoint=f'/agents?select=id,name&q=manager={worker_host}')
        assert response['status'] == 200, f'Failed when trying to obtain agent ID: {response}'
        try:
            agent_id = response['json']['data']['affected_items'][0]['id']
            agent_name = response['json']['data']['affected_items'][0]['name']
        except IndexError as e:
>           pytest.fail(f"Could not find any agent reporting to {worker_host}: {response['json']}")
E           Failed: Could not find any agent reporting to wazuh-worker2: {'data': {'affected_items': [], 'failed_items': [], 'total_affected_items': 0, 'total_failed_items': 0}, 'error': 0, 'message': 'No agent information was returned'}

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py:48: Failed
------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmpgu3ozqzp -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes' wazuh-master", exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "433",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Thu, 02 Dec 2021 11:21:11 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDQ0MDcxLCJleHAiOjE2Mzg0NDQ5NzEsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.Aet1-SYZUxS0dX10bWx0nAZQdUjgU-UtL31mK4YJO502OIfOyw3SNqOpCLTeq4vIXsxtuFNFBZpOjTA3apAKHcXxAIa9loPEwJlaKmY6YkZMmlGg4oVT9hyZNRph492L5JEIs1KGrZpm_zb0E7t3MfgWCdwH_Nr0RBL0LCK371_V8pAQ"\n        },\n        "error": 0\n    },\n    "msg": "OK (433 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/security/user/authenticate",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url=https://localhost:55000/security/user/authenticate user=wazuh password=wazuh method=GET  validate_certs=no force_basic_auth=yes', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '433',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Thu, 02 Dec 2021 11:21:11 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'token': 'eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDQ0MDcxLCJleHAiOjE2Mzg0NDQ5NzEsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.Aet1-SYZUxS0dX10bWx0nAZQdUjgU-UtL31mK4YJO502OIfOyw3SNqOpCLTeq4vIXsxtuFNFBZpOjTA3apAKHcXxAIa9loPEwJlaKmY6YkZMmlGg4oVT9hyZNRph492L5JEIs1KGrZpm_zb0E7t3MfgWCdwH_Nr0RBL0LCK371_V8pAQ'},
          'error': 0},
 'msg': 'OK (433 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/security/user/authenticate',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b'ansible --tree /tmp/tmpzfy01c5u -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m uri --args \'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'"\'"\'Authorization\'"\'"\': \'"\'"\'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDQ0MDcxLCJleHAiOjE2Mzg0NDQ5NzEsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.Aet1-SYZUxS0dX10bWx0nAZQdUjgU-UtL31mK4YJO502OIfOyw3SNqOpCLTeq4vIXsxtuFNFBZpOjTA3apAKHcXxAIa9loPEwJlaKmY6YkZMmlGg4oVT9hyZNRph492L5JEIs1KGrZpm_zb0E7t3MfgWCdwH_Nr0RBL0LCK371_V8pAQ\'"\'"\'}"  validate_certs=no\' wazuh-master', exit_status=0, stdout=b'wazuh-master | SUCCESS => {\n    "cache_control": "no-cache, no-store, must-revalidate, max-age=0",\n    "changed": false,\n    "connection": "close",\n    "content_length": "164",\n    "content_security_policy": "none",\n    "content_type": "application/json; charset=utf-8",\n    "cookies": {},\n    "cookies_string": "",\n    "date": "Thu, 02 Dec 2021 11:21:12 GMT",\n    "elapsed": 0,\n    "expires": "0",\n    "json": {\n        "data": {\n            "affected_items": [],\n            "failed_items": [],\n            "total_affected_items": 0,\n            "total_failed_items": 0\n        },\n        "error": 0,\n        "message": "No agent information was returned"\n    },\n    "msg": "OK (164 bytes)",\n    "pragma": "no-cache",\n    "redirected": false,\n    "referrer_policy": "no-referrer, strict-origin-when-cross-origin",\n    "status": 200,\n    "strict_transport_security": "max-age=63072000; includeSubdomains",\n    "url": "https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2",\n    "x_content_type_options": "nosniff",\n    "x_frame_options": "DENY",\n    "x_xss_protection": "1; mode=block"\n}\n', stderr=None)
INFO     testinfra:ansible.py:71 RUN Ansible('uri', 'url="https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2" method=GET headers="{\'Authorization\': \'Bearer eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNjM4NDQ0MDcxLCJleHAiOjE2Mzg0NDQ5NzEsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.Aet1-SYZUxS0dX10bWx0nAZQdUjgU-UtL31mK4YJO502OIfOyw3SNqOpCLTeq4vIXsxtuFNFBZpOjTA3apAKHcXxAIa9loPEwJlaKmY6YkZMmlGg4oVT9hyZNRph492L5JEIs1KGrZpm_zb0E7t3MfgWCdwH_Nr0RBL0LCK371_V8pAQ\'}"  validate_certs=no', {'check': False, 'become': False}): {'cache_control': 'no-cache, no-store, must-revalidate, max-age=0',
 'changed': False,
 'connection': 'close',
 'content_length': '164',
 'content_security_policy': 'none',
 'content_type': 'application/json; charset=utf-8',
 'cookies': {},
 'cookies_string': '',
 'date': 'Thu, 02 Dec 2021 11:21:12 GMT',
 'elapsed': 0,
 'expires': '0',
 'json': {'data': {'affected_items': [],
                   'failed_items': [],
                   'total_affected_items': 0,
                   'total_failed_items': 0},
          'error': 0,
          'message': 'No agent information was returned'},
 'msg': 'OK (164 bytes)',
 'pragma': 'no-cache',
 'redirected': False,
 'referrer_policy': 'no-referrer, strict-origin-when-cross-origin',
 'status': 200,
 'strict_transport_security': 'max-age=63072000; includeSubdomains',
 'url': 'https://localhost:55000/agents?select=id,name&q=manager=wazuh-worker2',
 'x_content_type_options': 'nosniff',
 'x_frame_options': 'DENY',
 'x_xss_protection': '1; mode=block'}
----------------------------------------------------------------------------------- Captured log teardown ------------------------------------------------------------------------------------
DEBUG    testinfra:base.py:286 RUN CommandResult(command=b"ansible --tree /tmp/tmp659h4jku -i /home/yanazaeva/git/wazuh-qa/tests/system/provisioning/basic_cluster/inventory.yml -m command --args 'service wazuh-agent restart' wazuh-agent3", exit_status=0, stdout=b'wazuh-agent3 | CHANGED | rc=0 >>\nKilling wazuh-modulesd... \nKilling wazuh-logcollector... \nKilling wazuh-syscheckd... \nKilling wazuh-agentd... \nKilling wazuh-execd... \nWazuh v4.3.0 Stopped\nStarting Wazuh v4.3.0...\nStarted wazuh-execd...\nStarted wazuh-agentd...\nStarted wazuh-syscheckd...\nStarted wazuh-logcollector...\nStarted wazuh-modulesd...\nCompleted.\n', stderr=b"[WARNING]: Consider using the service module rather than running 'service'.  If\nyou need to use command because service is insufficient you can add 'warn:\nfalse' to this command task or set 'command_warnings=False' in ansible.cfg to\nget rid of this message.\n")
INFO     testinfra:ansible.py:71 RUN Ansible('command', 'service wazuh-agent restart', {'check': False, 'become': False}): {'changed': True,
 'cmd': ['service', 'wazuh-agent', 'restart'],
 'delta': '0:00:07.733765',
 'end': '2021-12-02 12:21:20.503260',
 'rc': 0,
 'start': '2021-12-02 12:21:12.769495',
 'stderr': '',
 'stderr_lines': [],
 'stdout': 'Killing wazuh-modulesd... \n'
           'Killing wazuh-logcollector... \n'
           'Killing wazuh-syscheckd... \n'
           'Killing wazuh-agentd... \n'
           'Killing wazuh-execd... \n'
           'Wazuh v4.3.0 Stopped\n'
           'Starting Wazuh v4.3.0...\n'
           'Started wazuh-execd...\n'
           'Started wazuh-agentd...\n'
           'Started wazuh-syscheckd...\n'
           'Started wazuh-logcollector...\n'
           'Started wazuh-modulesd...\n'
           'Completed.',
 'stdout_lines': ['Killing wazuh-modulesd... ',
                  'Killing wazuh-logcollector... ',
                  'Killing wazuh-syscheckd... ',
                  'Killing wazuh-agentd... ',
                  'Killing wazuh-execd... ',
                  'Wazuh v4.3.0 Stopped',
                  'Starting Wazuh v4.3.0...',
                  'Started wazuh-execd...',
                  'Started wazuh-agentd...',
                  'Started wazuh-syscheckd...',
                  'Started wazuh-logcollector...',
                  'Started wazuh-modulesd...',
                  'Completed.'],
 'warnings': ['Consider using the service module rather than running '
              "'service'.  If you need to use command because service is "
              "insufficient you can add 'warn: false' to this command task or "
              "set 'command_warnings=False' in ansible.cfg to get rid of this "
              'message.']}
================================================================================== short test summary info ===================================================================================
FAILED ../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion - Failed: Could not find any agent reporting to wazuh-worker2: {'data': {'affec...
===================================================================================== 1 failed in 10.36s =====================================================================================

This error is solved by setting another sleep right before the agent is restarted, in order to give it time to connect to the worker again.

yanazaeva@pop-os:~/git/wazuh-qa/tests/system/provisioning/basic_cluster$ python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -vv
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, time-machine-2.4.1, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

=============================================================================== 1 passed in 144.23s (0:02:24) ================================================================================
yanazaeva@pop-os:~/git/wazuh-qa/tests/system/provisioning/basic_cluster$ python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -vv
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, time-machine-2.4.1, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

=============================================================================== 1 passed in 143.80s (0:02:23) ================================================================================
yanazaeva@pop-os:~/git/wazuh-qa/tests/system/provisioning/basic_cluster$ python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -vv
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, time-machine-2.4.1, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

=============================================================================== 1 passed in 143.97s (0:02:23) ================================================================================
yanazaeva@pop-os:~/git/wazuh-qa/tests/system/provisioning/basic_cluster$ python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -vv
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, time-machine-2.4.1, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

=============================================================================== 1 passed in 143.46s (0:02:23) ================================================================================
yanazaeva@pop-os:~/git/wazuh-qa/tests/system/provisioning/basic_cluster$ python3.9 -m pytest ../../test_cluster/test_agent_files_deletion/ -vv
/home/yanazaeva/.local/lib/python3.9/site-packages/_testinfra_renamed.py:5: DeprecationWarning: testinfra package has been renamed to pytest-testinfra. Please `pip install pytest-testinfra` and `pip uninstall testinfra` and update your package requirements to avoid this message
  warnings.warn((
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/yanazaeva/git/wazuh-qa/tests/system
plugins: aiohttp-0.3.0, time-machine-2.4.1, trio-0.7.0, testinfra-6.4.0, asyncio-0.16.0, cov-2.12.0, testinfra-6.0.0
collected 1 item                                                                                                                                                                             

../../test_cluster/test_agent_files_deletion/test_agent_files_deletion.py::test_agent_files_deletion PASSED                                                                            [100%]

=============================================================================== 1 passed in 145.40s (0:02:25) ================================================================================

@vikman90 vikman90 moved this to Triage in Release 4.3.0 Dec 3, 2021
@davidjiglesias davidjiglesias added this to the Release 4.3.0 RC 1 milestone Dec 9, 2021
@davidjiglesias davidjiglesias moved this from Triage to To do in Release 4.3.0 Dec 9, 2021
@davidjiglesias davidjiglesias moved this from To do to In progress in Release 4.3.0 Dec 10, 2021
@yanazaeva
Copy link
Contributor Author

Issue update


Execution nº Test name Pass XFail/Skip Fail Issues Ref. Status
1 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢
2 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢
3 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢

The results can be found here.

@yanazaeva
Copy link
Contributor Author

yanazaeva commented Dec 21, 2021

Issue update

Checking if the information was in the global.db database took too much time and there was no way to determine if that information was already in monitoring the logs. What we are doing instead is check the wdb socket, and see if the information is there. To do so, we will use the function query_wdb present in the file wazuh_db.py. This justifies these changes:

- name: "Clone wazuh-qa repository"
git:
repo: "https://github.com/wazuh/wazuh-qa"
dest: /wazuh-qa
version: "master"
depth: '1'

- name: "Install necessary dependencies"
command: /var/ossec/framework/python/bin/python3.9 -m pip install lockfile filetype certifi testinfra
- name: "Create a file to call the socket"
copy:
dest: "/send_msg.py"
content: |
import sys
sys.path.append('/wazuh-qa/deps/wazuh_testing')
from wazuh_testing import wazuh_db
result = wazuh_db.query_wdb(f'global sql select * from agent where id={sys.argv[1]}')
if result != []:
print(result)

These changes are present in both master and worker files as is checked in both of them.

The file messages.yml was added in order to monitor the logs we are interested in.

Also, this change was introduced in order to not end the test until the agent is restarted and active:

while True:
if int(host_manager.run_shell('wazuh-master',
f'{WAZUH_PATH}/bin/agent_control -l | grep Active | wc -l')) == 4:
break

Lastly, in the Ansible tasks, we set the wazuh-agent3 to be initialized before the wazuh-agent2, as if the third agent is started in the last place, we could face a problem of this agent being listed as never_connected. If we execute the test fast enough, it will fail as it will not find any agent connected to the second worker.

I will leave here the test results:

Execution nº Test name Pass XFail/Skip Fail Issues Ref. Status
1 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢
2 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢
3 tests/system/test_cluster/test_agent_enrollment/test_agent_file_deletion.py 1 0 0 🟢

The results can be found here.

Repository owner moved this from In progress to Done in Release 4.3.0 Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants