Designing a Comprehensive Testing Environment for Vulnerability Detector Feature Testing

[Rebits]

Description

To ensure effective testing of the Vulnerability Detector Feature, it is imperative to define the specific testing environments in which the evaluations will take place. The proposed environment should provide comprehensive information in the following key areas:

Indexers:

• Determine whether a single-node or multi-node indexer setup will be used.
• Clarify whether the indexer will reside on the same node as the master host.
• Specify the available resources (CPU, RAM, etc.) for the indexer.
• Indicate the operating system (OS) that the indexer is running on.

Managers:

• Specify whether a single-node or multi-node manager configuration will be employed.
• Identify the operating system(s) utilized by the manager(s).
• Clearly define the available resources for the manager(s) (CPU, RAM, etc.).
• Specify the OS of the manager(s).

Agents:

• Detail the operating system(s) on which the agents will be installed.
• Specify the resources allocated to each agent (CPU, RAM, etc.).
• If using a multi-node manager setup, describe the topology of manager-agent connections.

Note
Ensure that the operating systems used for agents include the initial coverage OS as proposed in testing issue.

The proposed testing scenario will be evaluated based on its cost-effectiveness and its impact on overall performance of the tests.

The final description of the testing environment should be presented in YAML format, adhering to the requirements outlined in the Wazuh QA Environment pipeline. This YAML should be included in the provision/environments wazuh qa repository path.

Lastly, it is highly recommended to conduct a Proof of Concept deployment of the proposed environment in the Wazuh QA Environment pipeline to validate its feasibility and functionality in a real-world scenario.

[Rebits]

It is proposed the following environment

manager1:
    roles: [manager, filebeat, indexer]
    os: ubuntu_22
    type: master

manager2:
    roles: [manager, filebeat]
    os: LINUX_BASED_OS
    type: worker

manager3:
    roles: [manager, filebeat]
    os: LINUX_BASED_OS
    type: worker

agent1:
    roles: [agent]
    os: windows_11
    manager: manager2
    resources: 
        cpu: 2
        memory: 2048
agent2:
    roles: [agent]
    os: windows_2022
    manager: manager2
    resources: 
        cpu: 2
        memory: 2048
agent3:
    roles: [agent]
    os: macos_ventura
    manager: manager3
    resources: 
        cpu: 2
        memory: 2048

agent4:
    roles: [agent]
    os: centOS_7
    manager: manager1
    resources: 
        cpu: 2
        memory: 2048

agent5:
    roles: [agent]
    os: ubuntu_22
    manager: manager2
    resources: 
        cpu: 2
        memory: 2048

Build: https://ci.wazuh.info/job/Wazuh_QA_environment/367/parameters/

[Rebits]

Is necessary to confirm infrastructure requirements with the management team. Confirming if it is necessary a multi-node approach the the recommended resources.

[Rebits]

Confirmed multinode requirement #4369 (comment)

[Rebits]

New proposed environment:

manager1:
    roles: [manager, filebeat, indexer, qa_framework]
    os: ubuntu_22
    type: master

manager2:
    roles: [manager, filebeat, qa_framework]
    os: LINUX_BASED_OS
    type: worker

agent1:
    roles: [agent, qa_framework]
    os: centos_7
    manager: manager1

agent2:
    roles: [agent, qa_framework]
    os: windows_11
    manager: manager2

agent3:
    roles: [agent, qa_framework]
    os: macos_sonoma
    manager: manager2

agent3:
    roles: [agent, qa_framework]
    os: ubuntu_22
    manager: manager1

We have removed one worker regarding #4369 (comment).
Furthermore, we've added the qa_framework role to provide access to the framework-defined functions, enabling a more streamlined workflow.

[Rebits]

QA Frameowork role does not work for Windows and macOS endpoints.

[Rebits]

The designated testing environment for the System Vulnerability Detector has been finalized and is as follows:

manager1:
    roles: [manager, filebeat, indexer, qa_framework]
    os: ubuntu_22
    type: master

manager2:
    roles: [manager, filebeat, qa_framework]
    os: LINUX_BASED_OS
    type: worker

agent1:
    roles: [agent, qa_framework]
    os: centos_7
    manager: manager1
    architecture: amd64

agent2:
    roles: [agent, qa_framework]
    os: centos_7
    manager: manager1
    architecture: arm

agent3:
    roles: [agent, qa_framework]
    os: windows_11
    manager: manager2

agent4:
    roles: [agent, qa_framework]
    os: macos_sonoma
    manager: manager2
    architecture: arm

agent5:
    roles: [agent, qa_framework]
    os: macos_sonoma
    manager: manager2
    architecture: intel

agent6:
    roles: [agent, qa_framework]
    os: ubuntu_22
    manager: manager1
    architecture: amd64

agent7:
    roles: [agent, qa_framework]
    os: ubuntu_22
    manager: manager1
    architecture: arm

It has been removed the qa_framework role due to this is not necessary using the current approach of the E2E tests

[damarisg]

@Rebits, I have some doubts:

• By default, what resources are used by the administrator and the agent?
• What operating system is LINUX_BASED_OS? Is it a random list?
• Why do we have CUSTOM_PARAMETERS? Why don't we have fields for REVISION, REPOSITORY and VERSION?
• ENVIRONMENT_DATA must contain the architecture. Why is it not added?

[Rebits]

Regarding this comment, it's crucial to note that the Wazuh QA Environment pipeline lacks documentation. We currently have an open issue to address this deficiency: https://github.com/wazuh/wazuh-jenkins/issues/3821. In this comment, we aim to succinctly address some of the questions raised:

By default, what resources are used by the administrator and the agent?

The default resource allocation is based on roles. Each role specifies a minimum set of resources required, and instances can use these or more if specified. The complete role list can be found here: https://github.com/wazuh/wazuh-qa/blob/master/provisioning/roles.yaml. If no role is provided, the default service resources, defined in DeploymentData, are used:

static final Map DEFAULT_RESOURCES = [
  'EC2': [cpu: 2, memory: 4096],
  'ECS': [cpu: 1, memory: 2048],
  'vagrant': [cpu: 2, memory: 6144]
]

What operating system is LINUX_BASED_OS? Is it a random list?

LINUX_BASED_OS Placeholder: LINUX_BASED_OS serves as a placeholder, indicating that any Linux-based instance is suitable for deployment. The specific instance chosen depends on the assigned roles. LINUX_BASED_OS is defined as ['ubuntu', 'centos', 'amazonlinux', 'debian']. For example, if you wish to provision the custom_role as follows:

Environment

test:
    roles: [custom_role]
    os: LINUX_BASED_OS

Roles

custom_role:
    playbook: ./playbooks/apache.yaml
    os: [centos, ubuntu]
    service: ANY

The Wazuh QA Environment will analyze the custom_role requirements and select either centos or ubuntu based on the order of definition.
In our case it is equivalent to use the ubuntu_22 AMI.

Why do we have CUSTOM_PARAMETERS? Why don't we have fields for REVISION, REPOSITORY and VERSION?

CUSTOM_PARAMETERS is used to define global parameters that apply to all instances. For instance, you can specify package data through this variable. You can customize these parameters for each host. For example, if you want to deploy a Wazuh manager version 4.6.0 and a Wazuh agent version 4.5.3, you can define the environment as:

Environment

agent:
    roles: [agent]
    os: LINUX_BASED_OS
    version: 4.5.3
    revision: 1
    repository: live
manager:
    roles: [manager]
    os: LINUX_BASED_OS
`` ` 

With CUSTOM_PARAMETERS:
```yaml
version: 4.6.0
revision: 1
repository: live

ENVIRONMENT_DATA must contain the architecture. Why is it not added?

It is not included globally in ENVIRONMENT_DATA because it is intended to be specified for each host individually. Currently, the environment design does not support multiple architectures. For more details, refer to https://github.com/wazuh/wazuh-jenkins/issues/5774.

Feel free to ask if you have any further questions or require additional clarification.

[davidjiglesias]

LGTM!