QACTL: Validate configuration file input parameters

[jmv74211]

qactl tool receives a config file that contains all data parameters for deployment, provisioning, testing...

It is necessary that before launching all these processes, the content and structure for this config file are validated.

It is requested to add a new initial stage in qactl that validates all input user parameters and their structure.

Tasks

In order to achieve this new initial stage, several tasks need to be done:

• Read jsonschema documentation
• Review JSON schema validator file
• Implement a validator function to check the conf.yaml parameters
• Review qa_ctl script code
• Modify & improve JSON schema validator file
• Create a new schema field for the test module
• Check the functionality of the new function

[antoniomanuelfr]

We have been discussing how are we going to validate the configuration file for QACTL. The best solution will be to add a validation phase in the qa-ctl tool that will use a validator tool, filtering problems like writing numbers in fields where a string is expected.
We have two options:

• Use JSON schema (https://json-schema.org/) for python.
• Use Cerberus (https://docs.python-cerberus.org) to validate the loaded dictionary.

We are going to start by using the JSON schema solution because this will avoid an extra dependency (the JSON schema is already installed to check that FIM produces the events in a valid JSON format). If we encounter any problem, we will think about using Cerberus or another tool like it.

[antoniomanuelfr]

working branch https://github.com/wazuh/wazuh-qa/tree/1711-qactl-validation

[jmv74211]

@fernandolojano will continue with this issue development

[fernandolojano]

During the first approach in implementing the new function and checking it's proper functionality, a few issues were found in the JSON schema file used as validator.

• There wasn't a proper object declared into the test module, this caused the failure of the validator function everytime we added the test module into the conf.yaml file.
• There were some format parameters inconsistencies between the schema file and the conf.yaml file.

After solving this problems and reviewing the validator function's code, It seems that the new implementation is working fine.

However, in order to definitely confirm that the new script is working correctly, it is necessary to make a deep testing of all the possible error cases. We need to check and improve the json schema file configuration as well.

[fernandolojano]

Proven error case scenarios

Deployment module

Vagrant scenario

Error Case	Function's output
No main entry given for deployment module (host1)	Error KeyError: 'provider'
No provider entry given (provider)	Error KeyError: 'provider'
No required entry given (enabled)	ValidationError: 'enabled' is a required property
No required entry parameter given (vagrantfile_path)	ValidationError: 'vagrantfile_path' is a required property
Incorrect format entry given (vm_memory: '2048')	ValidationError: '2048' is not of type 'integer'

Docker scenario

Error Case	Function's output
Empty required field (dockerfile_path)	ValidationError: None is not of type 'string'
Empty optional field (IP)	ValidationError: None is not of type 'string'
Wrong format argument given (stdout : 'True')	ValidationError: 'True' is not of type 'boolean'
No entries given for field (ports)	ValidationError: 'host_port' is a required property
Wrong format argument given(binded_port: 2222)	ValidationError: 2222 is not of type 'string'
Empty required parameter for ports field (host_port)	ValidationError: None is not of type 'integer'

Provision module

Host info section

Error Case	Function's output
Undeclared required field (host_info)	ValidationError: 'host_info' is a required property
Empty required field (connection_method)	ValidationError: None is not of type 'string'
Wrong format argument given(password: 12345)	ValidationError: 13245 is not of type 'string'
Wrong format argument given(connection_port: '22')	ValidationError: '22' is not of type 'integer'

Wazuh deployment section

Error Case	Function's output
Wrong argument for field(type:"source")	ValidationError: 'source' is not one of ['sources', 'package']
No required entry parameter given(target)	ValidationError: 'target' is a required property
No conditional required entry parameter given(local_package_path when type: package)	alidationError: 'local_package_path' is a required property
No conditional required entry parameter given(wazuh_branch when type: sources)	ValidationError: 'wazuh_branch' is a required property
No conditional required entry parameter given( manager_ip when target: agent)	ValidationError: 'manager_ip' is a required property

QA framework section

Error Case	Function's output
Wrong format argument for field(wazuh_qa_branch: 1123)	ValidationError: 1123 is not of type 'string'
No required entry parameter given (qa_workdir)	ValidationError: 'qa_workdir' is a required property

Testing module

Host info section

Error Case	Function's output
No password parameter given	ValidationError: 'ssh_private_key_file_path' is a required property
Undeclared required field (host_info)	ValidationError: 'host_info' is a required property
Empty required field (connection_method)	ValidationError: None is not of type 'string'
Wrong format argument given(password: 12345)	ValidationError: 13245 is not of type 'string'
Wrong format argument given(connection_port: '22')	ValidationError: '22' is not of type 'integer'

Test section

Error Case	Function's output
Wrong format argument given(hosts: [10.2.0.12])	ValidationError: ['10.2.0.12'] is not of type 'string'
No required parameter given(test_files_path)	ValidationError: 'test_files_path' is a required property
Empty entry for required parameter(run_tests_dir_path)	ValidationError: None is not of type 'string'
Wrong format argument given(stop_after_first_failure: "False")	ValidationError: 'False' is not of type 'boolean'

[fernandolojano]

After an exhaustive study of most possible error cases for the validation function, it is safe to confirm that the function is working properly for the deployment and provisioning modules of the conf.yaml file. It still remains to check it's proper working functionality for the testing module.

Some minor issues as new parameter inconsistencies between the schema file and the conf.yaml where found again while checking the JSON schema file.

New modifications into the schema were added as well:

• The ports field of the docker provider of the Deployment section was declared as an object type but didn't have any properties.
• It was necessary to create conditional requirements in case of specific parameters entries for type and target fields of the Provisioning module

All this new additions can be seen and reviewed in this commit

As a new requirement it was necessary to add a new field for the test module. This new field is similar to the actual host_info field given in the Provisioning module. See this changes.