Determine VDT System tests tiers, preconditions, implications and expectations

[Deblintrake09]

Target version	Related issue
4.8.0 Alpha 1	#4369

Description

This Issue aims to add detail to the test cases proposed in #4369. The goal is to determine tiers, preconditions, considerations, and expectations of each test case.

Tasks

• Assign tiers to test cases
• Determine test preconditions
• Determine results expectations and implications (Security, Performance, etc).
• Identify the dependencies among test cases.

[Deblintrake09]

Assign tiers for test cases

All tests cases are considered Tier 0 in terms of importance for development. This cases are all considered the minimum test cases to develop. This tiers are to the define the complexity and how common is their prevalence in day to day usage of the application.

Tier 0 - Basic Cases

• First syscollector scan (Rsync)
• Subsequent scan (Dbsync) without any installation

Tier 1 - Common usage

• Installation of a vulnerable package
• Updating a vulnerable package that remains vulnerable to the same CVE
• Updating a vulnerable package that becomes vulnerable to another CVE
• Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one
• Updating a vulnerable package that ceases to be vulnerable
• Deleting a vulnerable package
• Installation of a non-vulnerable package
• Updating a non-vulnerable package that becomes vulnerable
• Updating a non-vulnerable package that remains non-vulnerable
• Deleting a non-vulnerable package
• Updating the vulnerability feed with a new applicable vulnerability
• Updating the vulnerability feed with a new non-applicable vulnerability
• Updating the vulnerability feed removes a CVE and causes a package to cease to be vulnerable

Tier 2 - Non common cases

• Deleting the states index

[Deblintrake09]

Determine test preconditions and expected results

Tier 0 - Basic Cases

First syscollector scan (Rsync)

Scenario: First syscollector scan (Rsync)
   Given the Manager has updated the feeds and a new Agent is connected
   When the Agent's syscollector scan is run
   Then Vulnerability alerts are generated as "added" And the Agent's System states are stored

Subsequent scan (Dbsync) without any installation

Scenario: Subsequent scan (Dbsync) without any installation
   Given the Manager has an agent connected
   And a first Syscollector scan has been executed
   And no changes have been done in the agent
   When a second Syscollector Scan is started on the agent
   Then Vulnerability the agent System states and vulnerability alerts remain unchanged

Tier 1 - Common usage cases

Installation of a vulnerable package

Scenario: Installation of a vulnerable package
   Given the Manager has an agent connected
   And a first Syscollector Scan has been executed
   When a vulnerable package is installed in the agent 
   And the Agent's syscollector scan is run
   Then new entry for the package appears in the Agent's System states
   And a vulnerability "added" alert is generated

Updating a vulnerable package that remains vulnerable to the same CVE

Scenario: Updating a vulnerable package that remains vulnerable to the same CVE
   Given the Agent has a vulnerable package installed
   And a vulnerability alert has been already generated for it
   When the vulnerable package is updated to a version that is still vulnerable to the same CVE
   Then an alert is generated for the new version of the package being vulnerable
   And no "solved" alert is generated for the old version of the package

Updating a vulnerable package that becomes vulnerable to another CVE

Scenario: Updating a vulnerable package that becomes vulnerable to another CVE
   Given the Agent has a vulnerable package installed
   And a vulnerability alert has been already generated for the CVE
   When the vulnerable package is updated to a version that is vulnerable to a new CVE
   And the package is not vulnerable to the old CVE
   Then an "added" alert is generated for the new CVE for the new version of the package
   And a "solved" alert is generated for the old version of the package for the old CVE

Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one

Scenario: Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one
   Given the Agent has a vulnerable package installed
   And a vulnerability alert has been already generated for the CVE
   When the vulnerable package is updated to a version that is also vulnerable to a new CVE
   And the package is still vulnerable to the old CVE
   Then an alert is generated for the new CVE and for the old CVE for the new version of the package
   And no "solved" alert is generated for the old version of the package for the old CVE

Updating a vulnerable package that ceases to be vulnerable

Scenario: Updating a vulnerable package that ceases to be vulnerable
   Given the Agent has a vulnerable package installed
   And a vulnerability alert has been already generated for the CVE
   When the vulnerable package is updated to a version that no longer vulnerable to CVEs
   Then a "solved" alert is generated for the CVE
    And no other alerts are generated for the new package version.

Deleting a vulnerable package

Scenario: Deleting a vulnerable package
   Given the Agent has a vulnerable package installed
   And a vulnerability alert has been already generated for the CVE
   When the vulnerable package removed from the system
   Then a "solved" alert is generated for the CVE
    And the package is removed from the System states information.

Installation of a non-vulnerable package

Scenario: Installation of a non-vulnerable package
   Given the Manager has an agent connected
   And a first Vulnerability Scan has been executed
   When a package that is not vulnerable to any CVEs is installed in the agent 
   And the Agent's syscollector scan is run
   Then new entry for the package appears in the Agent's System states
   And a no vulnerability "added" alerts are generated for the package

Updating a non-vulnerable package that becomes vulnerable

Scenario: Updating a non-vulnerable package that becomes vulnerable
   Given the Manager has an agent connected that has a non-vulnerable package installed
   And the package has already been scanned
   And no alerts have been generated for the package
   When the package is updated to a version that is vulnerable to a CVE
   And the Agent's syscollector scan is run
   Then a vulnerability "added" alert is generated for the package

Updating of a non-vulnerable package that remains non-vulnerable

Scenario: Updating of a non-vulnerable package that remains non-vulnerable
   Given the Manager has an agent connected that has a non-vulnerable package installed
   And the package has already been scanned
   And no alerts have been generated for the package
   When the package is updated to a version that is not vulnerable to any CVE
   And the Agent's syscollector scan is run
   Then no vulnerability "added" alerts are generated for the package

Deletion of a non-vulnerable package

Scenario: Deletion of a non-vulnerable package
   Given the Manager has an agent connected that has a non-vulnerable package installed
   And the package has already been scanned
   And no alerts have been generated for the package
   When the package is removed
   And the Agent's syscollector scan is run
   Then no vulnerability "added" alerts are generated for the package
   And the package is removed from the System states information

Updating the vulnerability feed with a new applicable vulnerability

Scenario: Updating the vulnerability feed with a new applicable vulnerability
   Given the Manager has an agent connected that has a non-vulnerable package installed
   And the package has already been scanned
   And no alerts have been generated for the package
   When the vulnerability feed is updated containing a new applicable vulnerability
   Then a vulnerability "added" alert is generated for the package

Updating the vulnerability feed with a new non-applicable vulnerability

Scenario: Updating the vulnerability feed with a new non-applicable vulnerability
   Given the Manager has an agent connected that has a non-vulnerable package installed
   And the package has already been scanned
   And no alerts have been generated for the package
   When the vulnerability feed is updated containing a new non-applicable vulnerability
   Then no vulnerability "added" alert is generated for the package

Updating the vulnerability feed removes a CVE and causes package to cease to be vulnerable

Scenario: Updating the vulnerability feed removes a CVE and causes package to cease to be vulnerable
   Given the Manager has an agent connected with a vulnerable package
   And an alert has been generated for the corresponding CVE
   When the feed is updated and the CVE for the vulnerable package is removed
   Then a `vulnerability solved` alert is generated.

Note: this simulates a CVE being discarded. Will require a mocked or modified feed.

Tier 2 - Non common cases

Deleting the states index

Scenario: Deleting the states index
   Given the Manager has an agent connected that has it's system data indexed
   When the user deletes the `state index` 
   And a Syscollector scan is executed in the agent
   Then the state index is regenerated and new alerts for the corresponding vulnerabilities are generated.

[Deblintrake09]

Performance considerations

• Tests that require the feeds to download will take more time over time, as the feeds get larger. As real feeds are to be used, this should be taken into account.
• Tests that require the install or upgrade of a given package, will need to take into account the size of the packages for download, installation times and the time it takes between syscollector scans, for alerts to be generated and indexed.
  • It is important that the package's to be used are stored in S3 storage so the tests are not dependent on other companies' servers.

Security implications

• It is important that as actual vulnerable packages will be installed, it is important to verify that the possible exploitabilities are protected in the endpoints used.

[Rebits]

Tests Dependencies

In the context of Performance considerations, it is crucial to identify potential dependencies between tests to minimize redundant and time-consuming operations. Specifically, the primary goal here is to mitigate the need for unnecessary package installations and removals, as these are the most time-intensive aspects of the tasks.

Because of constraints related to installing multiple packages simultaneously, we've opted not to pursue a parallel approach for test cases. Instead, we will leverage parallelism solely for launching distinct use cases across the agents

Note
The proposed tests dependency is an optimal approach that could vary depending of the available vulnerabilities found for the proposed OS

Tests Codes

• E2E-VD-1: First syscollector scan (Rsync)
• E2E-VD-2: Subsequent scan (Dbsync) without any installation
• E2E-VD-3: Installation of a vulnerable package
• E2E-VD-4: Updating a vulnerable package that remains vulnerable to the same CVE
• E2E-VD-5: Updating a vulnerable package that becomes vulnerable to another
• E2E-VD-6: Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one
• E2E-VD-7: Updating a vulnerable package that ceases to be vulnerable
• E2E-VD-8: Deleting a vulnerable package
• E2E-VD-9: Installation of a non-vulnerable package
• E2E-VD-10: Updating a non-vulnerable package that remains non-vulnerable
• E2E-VD-11: Updating a non-vulnerable package that becomes vulnerable
• E2E-VD-12: Deleting a non-vulnerable package
• E2E-VD-13: Updating the vulnerability feed with a new applicable vulnerability
• E2E-VD-14: Updating the vulnerability feed with a new non-applicable vulnerability
• E2E-VD-15: Deleting the states index
• E2E-VD-16: Updating the vulnerability feed removes a CVE and causes a package to cease to be vulnerable

Dependency diagram

flowchart TD
    O[E2E-VD-1] -->  P[E2E-VD-2]
    P[E2E-VD-2] -->  L[E2E-VD-3]
    P[E2E-VD-2] -->  S[E2E-VD-9]
    P[E2E-VD-2] --> K[E2E-VD-15]
    L[E2E-VD-3] --> V[E2E-VD-16]
    L[E2E-VD-3] --> R[E2E-VD-4]
    R[E2E-VD-4] --> R2[E2E-VD-5]
    R2[E2E-VD-5] --> R3[E2E-VD-6]
    R3[E2E-VD-6] --> R4[E2E-VD-7]


    R4[E2E-VD-7] --> R5[E2E-VD-11]
    R5[E2E-VD-11] --> R7[E2E-VD-12]

    S[E2E-VD-9] --> R9[E2E-VD-10]
    R9[E2E-VD-10] --> R10[E2E-VD-8]
    
    S[E2E-VD-9]  --> S1[E2E-VD-13]
    S[E2E-VD-9]  --> S2[E2E-VD-14]

Loading

[davidjiglesias]

LGTM!