Feature: validation of detections against cms_main

[cmcginley-splunk]

Context

• We need to validate that our detections not only work in ES, but are also being parsed correctly by the content versioning service

Code change

• Added ContentVersioningService
  • Enables content versioning
  • Activates CMS parser
  • Validates detections against CMS events, reports any missing detections or other validation issues. Validations include:
    • Counts between cms_main and detections based on YAMLs match
    • Each detection can be matched to a cms_event
    • Each cms_event can be matched to a detection
    • No cms_event is repeated
    • UUIDs match between cms/detection pairs
    • Versions match between cms/detection pairs
    • Full search name matches between cms/detection pairs
    • Correlationsearch label matches between cms/detection pairs
  • Returns a list of setup functions which are optionally inserted into the main setup of DetectionTestingInfrastructure
• Aggregating errors across all threads (setup, testing, view, view shutdown) and reporting en masse
• Broke up setup functions list to allow for optional inclusion of the content versioning setup functions
• NOTE: Now we raise any exceptions that crop up during setup, instead of just logging them
• Added logic to DetectionTestingInfrastructure to query installed ES version, so we can determine when it is appropriate to run content versioning validation
  • Content versioning validation is only run when 1) integration testing is enabled, 2) ES 8.0.0+ is installed, and 3) when the testing mode is All
• Moved logging utility in CorrelationSearch (useful for debugging) to helper.utils.Utils so it could be re-used in ContentVersioningService

Testing

• I've tested locally w/ some quick and dirty edits to the savedsearches.conf file and all seems to work well
• Ran against escu-production and all results passed (see attached truncated log)

content_versioning_service.log

Caveats

• Note that we now raise exceptions from setup, instead of logging and dropping them; I believe this is desirable
• I went back and forth as to whether this validation should happen as part of setup, OR if I should create some new type of test case. I opted for the former as this type of testing doesn't fit our current testing paradigm; validation against the cms_main index should happen in aggregate, a test of the whole package, not individual detections. We do not have an easy way to insert testing at that level presently.
• Additionally, this validation probably only needs to be run against a single instance and not necessarily all instances in a single run, but I don't see an easy or clean way to do that as well, and there's no harm or impact on execution time as far as I can tell

TODOs

• disable logging
• remove TODOs or create follow up issues
• Test in CI when able

[pyth0n1c]

Very minor typo here. I assume it is Unable to determine versioning status instead of unable to versioning status?

[cmcginley-splunk]

Good catch!

[pyth0n1c]

The is probably a typo in here.
Can you give alittle more context about why this function exists? I think this is supposed to say something like

Force the cms_parser to run by disabling and re-enabling it. This is a hack to get the cms_parser to run on-demand

[cmcginley-splunk]

Another good catch haha. not sure how that got so garbled lol

[pyth0n1c]

I wonder if it's better to go in reverse now that we have the following function to generate a search name:

contentctl/contentctl/objects/abstract_security_content_objects/detection_abstract.py

Line 101 in 98573b0

def get_action_dot_correlationsearch_dot_label(self, app:CustomApp, max_stanza_length:int=ES_MAX_STANZA_LENGTH)->str:

This also means if we change the template used in the link above to generate the detection name, we don't have to change it again in this function.

[cmcginley-splunk]

Ahhhhh I don't think I knew about this function, I'll take a look!

[pyth0n1c]

This one is FAIRLY new. In the future, it might be a nice idea to actually associate the app config with a detection so that we don't have to carry/pass it around, making action_dot_correlationsearch_dot_label a cached_property instead of a function that takes an argument.

[pyth0n1c]

See a similar comment above about how we compose the name of the detection.