[Security Solution] Implement coverage overview dashboard API

[maximpn]

Addresses: #158238

Summary

This PR adds Coverage Overview API endpoint necessary to implement Coverage Overview Dashboard.

Details

The Coverage Overview API implementation is represented by one HTTP POST internal route /internal/detection_engine/rules/_coverage_overview hidden by a feature flag detectionsCoverageOverview. It returns response in the format defined in #159993.

Implementation is done in a quite simple way. It basically just fetches all the rules in chunks and adds them to appropriate MITRE ATT&CK category buckets depending on the assigned categories. The chunk size has been chosen to be 10000 as it's the default limit.

At the current stage the API doesn't handle available rules which means it doesn't return available rules in the response.

Sample response containing two rules looks like

{
  "coverage": {
    "TA001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"],
    "T001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"],
    "T001.001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"],
    "TA002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"],
    "T002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"],
    "T002.002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"],
  },
  "unmapped_rule_ids": [],
  "rules_data": {
    "e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Some rule", "activity": "disabled" },
    "e2f459f0-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Another rule", "activity": "enabled" },
  },
}

How to access the endpoint?

Make sure a feature detectionsCoverageOverview flag is set in config/kibana.dev.yml like

xpack.securitySolution.enableExperimental:
  - detectionsCoverageOverview

Then access the API via an HTTP client for example curl

• an empty filter

curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview

• filter by rule name

curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"search_term": "rule name"}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview

• filter by enabled rules

curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"activity": ["enabled"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview

• filter by prebuilt rules

curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"source": ["prebuilt"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview

Known problems

• Filtering by a tactic name doesn't guarantee the other tactics, techniques and sub-techniques will be filtered out. As one rule may be assigned to more than one tactic, technique and sub-technique filtering such rules by one tactic will lead to only rules assigned to the desired tactic be processed. But the result will include all the tactics, techniques and sub-techniques assigned to that rules.

UPD: leave as is for now

• Some of the implementation details are similar to find_rules endpoint. The difference is that find_rules accepts query parameter which is a KQL query while coverage_overview accepts filter fields and builds up a KQL query under the hood. Passing a prepared KQL query to coverage_overview looks too permissive and can lead to undesired filtering results. Some of KQL query building code is common and can be reused between FE and BE.

UPD: Solved

• One may ask why using an HTTP POST request instead of HTTP GET. In fact HTTP POST is needed only for convenience to send a JSON request query in the request body similar to GraphQL approach but it looks rather an overkill. One of the main reasons why HTTP POST is used is the limitation of io-ts schemas used to request query validation. It's handled by buildRouteValidation() which doesn't parse input parameters. For example there is a request with a query string /internal/detection_engine/rules/_coverage_overview?filter={"search_term": "rule 1"}, it's handled and the following object gets passed to buildRouteValidation()

{
  "filter": '{"search_term": "rule 1"}'
}

as you may notice '{"search_term": "rule 1"}' is a string so the io-ts schema validation fails while the request looks correct. In contrast a similar @kbn/config-schema schema used instead for the request query validation handles it correctly. As the reference it works here for internal/alerting/rules/_find endpoint, fields query parameter can be a JSON array and validation handles it correctly.

UPD: discussed with the team and decided that HTTP POST is more convenient for complex filters.

• During FTR tests implementation I've noticed the server fails if the second page (10000 - 20000 rules) is requested with an error

illegal_argument_exception: Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

There is a chance it can fail with the same error in prod.

UPD: The problem in reproducible in prod. To avoid a server crash the endpoint doesn't handle more than 10k rules. The problem will be addressed in #160698.

Posible improvements

• Move KQL utility functions into common folder to be shared between FE and BE (done)
• Implement stricter filtering to return only searched tactic, technique and sub-technique (leave as is for now)
• Use HTTP GET instead of HTTP POST (discussed with the team and decided that HTTP POST is more convenient for complex filters)
• Make sure pages above 10000 rules are handled properly (will be addresses in [Security Solution] Support more than 10k rules on Coverage Overview Dashboard #160698)

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenario

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[xcrzx]

Thanks for your implementation, @maximpn! 🎉 I've tested the PR locally and the expected path works as it should. However, I noticed that Kibana crashes when there are over 10,000 detection rules. I think we should address this issue as it appears quite critical. The overall implementation seems sound, with my suggestions primarily relating to reducing code duplication.

[maximpn]

@xcrzx thank you for the thorough review 🙏 I've addressed all your comments and the PR is ready for the second review.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 45423a7

Failed CI Steps

• Security Solution Tests #6

Test Failures

• [job] [logs] Security Solution Tests #6 / Detections : Page Filters Alert Page Filters Customization should be able to delete Controls
• [job] [logs] Security Solution Tests #6 / Exceptions viewer read only "before each" hook for "Cannot add an exception from empty viewer screen"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	4229	4230	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.0MB	11.0MB	+599.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	14	16	+2
securitySolution	413	418	+5
total			+7

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	15	17	+2
securitySolution	492	497	+5
total			+7

History

• 💔 Build #138768 failed 9af1945
• 💛 Build #137938 was flaky 5a82ce1a1d5e43e22c802c1e9a3a5ea1e842e16c
• 💔 Build #137928 failed c5f7f7a9baa8e79966c48dc1d84556055721852c
• 💔 Build #137916 failed 8e2423356e956e205563e60465ceaccf98eabcd8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn

[xcrzx]

Thank you for addressing my comments and the additional cleanup, @maximpn 👍

Just a side note - we've discussed how we might inform users that the returned data is capped at 10,000 rules. If they have more than 10,000 rules, they'll need to narrow down their search request to view complete coverage. So we might need to return the total count of matched rules from the API. However, this seems like a minor enhancement that can be implemented at a later stage.