Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Implement coverage overview dashboard API (#160480)
**Addresses:** #158238 ## Summary This PR adds Coverage Overview API endpoint necessary to implement Coverage Overview Dashboard. ## Details The Coverage Overview API implementation is represented by one HTTP POST internal route `/internal/detection_engine/rules/_coverage_overview` hidden by a feature flag `detectionsCoverageOverview`. It returns response in the format defined in #159993. Implementation is done in a quite simple way. It basically just fetches all the rules in chunks and adds them to appropriate MITRE ATT&CK category buckets depending on the assigned categories. The chunk size has been chosen to be `10000` as it's the default limit. At the current stage the API doesn't handle available rules which means it doesn't return available rules in the response. Sample response containing two rules looks like ```json { "coverage": { "TA001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "T001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "T001.001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "TA002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], "T002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], "T002.002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], }, "unmapped_rule_ids": [], "rules_data": { "e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Some rule", "activity": "disabled" }, "e2f459f0-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Another rule", "activity": "enabled" }, }, } ``` ### How to access the endpoint? Make sure a feature `detectionsCoverageOverview` flag is set in `config/kibana.dev.yml` like ```yaml xpack.securitySolution.enableExperimental: - detectionsCoverageOverview ``` Then access the API via an HTTP client for example `curl` - an empty filter ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by rule name ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"search_term": "rule name"}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by enabled rules ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"activity": ["enabled"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by prebuilt rules ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"source": ["prebuilt"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` ## Known problems - <del>Filtering by a tactic name doesn't guarantee the other tactics, techniques and sub-techniques will be filtered out. As one rule may be assigned to more than one tactic, technique and sub-technique filtering such rules by one tactic will lead to only rules assigned to the desired tactic be processed. But the result will include all the tactics, techniques and sub-techniques assigned to that rules.</del> UPD: leave as is for now - <del>Some of the implementation details are similar to `find_rules` endpoint. The difference is that `find_rules` accepts `query` parameter which is a KQL query while `coverage_overview` accepts filter fields and builds up a KQL query under the hood. Passing a prepared KQL query to `coverage_overview` looks too permissive and can lead to undesired filtering results. Some of KQL query building code is common and can be reused between FE and BE.</del> UPD: Solved - <del>One may ask why using an HTTP POST request instead of HTTP GET. In fact HTTP POST is needed only for convenience to send a JSON request query in the request body similar to GraphQL approach but it looks rather an overkill. One of the main reasons why HTTP POST is used is the limitation of `io-ts` schemas used to request query validation. It's handled by `buildRouteValidation()` which doesn't parse input parameters. For example there is a request with a query string `/internal/detection_engine/rules/_coverage_overview?filter={"search_term": "rule 1"}`, it's handled and the following object gets passed to `buildRouteValidation()`</del> ```ts { "filter": '{"search_term": "rule 1"}' } ``` <del>as you may notice `'{"search_term": "rule 1"}'` is a string so the `io-ts` schema validation fails while the request looks correct. In contrast a similar `@kbn/config-schema` schema used instead for the request query validation handles it correctly. As the reference it works [here](https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/routes/find_rules.ts#L100C16-L100C27) for `internal/alerting/rules/_find` endpoint, `fields` query parameter can be a JSON array and validation handles it correctly.</del> UPD: discussed with the team and decided that HTTP POST is more convenient for complex filters. - During FTR tests implementation I've noticed the server fails if the second page (10000 - 20000 rules) is requested with an error ``` illegal_argument_exception: Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. ``` There is a chance it can fail with the same error in prod. UPD: The problem in reproducible in prod. To avoid a server crash the endpoint doesn't handle more than 10k rules. The problem will be addressed in #160698. ## Posible improvements - [x] Move KQL utility functions into common folder to be shared between FE and BE (done) - Implement stricter filtering to return only searched tactic, technique and sub-technique (leave as is for now) - Use HTTP GET instead of HTTP POST (discussed with the team and decided that HTTP POST is more convenient for complex filters) - Make sure pages above 10000 rules are handled properly (will be addresses in #160698) ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenario
- Loading branch information
