[Security Solution] Implement an internal API endpoint to serve Protections/Detections Coverage Overview dashboard’s data #158238
Assignees
Labels
8.9 candidate
Feature:Rule Management
Security Solution Detection Rule Management area
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
v8.9.0
Comments
maximpn
added
Team:Detections and Resp
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
This was referenced May 23, 2023
Closed
Open
Closed
maximpn
changed the title
maximpn
added
the
Feature:Rule Management
3 tasks
maximpn
added a commit
that referenced
this issue
Jun 29, 2023
**Addresses:** #158238 ## Summary This PR adds Coverage Overview API endpoint necessary to implement Coverage Overview Dashboard. ## Details The Coverage Overview API implementation is represented by one HTTP POST internal route `/internal/detection_engine/rules/_coverage_overview` hidden by a feature flag `detectionsCoverageOverview`. It returns response in the format defined in #159993. Implementation is done in a quite simple way. It basically just fetches all the rules in chunks and adds them to appropriate MITRE ATT&CK category buckets depending on the assigned categories. The chunk size has been chosen to be `10000` as it's the default limit. At the current stage the API doesn't handle available rules which means it doesn't return available rules in the response. Sample response containing two rules looks like ```json { "coverage": { "TA001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "T001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "T001.001": ["e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d"], "TA002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], "T002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], "T002.002": ["e2f459f0-12d6-11ee-a0ab-c95a1fc4921d"], }, "unmapped_rule_ids": [], "rules_data": { "e2c9ee90-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Some rule", "activity": "disabled" }, "e2f459f0-12d6-11ee-a0ab-c95a1fc4921d": { "name": "Another rule", "activity": "enabled" }, }, } ``` ### How to access the endpoint? Make sure a feature `detectionsCoverageOverview` flag is set in `config/kibana.dev.yml` like ```yaml xpack.securitySolution.enableExperimental: - detectionsCoverageOverview ``` Then access the API via an HTTP client for example `curl` - an empty filter ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by rule name ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"search_term": "rule name"}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by enabled rules ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"activity": ["enabled"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` - filter by prebuilt rules ```sh curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -d '{"filter":{"source": ["prebuilt"]}}' http://localhost:5601/kbn/internal/detection_engine/rules/_coverage_overview ``` ## Known problems - <del>Filtering by a tactic name doesn't guarantee the other tactics, techniques and sub-techniques will be filtered out. As one rule may be assigned to more than one tactic, technique and sub-technique filtering such rules by one tactic will lead to only rules assigned to the desired tactic be processed. But the result will include all the tactics, techniques and sub-techniques assigned to that rules.</del> UPD: leave as is for now - <del>Some of the implementation details are similar to `find_rules` endpoint. The difference is that `find_rules` accepts `query` parameter which is a KQL query while `coverage_overview` accepts filter fields and builds up a KQL query under the hood. Passing a prepared KQL query to `coverage_overview` looks too permissive and can lead to undesired filtering results. Some of KQL query building code is common and can be reused between FE and BE.</del> UPD: Solved - <del>One may ask why using an HTTP POST request instead of HTTP GET. In fact HTTP POST is needed only for convenience to send a JSON request query in the request body similar to GraphQL approach but it looks rather an overkill. One of the main reasons why HTTP POST is used is the limitation of `io-ts` schemas used to request query validation. It's handled by `buildRouteValidation()` which doesn't parse input parameters. For example there is a request with a query string `/internal/detection_engine/rules/_coverage_overview?filter={"search_term": "rule 1"}`, it's handled and the following object gets passed to `buildRouteValidation()`</del> ```ts { "filter": '{"search_term": "rule 1"}' } ``` <del>as you may notice `'{"search_term": "rule 1"}'` is a string so the `io-ts` schema validation fails while the request looks correct. In contrast a similar `@kbn/config-schema` schema used instead for the request query validation handles it correctly. As the reference it works [here](https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/routes/find_rules.ts#L100C16-L100C27) for `internal/alerting/rules/_find` endpoint, `fields` query parameter can be a JSON array and validation handles it correctly.</del> UPD: discussed with the team and decided that HTTP POST is more convenient for complex filters. - During FTR tests implementation I've noticed the server fails if the second page (10000 - 20000 rules) is requested with an error ``` illegal_argument_exception: Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. ``` There is a chance it can fail with the same error in prod. UPD: The problem in reproducible in prod. To avoid a server crash the endpoint doesn't handle more than 10k rules. The problem will be addressed in #160698. ## Posible improvements - [x] Move KQL utility functions into common folder to be shared between FE and BE (done) - Implement stricter filtering to return only searched tactic, technique and sub-technique (leave as is for now) - Use HTTP GET instead of HTTP POST (discussed with the team and decided that HTTP POST is more convenient for complex filters) - Make sure pages above 10000 rules are handled properly (will be addresses in #160698) ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenario
|
Implemented in #160480 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Epic: https://github.com/elastic/security-team/issues/2905 (internal)
Depends on: #158202
Summary
Implement an API endpoint based on the API contract defined in #158202 providing rules information broken down by ATT&CK tactics, techniques and sub-techniques so the UI is able to consume it and handle filtering requests.
Details
As stated in #158202 we map our pre-built protections to ATT&CK tactics/techniques/sub-techniques where applicable. When creating custom rules, users can also map them to ATT&CK.
Users want to view the detection rules coverage based on MITRE ATT&CK framework. The implemented API endpoint has to match the API contract defined in #158202 i.e. cover the following items
The text was updated successfully, but these errors were encountered: