diff --git a/docs/siem/detections/api/rules-api-create.asciidoc b/docs/siem/detections/api/rules-api-create.asciidoc index 0273ee051f..3ca370e77a 100644 --- a/docs/siem/detections/api/rules-api-create.asciidoc +++ b/docs/siem/detections/api/rules-api-create.asciidoc @@ -1,13 +1,23 @@ [[rules-api-create]] === Create rule -Creates a new signal detection rule. +Creates a new detection rule. -You can create two types of rules: +You can create these types of rules: -* Query-based rules, which search the defined indices and creates a signal when +* *Query rules*: Searches the defined indices and creates an alert when a document matches the rule's query. -* {ml-cap} rules, which create a signal when a {ml} job discovers an anomaly above the defined threshold (see <>). +* *Threshold rules*: Searches the defined indices and creates an alert when the +number of times the specified field's value meets the threshold during a single +execution. When there are multiple values that meet the threshold, an alert is +generated for each value. ++ +For example, if the threshold `field` is `source.ip` and its `value` is `10`, an +alert is generated for every source IP address that appears in at least 10 of +the rule's search results. If you're interested, see +{ref}/search-aggregations-bucket-terms-aggregation.html[Terms Aggregation] for +more information. +* *{ml-cap} rules*: Creates an alert when a {ml} job discovers an anomaly above the defined threshold (see <>). IMPORTANT: To create {ml} rules, you must have the https://www.elastic.co/subscriptions[appropriate license] or use a @@ -32,7 +42,7 @@ the `groups` field can be used to create rules: ... -------------------------------------------------- -Additionally, you can set up notifications for when rules create signals. The +Additionally, you can set up notifications for when rules create alerts. The notifications use the {kib} {kibana-ref}/alerting-getting-started.html[Alerting and Actions framework]. Each action type requires a connector. Connectors store the information required to send notifications via external systems. These action types are @@ -43,15 +53,17 @@ supported for rule notifications: * PagerDuty * Webhook -NOTE: For more information on PagerDuty fields, see https://v2.developer.pagerduty.com/v2/docs/send-an-event-events-api-v2[PagerDuty Send a v2 Event API]. +NOTE: For more information on PagerDuty fields, see +https://developer.pagerduty.com/docs/events-api-v2/trigger-events/[Send a v2 Event]. To retrieve connector IDs, which are required to configure rule notifications, -call `GET :/api/action/_find`. +call the {kib} {kibana-ref}/saved-objects-api-find.html[Find objects API] with +`"type": "action"` in the request payload. For detailed information on {kib} actions and alerting, and additional API calls, see: -* https://github.com/elastic/kibana/tree/master/x-pack/plugins/alerting +* https://github.com/elastic/kibana/tree/master/x-pack/plugins/alerts * https://github.com/elastic/kibana/tree/master/x-pack/plugins/actions ==== Request URL @@ -60,145 +72,240 @@ calls, see: ==== Request body -A JSON object with these fields: +A JSON object that defines the rule's values. + +===== Required fields for all rule types [width="100%",options="header"] |============================================== -|Name |Type |Description |Required +|Name |Type |Description -|actions |<> |Array defining the automated -actions (notifications) taken when signals are produced. |No +|description |String |The rule's description. -|anomaly_threshold |Integer |Anomaly score threshold above which the rule -creates signals. Valid values are from `0` to `100`. |Yes, for {ml} rules. Not -allowed in `query` rule types. +|name |String |The rule's name. -|description |String |The rule's description. |Yes +|risk_score |Integer a|A numerical representation of the alert's severity from +0 to 100, where: -|enabled |Boolean |Determines whether the rule is enabled. |No, defaults to -`true`. +* `0` - `21` represents low severity +* `22` - `47` represents medium severity +* `48` - `73` represents high severity +* `74` - `100` represents critical severity -|false_positives |String[] |String array used to describe common reasons why -the rule may issue false-positive signals. |No, defaults to an empty array. +|severity |String a|Severity level of alerts produced by the rule, which must +be one of the following: -// |filter |Object |{kibana-ref}/field-filter.html[Filter] used by the rule to -// create a signal |Yes, for filter-based rules only +* `low`: Alerts that are of interest but generally not considered to be +security incidents +* `medium`: Alerts that require investigation +* `high`: Alerts that require immediate investigation +* `critical`: Alerts that indicate it is highly likely a security incident has +occurred -|filters |Object[] |The {ref}/query-filter-context.html[query and filter -context] array used to define the conditions for when signals are created from -events. |No, defaults to an empty array. +|type |String a|Data type on which the rule is based: -[[detection-rules-from]] -|from |String |Time from which data is analyzed each time the rule executes, -using a {ref}/common-options.html#date-math[date math range]. For example, -`now-4200s` means the rule analyzes data from 70 minutes before its start -time. |No, defaults to `now-6m` (analyzes data from 6 minutes before the start -time). +* `query`: query with or without additional filters. +* `saved_query`: saved search, identified in the `saved_id` field. +* `machine_learning`: rule based on a {ml} job's anomaly scores. +* `threshold`: rule based on the number of times a `query` matches the +specified field. -|rule_id |String |Unique ID used to identify rules. For example, when a rule -is converted from a third-party security solution. |No, automatically created -when it is not provided. +|============================================== -|index |String[] |Indices on which the rule functions. |No. For query rules, -defaults to the {siem-soln} indices defined on the {kib} Advanced Settings page -(*Kibana* → *Management* → *Advanced Settings* → `siem:defaultIndex`). +===== Required field for query and threshold rules -|interval |String |Frequency of rule execution, using a -{ref}/common-options.html#date-math[date math range]. For example, `"1h"` -means the rule runs every hour. |No, defaults to `5m` (5 minutes). +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|query |String a|{kibana-ref}/search.html[Query] used by the rule to create +alerts. Technically, this is not required and defaults to an empty string but +that's not very useful. + +|============================================== + +===== Required field for threshold rules + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|threshold |Object a|Defines the field and threshold value for when alerts +are generated, where: + +* `field` (string, required): The field on which the threshold is applied. If +you specify an empty field (`""`), alerts are generated when the query returns +at least the number of results specified in the `value` field. +* `value` (integer, required): The threshold value from which an alert is +generated. + +|============================================== + +===== Required field for saved-query rules + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|saved_id |String |Kibana saved search used by the rule to create alerts. + +|============================================== + +===== Required fields for machine-learning rules + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|anomaly_threshold |Integer |Anomaly score threshold above which the rule +creates an alert. Valid values are from `0` to `100`. |machine_learning_job_id |String |{ml-cap} job ID the rule monitors for -anomaly scores. |Yes, for {ml} rules. Not allowed in `query` rules types. +anomaly scores. -|query |String |{kibana-ref}/search.html[Query] used by the rule to create a -signal. |No. For `query` rules types, defaults to an empty string. Not allowed -in `machine-learning` rule types. +|============================================== -|language |String |Determines the query language, which must be -`kuery` or `lucene`. |No. For `query` rule types, defaults to `kuery`. Not -allowed in `machine-learning` rule types. +===== Optional fields for all rule types -|output_index |String |Index to which signals detected by the rule are saved. -|No, if unspecified signals are saved to `.siem-signals-` index, -where `` is the name of the {kib} space in which the rule exists. +[width="100%",options="header"] +|============================================== +|Name |Type |Description -|saved_id |String |Kibana saved search used by the rule to create signals. -|Yes, for `saved_query` rules only. +|actions |<> |Array defining the automated +actions (notifications) taken when alerts are generated. -|meta |Object |Placeholder for metadata about the rule. |No +|author |String[] |The rule's author. -|risk_score |Integer a|A numerical representation of the signal's severity from -0 to 100, where: +|building_block_type |String |Determines if the rule acts as a building block. +By default, building-block alerts are not displayed in the UI. These rules are +used as a foundation for other rules that do generate alerts. Its value must be +`default`. For more information, see BEN - add link later. -* `0` - `21` represents low severity -* `22` - `47` represents medium severity -* `48` - `73` represents high severity -* `74` - `100` represents critical severity +|enabled |Boolean |Determines whether the rule is enabled. Defaults to `true`. -|Yes +|false_positives |String[] |String array used to describe common reasons why +the rule may issue false-positive alerts. Defaults to an empty array. -|max_signals |Integer |Maximum number of signals the rule can create during a -single execution. |No, defaults to `100`. +[[detection-rules-from]] +|from |String |Time from which data is analyzed each time the rule executes, +using a {ref}/common-options.html#date-math[date math range]. For example, +`now-4200s` means the rule analyzes data from 70 minutes before its start +time. Defaults to `now-6m` (analyzes data from 6 minutes before the start +time). -|name |String |The rule's name. |Yes +|interval |String |Frequency of rule execution, using a +{ref}/common-options.html#date-math[date math range]. For example, `"1h"` +means the rule runs every hour. Defaults to `5m` (5 minutes). -|note |String |Notes to help investigate signals produced by the rule. |No +|license |String |The rule's license. -|severity |String a|Severity level of signals produced by the rule, which must -be one of the following: +|max_signals |Integer |Maximum number of alerts the rule can create during a +single execution. Defaults to `100`. -* `low`: Signals that are of interest but generally not considered to be -security incidents -* `medium`: Signals that require investigation -* `high`: Signals that require immediate investigation -* `critical`: Signals that indicate it is highly likely a security incident has -occurred +|meta |Object |Placeholder for metadata about the rule. -|Yes +|note |String |Notes to help investigate alerts produced by the rule. + +|output_index |String |Index to which alerts created by the rule are saved. +If unspecified alerts are saved to `.siem-signals-` index, +where `` is the name of the {kib} space in which the rule exists. + +|references |String[] |Array containing notes about or references to +relevant information about the rule. Defaults to an empty array. + +|rule_id |String |Unique ID used to identify rules. For example, when a rule +is converted from a third-party security solution. Automatically created when +it is not provided. |tags |String[] |String array containing words and phrases to help categorize, -filter, and search rules. |No, defaults to an empty array. +filter, and search rules. Defaults to an empty array. + +|threat |<> |Object containing attack +information about the type of threat the rule monitors, see +{ecs-ref}/ecs-threat.html[ECS threat fields]. Defaults to an empty array. |throttle |String a|Determines how often actions are taken: * `no_actions`: Never -* `rule`: Every time new signals are detected +* `rule`: Every time new alerts are detected * `1h`: Every hour * `1d`: Every day * `7d`: Every week -|Yes, when actions are used to send notifications. +Required when `actions` are used to send notifications. -// |to |String |Time to which data is analyzed each time the rule executes, using a -// {ref}/common-options.html#date-math[date math range]. For example, `"now-300s"` -// means the rule analyzes data until 5 minutes before its starts time. |Yes +|version |Integer |The rule's version number. Defaults to `1`. -|type |String a|Data type on which the rule is based: +|============================================== -* `query`: query-based conditions with or without additional filters. -* `saved_query`: saved search, identified in the `saved_id` field. -* `machine_learning`: rule based on a {ml} job's anomaly scores. +===== Optional fields for query and threshold rules -|Yes +[width="100%",options="header"] +|============================================== +|Name |Type |Description +|exceptions_list |Object[] a|Array of exception containers, which define +exceptions that prevent the rule from generating alerts even when its other +criteria are met. The object has these fields: -|threat |<> |Object containing attack -information about the type of threat the rule monitors, see -{ecs-ref}/ecs-threat.html[ECS threat fields]. |No, defaults to an empty array. +* `id` (string, required): List ID of the exception container. +* `namespace_type` (string required): Determines whether the exceptions are +valid in only the rule's {kib} space (`single`) or in all {kib} spaces +(`agnostic`). +* `type` (string, required): The exception type, which must be either +a detection rule exception (`detection`) or an endpoint exception (`endpoint`). -|references |String[] |String array containing notes about or references to -relevant information about the rule. |No, defaults to an empty array. +BEN - add link later + +|filters |Object[] |The {ref}/query-filter-context.html[query and filter +context] array used to define the conditions for when alerts are created from +events. Defaults to an empty array. + +|index |String[] |Indices on which the rule functions. Defaults to the +Security Solution indices defined on the {kib} Advanced Settings page +(*Kibana* → *Stack Management* → *Advanced Settings* → +`securitySolution:defaultIndex`). + +|language |String |Determines the query language, which must be +`kuery` or `lucene`. Defaults to `kuery`. -|version |Integer |The rule's version number. a|No, defaults to `1`. +|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with +a value from the source event: + +* `field` (string, required): Source event field used to override the default +`risk_score`. This field must be an integer. +* `operator` (string, required): Must be `equals`. +* `value`(string, required): Must be an empty string (`""`). + +|rule_name_override |String |Sets which field in the source event is used to +populate the alert's `signal.rule.name` value (in the UI, this value is +displayed in the *Rule* column on the Detections page). When unspecified, the +rule's `name` value is used. The source field must be a string data type. + +|severity_mapping |Object[] a|Overrides generated alerts' `severity` with +values from the source event: + +* `field` (string, required): Source event field used to override the default +`severity`. +* `operator` (string, required): Must be `equals`. +* `severity` (string, required): Mapped severity value, must be `low`, +`medium`, `high`, or `critical`. +* `value`(string, required): Field value used to determine the `severity`. + +|timestamp_override |String |Sets the time field used to query indices. +When unspecified, rules query the `@timestamp` field. The source field +must be an {es} date data type. |============================================== [[actions-object-schema]] ===== `actions` schema +All fields are required: + [width="100%",options="header"] |============================================== -|Name |Type |Description |Required +|Name |Type |Description |action_type_id |String a|The action type used for sending notifications, can be: @@ -208,12 +315,10 @@ be: * `.pagerduty` * `.webhook` -|Yes - -|group |String |Optionally groups actions by use cases. Use `default` for signal -notifications.|Yes +|group |String |Optionally groups actions by use cases. Use `default` for alert +notifications. -|id |String |The connector ID. |Yes +|id |String |The connector ID. |params |Object a|Object containing the allowed connector fields, which varies according to the connector type: @@ -227,15 +332,15 @@ sent. At least one field must have a value. * For Webhook: ** `body` (string, required): JSON payload. * For PagerDuty: -** `severity` (string, required): Severity of on the signal notification, can +** `severity` (string, required): Severity of on the alert notification, can be: `Critical`, `Error`, `Warning` or `Info`. ** `eventAction` (string, required): Event https://v2.developer.pagerduty.com/docs/events-api-v2#event-action[action type], which can be `trigger`, `resolve`, or `acknowledge`. -** `dedupKey` (string, optional): Groups signal notifications with the same +** `dedupKey` (string, optional): Groups alert notifications with the same PagerDuty alert. ** `timestamp` (DateTime, optional): https://v2.developer.pagerduty.com/v2/docs/types#datetime[ISO-8601 format timestamp]. ** `component` (string, optional): Source machine component responsible for the -event, for example `siem`. +event, for example `security-solution`. ** `group` (string, optional): Enables logical grouping of service components. ** `source` (string, optional): The affected system. Defaults to the {kib} saved object ID of the action. @@ -243,30 +348,28 @@ saved object ID of the action. `No summary provided`. Maximum length is 1024 characters. ** `class` (string, optional): Value indicating the class/type of the event. -|Yes - |============================================== All text fields (such as `message` fields) can contain placeholders for rule -and signal details: +and alert details: -* `{{state.signals_count}}`: Number of signals detected -* `{{{context.results_link}}}`: URL to the signals in {kib} +* `{{state.signals_count}}`: Number of alerts detected +* `{{{context.results_link}}}`: URL to the alerts in {kib} * `{{context.rule.anomaly_threshold}}`: Anomaly threshold score above which -signals are generated ({ml} rules only) +alerts are generated ({ml} rules only) * `{{context.rule.description}}`: Rule description * `{{context.rule.false_positives}}`: Rule false positives -* `{{context.rule.filters}}`: Rule filters (query-based rules only) +* `{{context.rule.filters}}`: Rule filters (query rules only) * `{{context.rule.id}}`: Unique rule ID returned after creating the rule -* `{{context.rule.index}}`: Indices rule runs on (query-based rules only) -* `{{context.rule.language}}`: Rule query language (query-based rules only) +* `{{context.rule.index}}`: Indices rule runs on (query rules only) +* `{{context.rule.language}}`: Rule query language (query rules only) * `{{context.rule.machine_learning_job_id}}`: ID of associated {ml} job ({ml} rules only) -* `{{context.rule.max_signals}}`: Maximum allowed number of signals per rule +* `{{context.rule.max_signals}}`: Maximum allowed number of alerts per rule execution * `{{context.rule.name}}`: Rule name -* `{{context.rule.output_index}}`: Index to which signals are written -* `{{context.rule.query}}`: Rule query (query-based rules only) +* `{{context.rule.output_index}}`: Index to which alerts are written +* `{{context.rule.query}}`: Rule query (query rules only) * `{{context.rule.references}}`: Rule references * `{{context.rule.risk_score}}`: Rule risk score * `{{context.rule.rule_id}}`: Generated or user-defined rule ID that can be @@ -274,6 +377,7 @@ used as an identifier across systems * `{{context.rule.saved_id}}`: Saved search ID * `{{context.rule.severity}}`: Rule severity * `{{context.rule.threat}}`: Rule threat framework +* `{{context.rule.threshold}}`: Rule threshold (threshold rules only) * `{{context.rule.timeline_id}}`: Associated timeline ID * `{{context.rule.timeline_title}}`: Associated timeline name * `{{context.rule.type}}`: Rule type @@ -282,15 +386,13 @@ used as an identifier across systems [[threats-object-create]] ===== `threat` schema -NOTE: Only threats described using the MITRE ATT&CK^TM^ framework are displayed -in the UI (*SIEM* -> *Detections* -> *Manage signal detection rules* -> -). +All fields are required: [width="100%",options="header"] |============================================== -|Name |Type |Description |Required +|Name |Type |Description -|framework |String |Relevant attack framework. |Yes +|framework |String |Relevant attack framework. |tactic |Object a|Object containing information on the attack type: @@ -298,8 +400,6 @@ in the UI (*SIEM* -> *Detections* -> *Manage signal detection rules* -> * `name` - string, required * `reference` - string, required -|Yes - |technique |Object a|Object containing information on the attack technique: @@ -307,15 +407,17 @@ technique: * `name` - string, required * `reference` - string, required -|Yes - |============================================== +NOTE: Only threats described using the MITRE ATT&CK^TM^ framework are displayed +in the UI (*Security* -> *Detections* -> *Manage detection rules* -> ). + ===== Example requests *Example 1* -Query-based rule that searches for processes started by MS Office: +Query rule that searches for processes started by MS Office: [source,console] -------------------------------------------------- @@ -360,7 +462,79 @@ When it runs next, at 16:00, it will analyze data from 14:50 until 16:00. *Example 2* -{ml-cap} rule that creates signals, and sends Slack notifications, when the +Threshold rule that detects multiple failed login attempts to a Windows host +from the same external source IP address, and maps the `severity` value to +custom source event fields: + +[source,console] +-------------------------------------------------- +POST api/detection_engine/rules +{ + "description": "Detects when there are 20 or more failed login attempts from the same IP address with a 2 minute time frame.", + "enabled": true, + "exceptions_list": [ <1> + { + "id": "int-ips", + "namespace_type": "single", + "type": "detection" + } + ], + "from": "now-180s", + "index": [ + "winlogbeat-*" + ], + "interval": "2m", + "name": "Liverpool Windows server prml-19", + "query": "host.name:prml-19 and event.category:authentication and event.outcome:failure", + "risk_score": 30, + "rule_id": "liv-win-ser-logins", + "severity": "low", + "severity_mapping": [ <2> + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "low", + "value": "Manchester" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "medium", + "value": "London" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "high", + "value": "Birmingham" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "critical", + "value": "Wallingford" + } + ], + "tags": [ + "Brute force" + ], + "threshold": { <3> + "field": "source.ip", + "value": 20 + }, + "type": "threshold" +} +-------------------------------------------------- +// KIBANA + +<1> Exception list container used to exclude internal IP addresses. +<2> Alert severity levels are mapped according to the defined field values. +<3> Alerts are generated when the same source IP address is discovered in at +least 20 results. + +*Example 3* + +{ml-cap} rule that creates alerts, and sends Slack notifications, when the `linux_anomalous_network_activity_ecs` {ml} job discovers anomalies with a threshold of 70 or above: @@ -372,7 +546,7 @@ POST api/detection_engine/rules "rule_id": "ml_linux_network_high_threshold", "risk_score": 70, "machine_learning_job_id": "linux_anomalous_network_activity_ecs", - "description": "Generates signals when the job discovers anomalies over 70", + "description": "Generates alerts when the job discovers anomalies over 70", "interval": "5m", "name": "Anomalous Linux network activity", "note": "Shut down the internet.", @@ -410,7 +584,7 @@ A JSON object that includes a unique ID, the time the rule was created, and its version number. If the request payload did not include a `rule_id` field, a unique rule ID is also generated. -Example response for a query-based rule: +Example response for a query rule: [source,json] -------------------------------------------------- @@ -468,7 +642,7 @@ Example response for a {ml} job rule: "created_at": "2020-04-07T14:45:15.679Z", "updated_at": "2020-04-07T14:45:15.892Z", "created_by": "LiverpoolFC", - "description": "Generates signals when the job discovers anomalies over 70", + "description": "Generates alerts when the job discovers anomalies over 70", "enabled": true, "false_positives": [], "from": "now-6m", @@ -509,3 +683,82 @@ Example response for a {ml} job rule: "machine_learning_job_id": "linux_anomalous_network_activity_ecs" } -------------------------------------------------- + +Example response for a threshold rule: + +[source,json] +-------------------------------------------------- +{ + "author": [], + "created_at": "2020-07-22T10:27:23.486Z", + "updated_at": "2020-07-22T10:27:23.673Z", + "created_by": "LiverpoolFC", + "description": "Detects when there are 20 or more failed login attempts from the same IP address with a 2 minute time frame.", + "enabled": true, + "false_positives": [], + "from": "now-180s", + "id": "15dbde26-b627-4d74-bb1f-a5e0ed9e4993", + "immutable": false, + "interval": "2m", + "rule_id": "liv-win-ser-logins", + "output_index": ".siem-signals-default", + "max_signals": 100, + "risk_score": 30, + "risk_score_mapping": [], + "name": "Liverpool Windows server prml-19", + "references": [], + "severity": "low", + "severity_mapping": [ + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "low", + "value": "Manchester" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "medium", + "value": "London" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "high", + "value": "Birmingham" + }, + { + "field": "source.geo.city_name", + "operator": "equals", + "severity": "critical", + "value": "Wallingford" + } + ], + "updated_by": "LiverpoolFC", + "tags": [ + "Brute force" + ], + "to": "now", + "type": "threshold", + "threat": [], + "version": 1, + "exceptions_list": [ + { + "id": "int-ips", + "namespace_type": "single", + "type": "detection" + } + ], + "actions": [], + "index": [ + "winlogbeat-*" + ], + "throttle": "no_actions", + "query": "host.name:prml-19 and event.category:authentication and event.outcome:failure", + "language": "kuery", + "threshold": { + "field": "source.ip", + "value": 20 + } +} +-------------------------------------------------- diff --git a/docs/siem/detections/api/rules-api-update.asciidoc b/docs/siem/detections/api/rules-api-update.asciidoc index 5c753a8148..c1fc8fe3d3 100644 --- a/docs/siem/detections/api/rules-api-update.asciidoc +++ b/docs/siem/detections/api/rules-api-update.asciidoc @@ -1,7 +1,7 @@ [[rules-api-update]] === Update rule -Updates an existing signal detection rule. +Updates an existing detection rule. You can use `PUT` or `PATCH` methods to update rules, where: @@ -25,145 +25,240 @@ IMPORTANT: If you call `PUT` to update a rule, all unspecified fields are deleted. You cannot modify the `id` or `rule_id` values. For `PATCH` calls, any of the fields can be modified, whereas for `PUT` calls, -some fields are required. +some fields are required. + +===== Fields required for `PUT` calls [width="100%",options="header"] |============================================== -|Name |Type |Description |Required (`PUT` calls) +|Name |Type |Description -|actions |<> |Array defining the -automated actions (notifications) taken when signals are produced. |No +|description |String |The rule's description. -|anomaly_threshold |Integer |Anomaly score threshold above which the rule -creates signals. Valid values are from `0` to `100`. |Yes, for {ml} rules. Not -allowed in `query` rule types. +|name |String |The rule's name. -|description |String |The rule's description. |Yes +|risk_score |Integer a|A numerical representation of the alert's severity from +0 to 100, where: -|enabled |Boolean |Determines whether the rule is enabled. |No, defaults to -`true`. +* `0` - `21` represents low severity +* `22` - `47` represents medium severity +* `48` - `73` represents high severity +* `74` - `100` represents critical severity -|false_positives |String[] |String array used to describe common reasons why -the rule may issue false-positive signals. |No, defaults to an empty array. +|severity |String a|Severity level of alerts produced by the rule, which must +be one of the following: -// |filter |Object |{kibana-ref}/field-filter.html[Filter] used by the rule to -// create a signal |Yes, for filter-based rules only +* `low`: Alerts that are of interest but generally not considered to be +security incidents +* `medium`: Alerts that require investigation +* `high`: Alerts that require immediate investigation +* `critical`: Alerts that indicate it is highly likely a security incident has +occurred -|filters |Object[] |The {ref}/query-filter-context.html[query and filter -context] array used to define the conditions for when signals are created from -events. |No, defaults to an empty array. +|type |String a|Data type on which the rule is based: -|from |String |Time from which data is analyzed each time the rule executes, -using a {ref}/common-options.html#date-math[date math range]. For example, -`now-4200s` means the rule analyzes data from 70 minutes before its start -time. |No, defaults to `now-6m` (analyzes data from 6 minutes before the start -time). +* `query`: query with or without additional filters. +* `saved_query`: saved search, identified in the `saved_id` field. +* `machine_learning`: rule based on a {ml} job's anomaly scores. +* `threshold`: rule based on the number of times a `query` matches the +specified field. -|index |String[] |Indices on which the rule functions. |No, defaults to the -{siem-soln} indices defined on the {kib} Advanced Settings page (*Kibana* → -*Management* → *Advanced Settings* → `siem:defaultIndex`). +|============================================== -|interval |String |Frequency of rule execution, using a -{ref}/common-options.html#date-math[date math range]. For example, `"1h"` -means the rule runs every hour. |No, defaults to `5m` (5 minutes). +===== Field required for query and threshold rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|query |String a|{kibana-ref}/search.html[Query] used by the rule to create +alerts. Technically, this is not required and defaults to an empty string but +that's not very useful. + +|============================================== + +===== Field required for threshold rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|threshold |Object a|Defines the field and threshold value for when alerts +are generated, where: + +* `field` (string, required): The field on which the threshold is applied. If +you specify an empty field (`""`), alerts are generated when the query returns +at least the number of results specified in the `value` field. +* `value` (integer, required): The threshold value from which an alert is +generated. + +|============================================== + +===== Field required for saved-query rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|saved_id |String |Kibana saved search used by the rule to create alerts. + +|============================================== + +===== Fields required for machine-learning rules `PUT` calls + +[width="100%",options="header"] +|============================================== +|Name |Type |Description + +|anomaly_threshold |Integer |Anomaly score threshold above which the rule +creates an alert. Valid values are from `0` to `100`. |machine_learning_job_id |String |{ml-cap} job ID the rule monitors for -anomaly scores. |Yes, for {ml} rules. Not allowed in `query` rules types. +anomaly scores. -|query |String |{kibana-ref}/search.html[Query] used by the rule to create a -signal. |No. For `query` rules types, defaults to an empty string. Not allowed -in `machine-learning` rule types. +|============================================== -|language |String |Determines the query language, which must be -`kuery` or `lucene`. |No. For `query` rule types, defaults to `kuery`. Not -allowed in `machine-learning` rule types. +===== Optional fields for all rule types -|output_index |String |Index to which signals detected by the rule are saved. -|No, if unspecified signals are saved to `.siem-signals-` index, -where `` is the name of the {kib} space in which the rule exists. +[width="100%",options="header"] +|============================================== +|Name |Type |Description -|saved_id |String |Kibana saved search used by the rule to create signals. -|Yes, for `saved_query` rules only. +|actions |<> |Array defining the automated +actions (notifications) taken when alerts are generated. -|meta |Object |Placeholder for metadata about the rule. |No +|author |String[] |The rule's author. -|risk_score |Integer a|A numerical representation of the signal's severity from -0 to 100, where: +|building_block_type |String |Determines if the rule acts as a building block. +By default, building-block alerts are not displayed in the UI. These rules are +used as a foundation for other rules that do generate alerts. Its value must be +`default`. For more information, see BEN - add link later. -* `0` - `21` represents low severity -* `22` - `47` represents medium severity -* `48` - `73` represents high severity -* `74` - `100` represents critical severity +|enabled |Boolean |Determines whether the rule is enabled. Defaults to `true`. -|Yes +|false_positives |String[] |String array used to describe common reasons why +the rule may issue false-positive alerts. Defaults to an empty array. -|max_signals |Integer |Maximum number of signals the rule can create during a -single execution. |No, defaults to `100`. +[[detection-rules-from]] +|from |String |Time from which data is analyzed each time the rule executes, +using a {ref}/common-options.html#date-math[date math range]. For example, +`now-4200s` means the rule analyzes data from 70 minutes before its start +time. Defaults to `now-6m` (analyzes data from 6 minutes before the start +time). -|name |String |The rule's name. |Yes +|interval |String |Frequency of rule execution, using a +{ref}/common-options.html#date-math[date math range]. For example, `"1h"` +means the rule runs every hour. Defaults to `5m` (5 minutes). -|note |String |Notes to help investigate signals produced by the rule. |No +|license |String |The rule's license. -|severity |String a|Severity level of signals produced by the rule, which must -be one of the following: +|max_signals |Integer |Maximum number of alerts the rule can create during a +single execution. Defaults to `100`. -* `low`: Signals that are of interest but generally not considered to be -security incidents -* `medium`: Signals that require investigation -* `high`: Signals that require immediate investigation -* `critical`: Signals that indicate it is highly likely a security incident has -occurred +|meta |Object |Placeholder for metadata about the rule. -|Yes +|note |String |Notes to help investigate alerts produced by the rule. + +|output_index |String |Index to which alerts created by the rule are saved. +If unspecified alerts are saved to `.siem-signals-` index, +where `` is the name of the {kib} space in which the rule exists. + +|references |String[] |Array containing notes about or references to +relevant information about the rule. Defaults to an empty array. |tags |String[] |String array containing words and phrases to help categorize, -filter, and search rules. |No, defaults to an empty array. +filter, and search rules. Defaults to an empty array. -// |to |String |Time to which data is analyzed each time the rule executes, using a -// {ref}/common-options.html#date-math[date math range]. For example, `"now-300s"` -// means the rule analyzes data until 5 minutes before its starts time. +|threat |<> |Object containing attack +information about the type of threat the rule monitors, see +{ecs-ref}/ecs-threat.html[ECS threat fields]. Defaults to an empty array. |throttle |String a|Determines how often actions are taken: * `no_actions`: Never -* `rule`: Every time new signals are detected +* `rule`: Every time new alerts are detected * `1h`: Every hour * `1d`: Every day * `7d`: Every week -|Yes, when actions are used to send notifications. +Required when `actions` are used to send notifications. -|type |String a|Data type on which the rule is based: +|version |Integer a|The rule's version number. If this is not provided, the +rule's version number is incremented by 1. -* `query`: query-based conditions with or without additional filters -* `saved_query`: saved search, identified in the `saved_id` field -* `machine_learning`: rule based on a {ml} job's anomaly scores. +`PATCH` calls enabling and disabling the rule do not increment its version +number. + +|============================================== -|Yes +===== Optional fields for query and threshold rules -|threat |<> |Object containing attack -information about the type of threat the rule monitors, see -{ecs-ref}/ecs-threat.html[ECS threat fields]. |No, defaults to an empty array. +[width="100%",options="header"] +|============================================== +|Name |Type |Description -|references |String[] |String array containing notes about or references to -relevant information about the rule. |No, defaults to an empty array. +|exceptions_list |Object[] a|Array of exception containers, which define +exceptions that prevent the rule from generating alerts even when its other +criteria are met. The object has these fields: -|version |Integer a|The rule's version number. If this is not provided, the -rule's version number is incremented by 1. +* `id` (string, required): List ID of the exception container. +* `namespace_type` (string required): Determines whether the exceptions are +valid in only the rule's {kib} space (`single`) or in all {kib} spaces +(`agnostic`). +* `type` (string, required): The exception type, which must be either +a detection rule exception (`detection`) or an endpoint exception (`endpoint`). -`PATCH` calls enabling and disabling the rule do not increment its version -number. +BEN - add link later + +|filters |Object[] |The {ref}/query-filter-context.html[query and filter +context] array used to define the conditions for when alerts are created from +events. Defaults to an empty array. + +|index |String[] |Indices on which the rule functions. Defaults to the +Security Solution indices defined on the {kib} Advanced Settings page +(*Kibana* → *Stack Management* → *Advanced Settings* → +`securitySolution:defaultIndex`). + +|language |String |Determines the query language, which must be +`kuery` or `lucene`. Defaults to `kuery`. -|No +|risk_score_mapping |Object[] a|Overrides generated alerts' `risk_score` with +a value from the source event: + +* `field` (string, required): Source event field used to override the default +`risk_score`. This field must be an integer. +* `operator` (string, required): Must be `equals`. +* `value`(string, required): Must be an empty string (`""`). + +|rule_name_override |String |Sets which field in the source event is used to +populate the alert's `signal.rule.name` value (in the UI, this value is +displayed in the *Rule* column on the Detections page). When unspecified, the +rule's `name` value is used. The source field must be a string data type. + +|severity_mapping |Object[] a|Overrides generated alerts' `severity` with +values from the source event: + +* `field` (string, required): Source event field used to override the default +`severity`. +* `operator` (string, required): Must be `equals`. +* `severity` (string, required): Mapped severity value, must be `low`, +`medium`, `high`, or `critical`. +* `value`(string, required): Field value used to determine the `severity`. + +|timestamp_override |String |Sets the time field used to query indices. +When unspecified, rules query the `@timestamp` field. The source field +must be an {es} date data type. |============================================== [[actions-object-schema-update]] ===== `actions` schema +These fields are required when calling `PUT` to modify the `actions` object: + [width="100%",options="header"] |============================================== -|Name |Type |Description |Required (`PUT` calls) +|Name |Type |Description |action_type_id |String a|The action type used for sending notifications, can be: @@ -173,12 +268,10 @@ be: * `.pagerduty` * `.webhook` -|Yes - -|group |String |Optionally groups actions by use cases. Use `default` for signal -notifications.|Yes +|group |String |Optionally groups actions by use cases. Use `default` for alert +notifications. -|id |String |The connector ID. |Yes +|id |String |The connector ID. |params |Object a|Object containing the allowed connector fields, which varies according to the connector type: @@ -192,15 +285,15 @@ sent. At least one field must have a value. * For Webhook: ** `body` (string, required): JSON payload. * For PagerDuty: -** `severity` (string, required): Severity of on the signal notification, can +** `severity` (string, required): Severity of on the alert notification, can be: `Critical`, `Error`, `Warning` or `Info`. ** `eventAction` (string, required): Event https://v2.developer.pagerduty.com/docs/events-api-v2#event-action[action type], which can be `trigger`, `resolve`, or `acknowledge`. -** `dedupKey` (string, optional): Groups signal notifications with the same +** `dedupKey` (string, optional): Groups alert notifications with the same PagerDuty alert. ** `timestamp` (DateTime, optional): https://v2.developer.pagerduty.com/v2/docs/types#datetime[ISO-8601 format timestamp]. ** `component` (string, optional): Source machine component responsible for the -event, for example `siem`. +event, for example `security-solution`. ** `group` (string, optional): Enables logical grouping of service components. ** `source` (string, optional): The affected system. Defaults to the {kib} saved object ID of the action. @@ -208,23 +301,19 @@ saved object ID of the action. `No summary provided`. Maximum length is 1024 characters. ** `class` (string, optional): Value indicating the class/type of the event. -|Yes - |============================================== [[threats-object-update]] ===== `threat` schema -NOTE: Only threats described using the MITRE ATT&CK^TM^ framework are displayed -in the UI (*SIEM* -> *Detections* -> *Manage signal detection rules* -> -). +These fields are required when calling `PUT` to modify the `threat` object: [width="100%",options="header"] |============================================== -|Name |Type |Description |Required (`PUT` calls) +|Name |Type |Description -|framework |String |Relevant attack framework. |Yes +|framework |String |Relevant attack framework. |tactic |Object a|Object containing information on the attack type: @@ -232,8 +321,6 @@ in the UI (*SIEM* -> *Detections* -> *Manage signal detection rules* -> * `name` - string, required * `reference` - string, required -|Yes - |technique |Object a|Object containing information on the attack technique: @@ -241,10 +328,12 @@ technique: * `name` - string, required * `reference` - string, required -|Yes - |============================================== +NOTE: Only threats described using the MITRE ATT&CK^TM^ framework are displayed +in the UI (*Security* -> *Detections* -> *Manage detection rules* -> ). + ===== Example request Updates the `threat` object: diff --git a/docs/siem/detections/api/signals-api-overview.asciidoc b/docs/siem/detections/api/signals-api-overview.asciidoc index ee7414cda5..d7cc1d279e 100644 --- a/docs/siem/detections/api/signals-api-overview.asciidoc +++ b/docs/siem/detections/api/signals-api-overview.asciidoc @@ -2,17 +2,17 @@ [role="xpack"] === Signals endpoint -The signals endpoint is for retrieving, aggregating, and updating signals. -For detailed information on how to retrieve and aggregate results from the -indices, see: +The signals endpoint is for retrieving, aggregating, and updating detection +alerts. For detailed information on how to retrieve and aggregate results from +the indices, see: -* {ref}/getting-started-search.html[Start searching] +* {ref}/getting-started-search.html[Start searching indices] * {ref}/search-aggregations.html[Aggregations] * {ref}/query-dsl.html[Query DSL] -==== Get signals +==== Get alerts -Aggregates and returns signals. +Aggregates and returns alerts. ===== Request URL @@ -24,9 +24,9 @@ A query DSL that determines which results are returned. ====== Example request -Gets aggregated results of all open signals with a risk score equal to or +Gets aggregated results of all open alerts with a risk score equal to or greater than 70. It also returns the timestamps of the oldest and -newest signals that meet the query's criteria. +newest alerts that meet the query's criteria. [source,console] -------------------------------------------------- @@ -61,8 +61,35 @@ POST api/detection_engine/signals/search } ] } - }, - "size": 0 + } +} +-------------------------------------------------- + +Gets all in-progress alerts with a risk score equal to or +greater than 70. + +[source,console] +-------------------------------------------------- +POST api/detection_engine/signals/search +{ + "query": { + "bool": { + "filter": [ + { + "match": { + "signal.status": "in-progress" + } + }, + { + "range": { + "signal.rule.risk_score": { + "gte": 70 + } + } + } + ] + } + } } -------------------------------------------------- // KIBANA @@ -74,7 +101,7 @@ POST api/detection_engine/signals/search ====== Response payload -A JSON object with the aggregated values and requested signals. +A JSON object with the aggregated values and requested alerts. Example response: @@ -110,9 +137,9 @@ Example response: } -------------------------------------------------- -==== Set signal status +==== Set alert status -Sets the status of one or more signals. +Sets the status of one or more alert. ===== Request URL @@ -126,17 +153,20 @@ A JSON object with either a `query` or `signals_id` field: |============================================== |Name |Type |Description |Required -|`signal_ids` |String[] |Array of rule IDs. |Yes, when the `query` field is not used. +|`signal_ids` |String[] |Array of alert IDs. |Yes, when the `query` field is +not used. -|`query` |Query DSL |Query that determines which signals are updated. |Yes, when the `signal_ids` field is not used. +|`query` |Query DSL |Query that determines which alerts are updated. |Yes, when +the `signal_ids` field is not used. -|`status` |String |The new status, which can be `open` or `closed`. |Yes. +|`status` |String |The new status, which can be `open`, `in-progress` or +`closed`. |Yes. |============================================== ====== Example requests -Closes signal with signal IDs: +Closes alerts with `signal_ids`: [source,console] -------------------------------------------------- @@ -151,7 +181,7 @@ POST api/detection_engine/signals/status -------------------------------------------------- // KIBANA -Closes signals that are over a month old and have a risk score less than or +Closes alerts that are over a month old and have a risk score less than or equal to 20: [source,json] @@ -190,7 +220,7 @@ POST api/detection_engine/signals/status ====== Response payload -A JSON object containing the number of updated signals. +A JSON object containing the number of updated alerts. Example response: