diff --git a/filebeat/docs/reference/configuration.asciidoc b/filebeat/docs/reference/configuration.asciidoc index 8ab66d00c8c..309158a36ed 100644 --- a/filebeat/docs/reference/configuration.asciidoc +++ b/filebeat/docs/reference/configuration.asciidoc @@ -12,7 +12,6 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> -* <> * <> * <> * <> @@ -22,6 +21,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> include::configuration/filebeat-options.asciidoc[] diff --git a/filebeat/docs/reference/configuration/filebeat-options.asciidoc b/filebeat/docs/reference/configuration/filebeat-options.asciidoc index 8910fcdb1c3..469b5502e2c 100644 --- a/filebeat/docs/reference/configuration/filebeat-options.asciidoc +++ b/filebeat/docs/reference/configuration/filebeat-options.asciidoc @@ -566,11 +566,11 @@ filebeat.shutdown_timeout: 5s include::../../../../libbeat/docs/generalconfig.asciidoc[] -include::../../../../libbeat/docs/processors-config.asciidoc[] - include::../../../../libbeat/docs/outputconfig.asciidoc[] include::../../../../libbeat/docs/shared-path-config.asciidoc[] include::../../../../libbeat/docs/loggingconfig.asciidoc[] +include::../../../../libbeat/docs/processors-config.asciidoc[] + diff --git a/libbeat/docs/processors-config.asciidoc b/libbeat/docs/processors-config.asciidoc index f339b467932..c1201dcabed 100644 --- a/libbeat/docs/processors-config.asciidoc +++ b/libbeat/docs/processors-config.asciidoc @@ -11,22 +11,21 @@ ////////////////////////////////////////////////////////////////////////// [[configuration-processors]] -=== Processors Configuration +== Processors include::../../libbeat/docs/processors.asciidoc[] -Each processor has associated an action with a set of parameters and optionally a condition. If the condition is -present, then the action is executed only if the condition -is fulfilled. If no condition is passed then the action is always executed. +To define a processor, you specify the processor name, an optional condition, +and a set of parameters: [source,yaml] ------ processors: - - : + - : when: - - : + - : when: @@ -34,21 +33,33 @@ processors: ------ -where can be a way to select the fields that are exported or a way to add meta data to the event , contains the definition of the condition. -and is the list of parameters passed along the . +Where: + +* specifies a <> that performs some kind +of action, such as selecting the fields that are exported or adding metadata to +the event. +* specifies an optional <>. If the +condition is present, then the action is executed only if the condition is +fulfilled. If no condition is passed, then the action is always executed. +* is the list of parameters to pass to the processor. See <> for specific {beatname_uc} examples. -[[filtering-condition]] -==== Condition +[float] +[[conditions]] +=== Conditions -Each condition receives a field to compare or multiple fields under the same condition and then `AND` is used between -them. You can see a list of the <>. +Each condition receives a field to compare. You can specify multiple fields +under the same condition by using `AND` between the fields (for example, +`field1 AND field2`). -For each field, you can specify a simple field name or a nested map, for example `dns.question.name`. +For each field, you can specify a simple field name or a nested map, for +example `dns.question.name`. +See <> for a list of all the fields that are exported by +{beatname_uc}. -A condition can be: +The supported conditions are: * <> * <> @@ -59,14 +70,15 @@ A condition can be: * <> - +[float] [[condition-equals]] -===== equals +==== equals -With the `equals` condition, you can compare if a field has a certain value. The condition accepts only an integer or a string -value. +With the `equals` condition, you can compare if a field has a certain value. +The condition accepts only an integer or a string value. -For example, the following condition checks if the response code of the HTTP transaction is 200: +For example, the following condition checks if the response code of the HTTP +transaction is 200: [source,yaml] ------- @@ -74,13 +86,15 @@ equals: http.response.code: 200 ------- +[float] [[condition-contains]] -===== contains +==== contains The `contains` condition checks if a value is part of a field. The field can be a string or an array of strings. The condition accepts only a string value. -For example, the following condition checks if an error is part of the transaction status: +For example, the following condition checks if an error is part of the +transaction status: [source,yaml] ------ @@ -88,13 +102,15 @@ contains: status: "Specific error" ------ - +[float] [[condition-regexp]] -===== regexp +==== regexp -The `regexp` condition checks the field against a regular expression. The condition accepts only strings. +The `regexp` condition checks the field against a regular expression. The +condition accepts only strings. -For example, the following condition checks if the process name starts with `foo`: +For example, the following condition checks if the process name starts with +`foo`: [source,yaml] ----- @@ -102,14 +118,16 @@ reqexp: system.process.name: "foo.*" ----- +[float] [[condition-range]] -===== range +==== range -The `range` condition checks if the field is in a certain range of values. The condition supports `lt`, `lte`, `gt` and `gte`. The -condition accepts only integer or float values. +The `range` condition checks if the field is in a certain range of values. The +condition supports `lt`, `lte`, `gt` and `gte`. The condition accepts only +integer or float values. -For example, the following condition checks for failed HTTP transaction by comparing the `http.response.code` field with -400. +For example, the following condition checks for failed HTTP transactions by +comparing the `http.response.code` field with 400. [source,yaml] @@ -119,7 +137,7 @@ range: gte: 400 ------ -that can be also translated to: +This can also be written as: [source,yaml] ---- @@ -127,7 +145,8 @@ range: http.response.code.gte: 400 ---- -For example, the following condition checks if the CPU usage in percentage has a value between 0.5 and 0.8. +The following condition checks if the CPU usage in percentage has a value +between 0.5 and 0.8. [source,yaml] ------ @@ -137,8 +156,9 @@ range: ------ +[float] [[condition-or]] -===== OR +==== OR The `or` operator receives a list of conditions. @@ -152,7 +172,8 @@ or: ------- -For example the condition `http.response.code = 304 OR http.response.code = 404` translates to: +For example, to configure the condition +`http.response.code = 304 OR http.response.code = 404`: [source,yaml] ------ @@ -163,9 +184,9 @@ or: http.response.code: 404 ------ - +[float] [[condition-and]] -===== AND +==== AND The `and` operator receives a list of conditions. @@ -179,7 +200,8 @@ and: ------- -For example the condition `http.response.code = 200 AND status = OK` translates to: +For example, to configure the condition +`http.response.code = 200 AND status = OK`: [source,yaml] ------ @@ -202,8 +224,9 @@ or: ------ +[float] [[condition-not]] -===== NOT +==== NOT The `not` operator receives the condition to negate. @@ -214,7 +237,7 @@ not: ------- -For example the condition `NOT status = OK` translates to: +For example, to configure the condition `NOT status = OK`: [source,yaml] ------ @@ -223,82 +246,24 @@ not: status: OK ------ +[float] +[[processors]] +=== Processors +The supported processors are: -==== Actions - -The supported actions are: - - * <> - * <> - * <> * <> * <> - -See <> for the full list of possible fields. - -[[include-fields]] -===== include_fields - -The `include_fields` action specifies what fields to export if a certain condition is fulfilled. The condition is -optional and if it's missing then the defined fields are always exported. The `@timestamp` and -`type` fields are always exported, even if they are not defined in the `include_fields` list. - -[source,yaml] -------- -processors: - - include_fields: - when: - condition - fields: ["field1", "field2", ...] -------- - -You can specify multiple `include_fields` actions under the `processors` section. - - -NOTE: If you define an empty list of fields under `include_fields`, then only the required fields, `@timestamp` and `type`, are -exported. - - -[[drop-fields]] -===== drop_fields - -The `drop_fields` action specifies what fields to drop if a certain condition is fulfilled. The condition is optional -and if it's missing then the defined fields are always dropped. The `@timestamp` and `type` fields cannot be dropped, -even if they show up in the `drop_fields` list. - -[source,yaml] ------------------------------------------------------ -processors: - - drop_fields: - when: - condition - fields: ["field1", "field2", ...] ------------------------------------------------------ - -NOTE: If you define an empty list of fields under `drop_fields`, then no fields are dropped. - - -[[drop-event]] -===== drop_event - -The `drop_event` action drops the entire event if the associated condition is fulfilled. The condition is mandatory, as -without one all the events are dropped. - -[source,yaml] ------- -processors: - - drop_event: - when: - condition ------- + * <> + * <> + * <> [[add-cloud-metadata]] -===== add_cloud_metadata +=== add_cloud_metadata -The `add_cloud_metadata` action enriches each event with instance metadata from -the machine's hosting provider. At startup it will detect the hosting provider -and cache the instance metadata. +The `add_cloud_metadata` processor enriches each event with instance metadata +from the machine's hosting provider. At startup it will detect the hosting +provider and cache the instance metadata. Three cloud providers are supported. @@ -309,14 +274,16 @@ Three cloud providers are supported. The simple configuration below enables the processor. [source,yaml] --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- processors: - add_cloud_metadata: --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- -The `add_cloud_metadata` action has one optional configuration setting named +The `add_cloud_metadata` processor has one optional configuration setting named `timeout` that specifies the maximum amount of time to wait for a successful -response when detecting the hosting provider. The default timeout value is `3s`. +response when detecting the hosting provider. The default timeout value is +`3s`. + If a timeout occurs then no instance metadata will be added to the events. This makes it possible to enable this processor for all your deployments (in the cloud or on-premise). @@ -327,7 +294,7 @@ examples for each of the supported providers. _EC2_ [source,json] --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- { "meta": { "cloud": { @@ -339,12 +306,12 @@ _EC2_ } } } --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- _Digital Ocean_ [source,json] --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- { "meta": { "cloud": { @@ -354,12 +321,12 @@ _Digital Ocean_ } } } --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- _GCE_ [source,json] --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- { "meta": { "cloud": { @@ -371,12 +338,13 @@ _GCE_ } } } --------------------------------------------------------------------------------- +------------------------------------------------------------------------------- [[decode-json-fields]] -===== decode_json_fields +=== decode_json_fields -The `decode_json_fields` action decodes fields containing JSON strings and replaces the strings with valid JSON objects. +The `decode_json_fields` processor decodes fields containing JSON strings and +replaces the strings with valid JSON objects. [source,yaml] ----------------------------------------------------- @@ -387,8 +355,74 @@ processors: max_depth: 1 ----------------------------------------------------- -The `decode_json_fields` action has the following configuration settings: +The `decode_json_fields` processor has the following configuration settings: `fields`:: The fields containing JSON strings to decode. -`process_array`:: (Optional) A boolean that specifies whether to process arrays. The default is false. +`process_array`:: (Optional) A boolean that specifies whether to process +arrays. The default is false. `max_depth`:: (Optional) The maximum parsing depth. The default is 1. + +[[drop-event]] +=== drop_event + +The `drop_event` processor drops the entire event if the associated condition +is fulfilled. The condition is mandatory, because without one, all the events +are dropped. + +[source,yaml] +------ +processors: + - drop_event: + when: + condition +------ + +See <> for a list of supported conditions. + +[[drop-fields]] +=== drop_fields + +The `drop_fields` processor specifies which fields to drop if a certain +condition is fulfilled. The condition is optional. If it's missing, the +specified fields are always dropped. The `@timestamp` and `type` fields cannot +be dropped, even if they show up in the `drop_fields` list. + +[source,yaml] +----------------------------------------------------- +processors: + - drop_fields: + when: + condition + fields: ["field1", "field2", ...] +----------------------------------------------------- + +See <> for a list of supported conditions. + +NOTE: If you define an empty list of fields under `drop_fields`, then no fields +are dropped. + +[[include-fields]] +=== include_fields + +The `include_fields` processor specifies which fields to export if a certain +condition is fulfilled. The condition is optional. If it's missing, the +specified fields are always exported. The `@timestamp` and `type` fields are +always exported, even if they are not defined in the `include_fields` list. + +[source,yaml] +------- +processors: + - include_fields: + when: + condition + fields: ["field1", "field2", ...] +------- + +See <> for a list of supported conditions. + +You can specify multiple `include_fields` processors under the `processors` +section. + +NOTE: If you define an empty list of fields under `include_fields`, then only +the required fields, `@timestamp` and `type`, are exported. + diff --git a/metricbeat/docs/metricbeat-filtering.asciidoc b/metricbeat/docs/metricbeat-filtering.asciidoc index 82eb123c49d..c78a25a890c 100644 --- a/metricbeat/docs/metricbeat-filtering.asciidoc +++ b/metricbeat/docs/metricbeat-filtering.asciidoc @@ -1,18 +1,23 @@ [[filtering-and-enhancing-data]] == Filtering and Enhancing the Exported Data -When your use case requires only a subset of the data exported by Metricbeat or you need to add metadata, -you can <>, or you can <>. +Your use case might require only a subset of the data exported by Metricbeat, or +you might need to enhance the exported data (for example, by adding metadata). +Metricbeat provides a couple of options for filtering and enhancing exported +data. You can: +* <> to +configure each module to include or exclude specific events or fields. +* <> to configure global processing +across all data exported by Metricbeat. [float] [[metricbeat-filtering-overview]] -=== Metricbeat Module Filtering +=== Filtering at the Metricbeat Module Level Each module accepts a list of filters in its configuration. These filters are applied to the data generated by the module. These filters are applied to the -module data prior to the addition of the common beat fields like `beat.hostname` +module data prior to the addition of the common Beat fields like `beat.hostname` and `type` so they can only be used to filter fields from the module. The following example reduces the exported fields of the Redis module to @@ -37,8 +42,8 @@ metricbeat.modules: include::../../libbeat/docs/processors.asciidoc[] -For example, the following filters configuration reduces the exported fields by -dropping the `beat.name` and `beat.hostname` fields under `beat`, from all documents. +For example, the following configuration reduces the exported fields by +dropping the `beat.name` and `beat.hostname` fields under `beat` from all documents. [source, yaml] ---- diff --git a/metricbeat/docs/reference/configuration.asciidoc b/metricbeat/docs/reference/configuration.asciidoc index af2bdc7c0b3..4e47584f977 100644 --- a/metricbeat/docs/reference/configuration.asciidoc +++ b/metricbeat/docs/reference/configuration.asciidoc @@ -10,7 +10,6 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> -* <> * <> * <> * <> @@ -20,5 +19,6 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> include::configuration/metricbeat-options.asciidoc[] diff --git a/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc b/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc index 0a4f06bb0b3..0d0ac92c483 100644 --- a/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc +++ b/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc @@ -67,10 +67,11 @@ filters, see <>. include::../../../../libbeat/docs/generalconfig.asciidoc[] -include::../../../../libbeat/docs/processors-config.asciidoc[] - include::../../../../libbeat/docs/outputconfig.asciidoc[] include::../../../../libbeat/docs/shared-path-config.asciidoc[] include::../../../../libbeat/docs/loggingconfig.asciidoc[] + +include::../../../../libbeat/docs/processors-config.asciidoc[] + diff --git a/packetbeat/docs/packetbeat-filtering.asciidoc b/packetbeat/docs/packetbeat-filtering.asciidoc index af01dafc8d4..d8d75e16665 100644 --- a/packetbeat/docs/packetbeat-filtering.asciidoc +++ b/packetbeat/docs/packetbeat-filtering.asciidoc @@ -8,7 +8,7 @@ requests and their response codes are reported: [source, yaml] ----------------------------------------------------- -filters: +processors: - include_fields: fields: - bytes_in diff --git a/packetbeat/docs/reference/configuration.asciidoc b/packetbeat/docs/reference/configuration.asciidoc index 73f6bd07403..8abf934423c 100644 --- a/packetbeat/docs/reference/configuration.asciidoc +++ b/packetbeat/docs/reference/configuration.asciidoc @@ -13,7 +13,6 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> -* <> * <> * <> * <> @@ -24,6 +23,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> NOTE: Packetbeat maintains a real-time topology map of all the servers in your network. See <> for more details. diff --git a/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc b/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc index e62dd3dc2cf..75bf10008e4 100644 --- a/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc +++ b/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc @@ -794,8 +794,6 @@ process' command line as read from `/proc//cmdline`. include::../../../../libbeat/docs/generalconfig.asciidoc[] -include::../../../../libbeat/docs/processors-config.asciidoc[] - include::../../../../libbeat/docs/outputconfig.asciidoc[] include::../../../../libbeat/docs/shared-path-config.asciidoc[] @@ -804,3 +802,6 @@ include::../../../../libbeat/docs/loggingconfig.asciidoc[] include::./runconfig.asciidoc[] +include::../../../../libbeat/docs/processors-config.asciidoc[] + + diff --git a/winlogbeat/docs/reference/configuration.asciidoc b/winlogbeat/docs/reference/configuration.asciidoc index 8b8dbc5f349..8794d8ce750 100644 --- a/winlogbeat/docs/reference/configuration.asciidoc +++ b/winlogbeat/docs/reference/configuration.asciidoc @@ -11,7 +11,6 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> -* <> * <> * <> * <> @@ -21,6 +20,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> include::configuration/winlogbeat-options.asciidoc[] diff --git a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc index 20e8c74ceb2..c5922243bef 100644 --- a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc +++ b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc @@ -319,10 +319,11 @@ The metrics are served as a JSON document. The metrics include: include::../../../../libbeat/docs/generalconfig.asciidoc[] -include::../../../../libbeat/docs/processors-config.asciidoc[] - include::../../../../libbeat/docs/outputconfig.asciidoc[] include::../../../../libbeat/docs/shared-path-config.asciidoc[] include::../../../../libbeat/docs/loggingconfig.asciidoc[] + +include::../../../../libbeat/docs/processors-config.asciidoc[] +