Restructure beats config options (elastic#4716) (elastic#4759)

* Restructure Filebeat and Libbeat config options * Restructure Heartbeat config options * Restructure Packetbeat config options * Experiment: show packetbeat.reference.yml in the docs * Restructure Winlogbeat config options * Restructure Metricbeat config options * Restructure Auditbeat config options * Add reference configs to the docs * Move include statement for general options for consistency
tsg · Jul 27, 2017 · fd6939a · fd6939a
1 parent e44948d
commit fd6939a
Show file tree

Hide file tree

Showing 263 changed files with 2,239 additions and 1,974 deletions.
diff --git a/auditbeat/docs/auditbeat-general-options.asciidoc b/auditbeat/docs/auditbeat-general-options.asciidoc
@@ -0,0 +1,7 @@
+[[configuration-general-options]]
+== Specify general settings
+
+You can specify settings in the +{beatname_lc}.yml+ config file to control the
+general behavior of {beatname_uc}.
+
+include::../../libbeat/docs/generalconfig.asciidoc[]
diff --git a/auditbeat/docs/auditbeat-modules-config.asciidoc b/auditbeat/docs/auditbeat-modules-config.asciidoc
@@ -1,5 +1,5 @@
 [id="configuration-{beatname_lc}"]
-=== Specify which modules to run
+== Specify which modules to run
 
 To enable specific modules and metricsets, you add entries to the
 `auditbeat.modules` list in the +{beatname_lc}.yml+ config file. Each entry in

diff --git a/auditbeat/docs/configuring-howto.asciidoc b/auditbeat/docs/configuring-howto.asciidoc
@@ -20,22 +20,15 @@ _Beats Platform Reference_ for more about the structure of the config file.
 The following topics describe how to configure {beatname_uc}:
 
 * <<configuration-{beatname_lc}>>
-* <<configuration-general>>
-* <<elasticsearch-output>>
-* <<logstash-output>>
-* <<kafka-output>>
-* <<redis-output>>
-* <<file-output>>
-* <<console-output>>
-* <<configuration-output-ssl>>
-* <<configuration-output-codec>>
+* <<configuration-general-options>>
+* <<{beatname_lc}-configuration-reloading>>
+* <<configuring-output>>
+* <<filtering-and-enhancing-data>>
+* <<configuring-ingest-node>>
 * <<configuration-path>>
 * <<configuration-dashboards>>
+* <<configuration-template>>
 * <<configuration-logging>>
-* <<configuration-processors>>
-* <<filtering-and-enhancing-data>>
-* <<configuring-ingest-node>>
-* <<config-{beatname_lc}-logstash>>
 * <<using-environ-vars>>
 * <<yaml-tips>>
 
@@ -44,26 +37,31 @@ pick up the changes.
 
 --
 
-[[configuration-container]]
-== Configuration Options
-
-This section describes configuration options.
-
-//REVIEWERS: THIS CONTAINER IS A TEMPORARY PLACEHOLDER SO I CAN GET THE AUDITBEAT DOCS BUILDING WITHOUT ERRORS. I PLAN TO REMOVE THIS CONTAINER AND POP EVERYTHING UP A LEVEL IN THE HIERARCHY TO CREATE A STRUCTURE SIMILAR TO THE ONE DESCRIBED IN THIS FILEBEAT ISSUE: https://github.com/elastic/beats/issues/4422
-
 include::./auditbeat-modules-config.asciidoc[]
 
-include::../../libbeat/docs/generalconfig.asciidoc[]
+include::./auditbeat-general-options.asciidoc[]
 
-include::../../libbeat/docs/processors-config.asciidoc[]
+include::./reload-configuration.asciidoc[]
 
+:allplatforms:
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
+include::./auditbeat-filtering.asciidoc[]
+
+include::../../libbeat/docs/shared-config-ingest.asciidoc[]
+
 include::../../libbeat/docs/shared-path-config.asciidoc[]
 
 include::../../libbeat/docs/setup-config.asciidoc[]
 
 include::../../libbeat/docs/loggingconfig.asciidoc[]
 
-include::./reload-configuration.asciidoc[]
+:standalone:
+include::../../libbeat/docs/shared-env-vars.asciidoc[]
+
+:standalone:
+:allplatforms:
+include::../../libbeat/docs/yaml.asciidoc[]
+
+include::../../libbeat/docs/reference-yml.asciidoc[]
 
diff --git a/auditbeat/docs/configuring-logstash.asciidoc b/auditbeat/docs/configuring-logstash.asciidoc
diff --git a/auditbeat/docs/getting-started.asciidoc b/auditbeat/docs/getting-started.asciidoc
@@ -176,8 +176,8 @@ output.elasticsearch:
   hosts: ["192.168.1.42:9200"]
 ----------------------------------------------------------------------
 +
-If you are sending output to Logstash, see <<config-{beatname_lc}-logstash>>
-instead.
+If you are sending output to Logstash, see
+<<logstash-output,Configure the Logstash output>> instead.
 
 . If you plan to use the sample Kibana dashboards provided with {beatname_uc},
 configure the Kibana endpoint:

diff --git a/auditbeat/docs/index.asciidoc b/auditbeat/docs/index.asciidoc
@@ -33,19 +33,6 @@ include::../../libbeat/docs/repositories.asciidoc[]
 
 include::./configuring-howto.asciidoc[]
 
-include::./auditbeat-filtering.asciidoc[]
-
-include::../../libbeat/docs/shared-config-ingest.asciidoc[]
-
-include::./configuring-logstash.asciidoc[]
-
-:standalone:
-include::../../libbeat/docs/shared-env-vars.asciidoc[]
-
-:standalone:
-:allplatforms:
-include::../../libbeat/docs/yaml.asciidoc[]
-
 include::./modules.asciidoc[]
 
 include::./fields.asciidoc[]

diff --git a/auditbeat/docs/reload-configuration.asciidoc b/auditbeat/docs/reload-configuration.asciidoc
@@ -1,5 +1,5 @@
 [id="{beatname_lc}-configuration-reloading"]
-=== Reload the configuration dynamically
+== Reload the configuration dynamically
 
 beta[]
 

diff --git a/filebeat/docs/command-line.asciidoc b/filebeat/docs/command-line.asciidoc
@@ -1,5 +1,5 @@
 [[command-line-options]]
-=== Command Line Options
+=== Command line options
 
 The following command line option is specific to Filebeat.
 

diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc
@@ -3,30 +3,73 @@
 
 [partintro]
 --
-After following the <<filebeat-configuration,configuration steps>> in the
-Getting Started, you might want to fine tune the behavior of Filebeat. This section
-describes some common use cases for changing configuration options.
+Before modifying configuration settings, make sure you've completed the
+<<filebeat-configuration,configuration steps>> in the Getting Started.
+This section describes some common use cases for changing configuration options.
 
-To configure {beatname_uc}, you edit the configuration file. For rpm and deb, you’ll find the default configuration file at
-+/etc/{beatname_lc}/{beatname_lc}.yml+. There's also a full example configuration file at
-+/etc/{beatname_lc}/{beatname_lc}.reference.yml+ that shows all non-deprecated options.  For mac and win, look in the archive that you extracted.
+To configure {beatname_uc}, you edit the configuration file. For rpm and deb,
+you’ll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+.
+There's also a full example configuration file at
++/etc/{beatname_lc}/{beatname_lc}.reference.yml+ that shows all non-deprecated
+options. For mac and win, look in the archive that you extracted.
 
-See the
-{libbeat}/config-file-format.html[Config File Format] section of the
+The {beatname_uc} configuration file uses http://yaml.org/[YAML] for its syntax.
+See the {libbeat}/config-file-format.html[Config File Format] section of the
 _Beats Platform Reference_ for more about the structure of the config file.
 
 The following topics describe how to configure Filebeat:
 
-* <<filebeat-configuration-details>>
-* <<filtering-and-enhancing-data>>
+* <<configuration-filebeat-options>>
 * <<multiline-examples>>
+* <<configuration-general-options>>
+* <<filebeat-configuration-reloading>>
+* <<configuring-output>>
+* <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<configuration-path>>
+* <<configuration-dashboards>>
+* <<configuration-template>>
+* <<configuration-logging>>
 * <<using-environ-vars>>
-* <<multiple-prospectors>>
 * <<load-balancing>>
 * <<yaml-tips>>
 * <<regexp-support>>
 
 --
 
-include::reference/configuration.asciidoc[]
+include::./filebeat-options.asciidoc[]
+
+include::./multiple-prospectors.asciidoc[]
+
+include::./multiline.asciidoc[]
+
+include::./filebeat-general-options.asciidoc[]
+
+include::./reload-configuration.asciidoc[]
+
+:allplatforms:
+include::../../libbeat/docs/outputconfig.asciidoc[]
+
+include::./load-balancing.asciidoc[]
+
+include::./filebeat-filtering.asciidoc[]
+
+include::../../libbeat/docs/shared-config-ingest.asciidoc[]
+
+include::../../libbeat/docs/shared-path-config.asciidoc[]
+
+include::../../libbeat/docs/setup-config.asciidoc[]
+
+include::../../libbeat/docs/loggingconfig.asciidoc[]
+
+:standalone:
+include::../../libbeat/docs/shared-env-vars.asciidoc[]
+
+:standalone:
+:allplatforms:
+include::../../libbeat/docs/yaml.asciidoc[]
+
+include::../../libbeat/docs/regexp.asciidoc[]
+
+include::../../libbeat/docs/reference-yml.asciidoc[]
+
diff --git a/filebeat/docs/faq.asciidoc b/filebeat/docs/faq.asciidoc
@@ -1,5 +1,5 @@
 [[faq]]
-== Frequently Asked Questions
+== Frequently asked questions
 
 This section contains frequently asked questions about Filebeat. Also check out the
 https://discuss.elastic.co/c/beats/filebeat[Filebeat discussion forum].

diff --git a/filebeat/docs/filebeat-filtering.asciidoc b/filebeat/docs/filebeat-filtering.asciidoc
@@ -1,5 +1,5 @@
 [[filtering-and-enhancing-data]]
-== Filtering and Enhancing the Exported Data
+== Filter and enhance the exported data
 
 Your use case might require only a subset of the data exported by Filebeat, or
 you might need to enhance the exported data (for example, by adding metadata).
@@ -11,9 +11,9 @@ This allows you to specify different filtering criteria for each prospector.
 To do this, you use the <<include-lines,`include_lines`>>,
 <<exclude-lines,`exclude_lines`>>, and <<exclude-files,`exclude_files`>>
 options under the `filebeat.prospectors` section of the config file (see
-<<configuration-filebeat-options,Filebeat configuration options>>). The
-disadvantage of this approach is that you need to implement a configuration
-option for each filtering criteria that you need.
+<<configuration-filebeat-options>>). The disadvantage of this approach is that
+you need to implement a configuration option for each filtering criteria that
+you need.
 
 Another approach (the one described here) is to define processors to configure
 global processing across all data exported by Filebeat.
@@ -27,7 +27,7 @@ include::../../libbeat/docs/processors.asciidoc[]
 
 [float]
 [[drop-event-example]]
-==== Drop Event Example
+==== Drop event example
 
 The following configuration drops all the DEBUG messages.
 
@@ -53,7 +53,7 @@ processors:
 
 [float]
 [[decode-json-example]]
-==== Decode JSON Example
+==== Decode JSON example
 
 In the following example, the fields exported by Filebeat include a
 field, `inner`, whose value is a JSON object encoded as a string:

diff --git a/filebeat/docs/filebeat-general-options.asciidoc b/filebeat/docs/filebeat-general-options.asciidoc
@@ -0,0 +1,121 @@
+[[configuration-general-options]]
+== Specify general settings
+
+You can specify settings in the +{beatname_lc}.yml+ config file to control the
+general behavior of {beatname_uc}. This includes:
+
+* <<configuration-global-options,Global options>> that control things like
+publisher behavior and the location of some files.
+
+* <<configuration-general,General options>> that are supported by all Elastic
+Beats.
+
+[float]
+[[configuration-global-options]]
+=== Global Filebeat configuration options
+
+These options are in the `filebeat` namespace.
+
+[float]
+==== `spool_size`
+
+The event count spool threshold. This setting forces a network flush if the number of events in the spooler exceeds
+the specified value.
+
+[source,yaml]
+-------------------------------------------------------------------------------------
+filebeat.spool_size: 2048
+-------------------------------------------------------------------------------------
+
+See <<load-balancing>> for more information about how this setting affects load balancing.
+
+[float]
+==== `publish_async`
+
+experimental[]
+deprecated[5.3.0]
+
+If enabled, the publisher pipeline in Filebeat operates in async mode preparing
+a new batch of lines while waiting for ACK. This option can improve load-balancing
+throughput at the cost of increased memory usage. The default value is false.
+
+See <<load-balancing>> for more information about how this setting affects load balancing.
+
+[float]
+==== `idle_timeout`
+
+A duration string that specifies how often the spooler is flushed. After the
+`idle_timeout` is reached, the spooler is flushed even if the `spool_size` has not been reached.
+
+[source,yaml]
+-------------------------------------------------------------------------------------
+filebeat.idle_timeout: 5s
+-------------------------------------------------------------------------------------
+
+
+[float]
+==== `registry_file`
+
+The name of the registry file. If a relative path is used, it is considered relative to the
+data path. See the <<directory-layout>> section for details. The default is `${path.data}/registry`.
+
+[source,yaml]
+-------------------------------------------------------------------------------------
+filebeat.registry_file: registry
+-------------------------------------------------------------------------------------
+
+It is not possible to use a symlink as registry file.
+
+NOTE: The registry file is only updated when new events are flushed and not on a predefined period.
+That means in case there are some states where the TTL expired, these are only removed when new event are processed.
+
+
+[float]
+==== `config_dir`
+
+The full path to the directory that contains additional prospector configuration files.
+Each configuration file must end with `.yml`. Each config file must also specify the full Filebeat
+config hierarchy even though only the prospector part of the file is processed. All global
+options, such as `spool_size`, are ignored.
+
+The `config_dir` option MUST point to a directory other than the directory where the main Filebeat config file resides.
+
+If the specified path is not absolute, it is considered relative to the configuration path. See the
+<<directory-layout>> section for details.
+
+[source,yaml]
+-------------------------------------------------------------------------------------
+filebeat.config_dir: path/to/configs
+-------------------------------------------------------------------------------------
+
+[float]
+[[shutdown-timeout]]
+==== `shutdown_timeout`
+
+How long Filebeat waits on shutdown for the publisher to finish sending events
+before Filebeat shuts down.
+
+By default, this option is disabled, and Filebeat does not wait for the
+publisher to finish sending events before shutting down. This means that any
+events sent to the output, but not acknowledged before Filebeat shuts down,
+are sent again when you restart Filebeat. For more details about how this
+works, see <<at-least-once-delivery>>.
+
+You can configure the `shutdown_timeout` option to specify the maximum amount
+of time that Filebeat waits for the publisher to finish sending events before
+shutting down. If all events are acknowledged before `shutdown_timeout` is
+reached, Filebeat will shut down.
+
+There is no recommended setting for this option because determining the correct
+value for `shutdown_timeout` depends heavily on the environment in which
+Filebeat is running and the current state of the output.
+
+Example configuration:
+
+[source,yaml]
+-------------------------------------------------------------------------------------
+filebeat.shutdown_timeout: 5s
+-------------------------------------------------------------------------------------
+
+include::../../libbeat/docs/generalconfig.asciidoc[]
+