Backport into 5.5: #4200, #4312, #4405, #4408, #4505

filebeat/docs/command-line.asciidoc

@@ -1,4 +1,4 @@
-[[filebeat-command-line]]
+[[command-line-options]]
 === Command Line Options
 
 The following command line option is specific to Filebeat.

filebeat/docs/getting-started.asciidoc

@@ -18,7 +18,7 @@ After installing the Elastic Stack, read the following topics to learn how to in
 * <<filebeat-starting>>
 * <<filebeat-index-pattern>>
 * <<filebeat-modules-quickstart>>
-* <<filebeat-command-line>>
+* <<command-line-options>>
 * <<directory-layout>>
 
 [[filebeat-installation]]
@@ -27,17 +27,7 @@ After installing the Elastic Stack, read the following topics to learn how to in
 Before running Filebeat, you need to install and configure the Elastic stack. See
 {libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack].
 
-To download and install Filebeat, use the commands that work with your system
-(<<deb, deb>> for Debian/Ubuntu, <<rpm, rpm>> for Redhat/Centos/Fedora, <<mac,
-mac>> for OS X, and <<win, win>> for Windows).
-
-[NOTE]
-==================================================
-If you use Apt or Yum, you can <<setup-repositories,install Filebeat from our repositories>> to update to the newest version more easily.
-
-See our https://www.elastic.co/downloads/beats/filebeat[download page] for other installation options, such as 32-bit images.
-
-==================================================
+include::../../libbeat/docs/shared-download-and-install.asciidoc[]
 
 [[deb]]
 *deb:*
@@ -96,6 +86,24 @@ tar xzvf filebeat-{version}-darwin-x86_64.tar.gz
 
 endif::[]
 
+[[docker]]
+*docker:*
+
+ifeval::["{release-state}"=="unreleased"]
+
+Version {stack-version} of {beatname_uc} has not yet been released.
+
+endif::[]
+
+ifeval::["{release-state}"!="unreleased"]
+
+["source", "shell", subs="attributes"]
+------------------------------------------------
+docker pull {dockerimage}
+------------------------------------------------
+
+endif::[]
+
 [[win]]
 *win:*
 
@@ -139,15 +147,7 @@ started, you can skip the content in this section, including the remaining
 getting started steps, and go directly to the <<filebeat-modules-quickstart>>
 page. 
 
-To configure Filebeat manually, you edit the configuration file. For rpm and deb,
-you'll find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and
-win, look in the archive that you just extracted. There’s also a full example
-configuration file called `filebeat.full.yml` that shows all non-deprecated
-options.
-
-See the
-{libbeat}/config-file-format.html[Config File Format] section of the
-_Beats Platform Reference_ for more about the structure of the config file.
+include::../../libbeat/docs/shared-configuring.asciidoc[]
 
 Here is a sample of the `filebeat` section of the `filebeat.yml` file. Filebeat uses predefined
 default values for most configuration options.
@@ -223,7 +223,7 @@ include::../../libbeat/docs/shared-template-load.asciidoc[]
 Start Filebeat by issuing the appropriate command for your platform. 
 
 NOTE: If you use an init.d script to start Filebeat on deb or rpm, you can't
-specify command line flags (see <<filebeat-command-line>>). To specify flags,
+specify command line flags (see <<command-line-options>>). To specify flags,
 start Filebeat in the foreground.
 
 *deb:*
@@ -240,6 +240,13 @@ sudo /etc/init.d/filebeat start
 sudo /etc/init.d/filebeat start
 ----------------------------------------------------------------------
 
+*docker:*
+
+["source", "shell", subs="attributes"]
+----------------------------------------------------------------------
+docker run {dockerimage}
+----------------------------------------------------------------------
+
 *mac:*
 
 [source,shell]

filebeat/docs/index.asciidoc

@@ -15,6 +15,7 @@ include::../../libbeat/docs/version.asciidoc[]
 :beatname_lc: filebeat
 :beatname_uc: Filebeat
 :security: X-Pack Security
+:dockerimage: docker.elastic.co/beats/{beatname_lc}:{version}
 
 include::./overview.asciidoc[]
 
@@ -28,6 +29,8 @@ include::../../libbeat/docs/shared-directory-layout.asciidoc[]
 
 include::../../libbeat/docs/repositories.asciidoc[]
 
+include::./running-on-docker.asciidoc[]
+
 include::./upgrading.asciidoc[]
 
 include::./how-filebeat-works.asciidoc[]
@@ -40,6 +43,7 @@ include::./multiline.asciidoc[]
 
 include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]

filebeat/docs/migration.asciidoc

@@ -304,7 +304,7 @@ options with Logstash Forwarder, make sure that you add your options to the
 configuration file. For naming changes, see <<renamed-options>>.
 
 Filebeat does provide command line options that are common to all Beats. For more details about
-these options, see <<filebeat-command-line>>.
+these options, see <<command-line-options>>.
 
 [[renamed-options]]
 [float]

filebeat/docs/reference/configuration/filebeat-options.asciidoc

@@ -390,11 +390,11 @@ somewhat from the patterns supported by Logstash. See <<regexp-support>> for a l
 +
 [options="header"]
 |=======================
-|Setting for `negate` | Setting for `match` | Result
-|`false`              | `after`             | Consecutive lines that match the pattern are appended to the previous line that doesn't match.
-|`false`              | `before`            | Consecutive lines that match the pattern are prepended to the next line that doesn't match.
-|`true`               | `after`             | Consecutive lines that don't match the pattern are appended to the previous line that does match.
-|`true`               | `before`            | Consecutive lines that don't match the pattern are prepended to the next line that does match.
+|Setting for `negate` | Setting for `match` | Result | Example `pattern: ^b`
+|`false`              | `after`             | Consecutive lines that match the pattern are appended to the previous line that doesn't match. | image:./images/false-after-multi.png[Lines a b b c b b become "abb" and "cbb"]
+|`false`              | `before`            | Consecutive lines that match the pattern are prepended to the next line that doesn't match. | image:./images/false-before-multi.png[Lines b b a b b c become "bba" and "bbc"]
+|`true`               | `after`             | Consecutive lines that don't match the pattern are appended to the previous line that does match. | image:./images/true-after-multi.png[Lines b a c b d e become "bac" and "bde"]
+|`true`               | `before`            | Consecutive lines that don't match the pattern are prepended to the next line that does match. | image:./images/true-before-multi.png[Lines a c b d e b become "acb" and "deb"]
 |=======================
 +
 NOTE: The `after` setting is equivalent to `previous` in https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html[Logstash], and `before` is equivalent to `next`.

filebeat/docs/running-on-docker.asciidoc

@@ -0,0 +1 @@
+include::../../libbeat/docs/shared-docker.asciidoc[]

heartbeat/docs/command-line.asciidoc

@@ -1,4 +1,4 @@
-[[heartbeat-command-line]]
+[[command-line-options]]
 === Command Line Options
 
 Heartbeat does not have any Heartbeat-specific command line options.

heartbeat/docs/getting-started.asciidoc

@@ -18,7 +18,7 @@ install, configure, and run Heartbeat:
 * <<heartbeat-configuration>>
 * <<heartbeat-template>>
 * <<heartbeat-starting>>
-* <<heartbeat-command-line>>
+* <<command-line-options>>
 * <<directory-layout>>
 
 
@@ -32,17 +32,7 @@ monitor are running.
 
 //TODO: Add a separate topic that explores deployment scenarios in more detail (like installing on a sub-network where there's a firewall etc.
 
-To download and install Heartbeat, use the commands that work with your
-system (<<deb, deb>> for Debian/Ubuntu, <<rpm, rpm>> for Redhat/Centos/Fedora,
-<<mac,mac>> for OS X, and <<win, win>> for Windows).
-
-[NOTE]
-==================================================
-If you use Apt or Yum, you can <<setup-repositories,install Heartbeat from our repositories>> to update to the newest version more easily.
-
-See our https://www.elastic.co/downloads/beats/heartbeat[download page] for other installation options, such as 32-bit images.
-
-==================================================
+include::../../libbeat/docs/shared-download-and-install.asciidoc[]
 
 [[deb]]
 *deb:*
@@ -101,6 +91,25 @@ tar xzvf heartbeat-{version}-darwin-x86_64.tar.gz
 
 endif::[]
 
+
+[[docker]]
+*docker:*
+
+ifeval::["{release-state}"=="unreleased"]
+
+Version {stack-version} of {beatname_uc} has not yet been released.
+
+endif::[]
+
+ifeval::["{release-state}"!="unreleased"]
+
+["source", "shell", subs="attributes"]
+------------------------------------------------
+docker pull {dockerimage}
+------------------------------------------------
+
+endif::[]
+
 [[win]]
 *win:*
 
@@ -147,15 +156,7 @@ options, see <<heartbeat-configuration-details>>.
 [[heartbeat-configuration]]
 === Step 2: Configuring Heartbeat
 
-To configure Heartbeat, you edit the configuration file. For rpm and deb,
-you'll find the configuration file at +/etc/heartbeat/heartbeat.yml+.
-For mac and win, look in the archive that you just extracted. There’s also a
-full example configuration file called `heartbeat.full.yml` that shows all
-non-deprecated options.
-
-See the
-{libbeat}/config-file-format.html[Config File Format] section of the
-_Beats Platform Reference_ for more about the structure of the config file.
+include::../../libbeat/docs/shared-configuring.asciidoc[]
 
 Heartbeat provides monitors to check the status of hosts at set intervals.
 You configure each monitor individually. Heartbeat currently provides monitors
@@ -233,7 +234,7 @@ include::../../libbeat/docs/shared-template-load.asciidoc[]
 Start Heartbeat by issuing the appropriate command for your platform.
 
 NOTE: If you use an init.d script to start Heartbeat on deb or rpm, you can't
-specify command line flags (see <<heartbeat-command-line>>). To specify flags,
+specify command line flags (see <<command-line-options>>). To specify flags,
 start Heartbeat in the foreground.
 
 *deb:*

heartbeat/docs/index.asciidoc

@@ -15,6 +15,7 @@ include::../../libbeat/docs/version.asciidoc[]
 :beatname_lc: heartbeat
 :beatname_uc: Heartbeat
 :security: X-Pack Security
+:dockerimage: docker.elastic.co/beats/{beatname_lc}:{version}
 
 include::./overview.asciidoc[]
 
@@ -26,6 +27,8 @@ include::../../libbeat/docs/shared-directory-layout.asciidoc[]
 
 include::../../libbeat/docs/repositories.asciidoc[]
 
+include::./running-on-docker.asciidoc[]
+
 //
 //include::./upgrading.asciidoc[]
 
@@ -41,6 +44,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 //points to shared topic because configuring-logstash.asciidoc is just a wrapper
 include::./configuring-logstash.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 :standalone:
@@ -57,5 +61,5 @@ include::./troubleshooting.asciidoc[]
 
 include::./faq.asciidoc[]
 
-// 
+//
 //include::./heartbeat-devguide.asciidoc[]

heartbeat/docs/running-on-docker.asciidoc

@@ -0,0 +1 @@
+include::../../libbeat/docs/shared-docker.asciidoc[]

libbeat/docs/config-file-format.asciidoc

@@ -110,7 +110,7 @@ Simple filebeat example with partially collapsed setting names and use of compac
 
 filebeat.prospectors:
 - input_type: log
-  paths: ["/var/log/*.log"] 
+  paths: ["/var/log/*.log"]
   multiline.pattern: '^['
   multiline.match: after
 
@@ -213,44 +213,7 @@ format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
 [[config-file-format-env-vars]]
 === Environment Variables
 
-Beats support use of environment variables in config files to set values that
-need to be configurable during deployment. Environment variable expansion is
-introduced using `${VAR}`, where `VAR` is the name of the environment variable.
-
-Note: Only values can be set using environment variables. Environment variables
-usage in namespace and setting names are not supported.
-
-Variable references are replaced when settings are read by beats. The
-replacement is case-sensitive and occurs after the YAML file itself has been
-parsed. References to undefined variables will lead to errors when dereferenced
-and no default value is specified. To specify a default value, use:
-
-`${VAR:default_value}`
-
-Where `default_value` is the value to use if the environment variable is
-undefined.
-
-If you need to use a literal `${` in your configuration file then you can write
-`$${` to escape the expansion. The `$` symbol can be used to escape other
-characters in the default_value like using `$}` in order to generate a `}`
-character without closing the variable expansion.
-
-After changing the value of an environment variable, the beat needs to be
-restarted to pick up the new value.
-
-[float]
-==== Examples
-
-Here are some examples of configurations that use environment variables
-and what each configuration looks like after replacement:
-
-[options="header"]
-|==================================
-|Config source	       |Environment setting   |Config after replacement
-|`name: ${NAME}`       |`export NAME=elastic` |`name: elastic`
-|`name: ${NAME:beats}` |no setting            |`name: beats`
-|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
-|==================================
+include::shared-env-vars.asciidoc[]
 
 [[config-gile-format-refs]]
 === Reference Variables
@@ -333,15 +296,15 @@ When installed via an RPM or DEB package, the config file at
 `/etc/{beatname}/{beatname}.yml` will have the proper owner and permissions. The
 file is owned by `root` and has file permissions of `0644` (`-rw-r--r--`).
 
-You may encounter the following errors if your config file fails these checks: 
+You may encounter the following errors if your config file fails these checks:
 
 ["source","sh"]
 --------------------------------------------------------------------------------
 Exiting: error loading config file: config file ("{beatname}.yml") must be
 owned by the beat user (uid=501) or root
 --------------------------------------------------------------------------------
 
-To correct this problem you can use either `chown root {beatname}.yml` or 
+To correct this problem you can use either `chown root {beatname}.yml` or
 `chown 501 {beatname}.yml` to change the owner of the configuration file.
 
 ["source","sh"]
@@ -370,7 +333,43 @@ use this, for example, for setting defaults in a base configuration file, and
 overwrite settings via local configuration files.
 
 In addition to overwriting settings using multiple configuration files,
-individual settings can be overwritten using `-E <setting>=<value>`.
+individual settings can be overwritten using `-E <setting>=<value>`. The
+`<value>` can be either a single value or a complex object, such as a list or
+dictionary.
+
+For example, given the following configuration:
+
+["source","yaml"]
+--------------------------------------------------------------------------------
+output.elasticsearch:
+  hosts: ["http://localhost:9200"]
+  username: username
+  password: password
+--------------------------------------------------------------------------------
+
+You can disable the Elasticsearch output and write all events to the console by
+setting:
+
+["source","sh"]
+--------------------------------------------------------------------------------
+-E output='{elasticsearch.enabled: false, console.pretty: true}'
+--------------------------------------------------------------------------------
+
+Any complex objects that you specify at the command line are merged with the
+original configuration, and the following configuration is passed to the Beat:
+
+["source","yaml"]
+--------------------------------------------------------------------------------
+output.elasticsearch:
+  enabled: false
+  hosts: ["http://localhost:9200"]
+  username: username
+  password: password
+
+output.console:
+  pretty: true
+--------------------------------------------------------------------------------
+
 
 [[config-file-format-tips]]
 === YAML Tips and Gotchas