Merge pull request elastic#4704 from tsg/automatic_merge_from_5.5_to_…

…5.x_branch

Automatic merge from 5.5 to 5.x branch

filebeat/docs/command-line.asciidoc

@@ -1,4 +1,4 @@
-[[filebeat-command-line]]
+[[command-line-options]]
 === Command Line Options
 
 The following command line option is specific to Filebeat.

filebeat/docs/getting-started.asciidoc

@@ -18,7 +18,7 @@ After installing the Elastic Stack, read the following topics to learn how to in
 * <<filebeat-starting>>
 * <<filebeat-index-pattern>>
 * <<filebeat-modules-quickstart>>
-* <<filebeat-command-line>>
+* <<command-line-options>>
 * <<directory-layout>>
 
 [[filebeat-installation]]
@@ -27,17 +27,7 @@ After installing the Elastic Stack, read the following topics to learn how to in
 Before running Filebeat, you need to install and configure the Elastic stack. See
 {libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack].
 
-To download and install Filebeat, use the commands that work with your system
-(<<deb, deb>> for Debian/Ubuntu, <<rpm, rpm>> for Redhat/Centos/Fedora, <<mac,
-mac>> for OS X, and <<win, win>> for Windows).
-
-[NOTE]
-==================================================
-If you use Apt or Yum, you can <<setup-repositories,install Filebeat from our repositories>> to update to the newest version more easily.
-
-See our https://www.elastic.co/downloads/beats/filebeat[download page] for other installation options, such as 32-bit images.
-
-==================================================
+include::../../libbeat/docs/shared-download-and-install.asciidoc[]
 
 [[deb]]
 *deb:*
@@ -96,6 +86,24 @@ tar xzvf filebeat-{version}-darwin-x86_64.tar.gz
 
 endif::[]
 
+[[docker]]
+*docker:*
+
+ifeval::["{release-state}"=="unreleased"]
+
+Version {stack-version} of {beatname_uc} has not yet been released.
+
+endif::[]
+
+ifeval::["{release-state}"!="unreleased"]
+
+["source", "shell", subs="attributes"]
+------------------------------------------------
+docker pull {dockerimage}
+------------------------------------------------
+
+endif::[]
+
 [[win]]
 *win:*
 
@@ -139,15 +147,7 @@ started, you can skip the content in this section, including the remaining
 getting started steps, and go directly to the <<filebeat-modules-quickstart>>
 page. 
 
-To configure Filebeat manually, you edit the configuration file. For rpm and deb,
-you'll find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and
-win, look in the archive that you just extracted. There’s also a full example
-configuration file called `filebeat.full.yml` that shows all non-deprecated
-options.
-
-See the
-{libbeat}/config-file-format.html[Config File Format] section of the
-_Beats Platform Reference_ for more about the structure of the config file.
+include::../../libbeat/docs/shared-configuring.asciidoc[]
 
 Here is a sample of the `filebeat` section of the `filebeat.yml` file. Filebeat uses predefined
 default values for most configuration options.
@@ -223,7 +223,7 @@ include::../../libbeat/docs/shared-template-load.asciidoc[]
 Start Filebeat by issuing the appropriate command for your platform. 
 
 NOTE: If you use an init.d script to start Filebeat on deb or rpm, you can't
-specify command line flags (see <<filebeat-command-line>>). To specify flags,
+specify command line flags (see <<command-line-options>>). To specify flags,
 start Filebeat in the foreground.
 
 *deb:*
@@ -240,6 +240,13 @@ sudo /etc/init.d/filebeat start
 sudo /etc/init.d/filebeat start
 ----------------------------------------------------------------------
 
+*docker:*
+
+["source", "shell", subs="attributes"]
+----------------------------------------------------------------------
+docker run {dockerimage}
+----------------------------------------------------------------------
+
 *mac:*
 
 [source,shell]

filebeat/docs/index.asciidoc

@@ -15,6 +15,7 @@ include::../../libbeat/docs/version.asciidoc[]
 :beatname_lc: filebeat
 :beatname_uc: Filebeat
 :security: X-Pack Security
+:dockerimage: docker.elastic.co/beats/{beatname_lc}:{version}
 
 include::./overview.asciidoc[]
 
@@ -28,6 +29,8 @@ include::../../libbeat/docs/shared-directory-layout.asciidoc[]
 
 include::../../libbeat/docs/repositories.asciidoc[]
 
+include::./running-on-docker.asciidoc[]
+
 include::./upgrading.asciidoc[]
 
 include::./how-filebeat-works.asciidoc[]
@@ -40,6 +43,7 @@ include::./multiline.asciidoc[]
 
 include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]

filebeat/docs/migration.asciidoc

@@ -304,7 +304,7 @@ options with Logstash Forwarder, make sure that you add your options to the
 configuration file. For naming changes, see <<renamed-options>>.
 
 Filebeat does provide command line options that are common to all Beats. For more details about
-these options, see <<filebeat-command-line>>.
+these options, see <<command-line-options>>.
 
 [[renamed-options]]
 [float]

filebeat/docs/modules-getting-started.asciidoc

@@ -15,7 +15,7 @@ modules, see <<filebeat-modules>>.
 
 If you are using a log file type that isn't supported by one of the available
 Filebeat modules, you'll need to set up and configure Filebeat manually by
-following the numbered steps under <<filebeat-getting-started>>. 
+following the numbered steps under <<filebeat-getting-started>>.
 
 ==== Prerequisites
 
@@ -116,4 +116,4 @@ Open the dashboard and explore the visualizations for your parsed logs.
 
 Here's an example of the syslog dashboard:
 
-image:./images/kibana-system.png[Sylog dashboard]
+image:./images/kibana-system.png[Syslog dashboard]

filebeat/docs/modules-overview.asciidoc

@@ -32,6 +32,8 @@ NOTE: At the moment, Filebeat modules require using the Elasticsearch
 be able to also configure Logstash as a more powerful alternative to Ingest
 Node.
 
+Filebeat modules require Elasticsearch 5.2 or later.
+
 === Tutorial
 
 This tutorial assumes you have Elasticsearch and Kibana installed and

filebeat/docs/modules.asciidoc

@@ -7,6 +7,8 @@ This section contains an <<filebeat-modules-overview,overview>> of the Filebeat
 modules feature as well as details about each of the currently supported
 modules.
 
+Filebeat modules require Elasticsearch 5.2 or later.
+
 //pass macro block used here to remove Edit links from modules documentation because it is generated
 pass::[<?edit_url?>]
 include::modules_list.asciidoc[]

filebeat/docs/multiline.asciidoc

@@ -3,26 +3,26 @@
 
 The files harvested by {beatname_uc} may contain messages that span multiple lines of text. In order to correctly handle
 these multiline events, you need to configure `multiline` settings in the +{beatname_lc}.yml+ file to specify which
-lines are part of a single event. 
+lines are part of a single event.
 
 IMPORTANT: If you are sending multiline events to Logstash, use the options described here to handle multiline events
 before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by
-using the Logstash multiline codec) may result in the mixing of streams and corrupted data. 
+using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
 
 At a minimum, you need to configure these `multiline` options:
 
-* the `pattern` option, which specifies a regular expression. Depending on how you configure other multiline options, 
+* the `pattern` option, which specifies a regular expression. Depending on how you configure other multiline options,
 lines that match the specified regular expression are considered either continuations of a previous line or the start of a new multiline event. You can set the `negate` option to negate the pattern.
 
 * the `match` option, which specifies how Filebeat combines matching lines into an event. You can specify `before` or `after`.
 
-See the full documentation for <<multiline>> to learn more about these options. Also read <<yaml-tips>> and 
+See the full documentation for <<multiline>> to learn more about these options. Also read <<yaml-tips>> and
 <<regexp-support>> to avoid common mistakes.
 
 [float]
 === Testing Your Regexp Pattern for Multiline
 
-To make it easier for you to test the regexp patterns in your multiline config, we've created a 
+To make it easier for you to test the regexp patterns in your multiline config, we've created a
 https://play.golang.org/p/uAd5XHxscu[Go Playground]. You can simply plug in the regexp pattern along with
 the `negate` setting that you plan to use, and paste a sample message between the content backticks (` `).
 Then click Run, and you'll see which lines in the message match your specified configuration. For example:
@@ -63,7 +63,7 @@ multiline.match: after
 
 This configuration merges any line that begins with whitespace up to the previous line.
 
-Here's a Java stack trace that presents a slightly more complex example: 
+Here's a Java stack trace that presents a slightly more complex example:
 
 ["source","sh",subs="attributes,callouts"]
 -------------------------------------------------------------------------------------
@@ -93,7 +93,7 @@ In this example, the pattern matches the following lines:
 [float]
 ==== Line Continuations
 
-Several programming languages use the backslash (`\`) character at the end of a line to denote that the line continues, 
+Several programming languages use the backslash (`\`) character at the end of a line to denote that the line continues,
 as in this example:
 
 [source,c]
@@ -134,7 +134,7 @@ multiline.negate: true
 multiline.match: after
 -------------------------------------------------------------------------------------
 
-This configuration uses the `negate: true` and `match: after` settings to specify that any line that does not match the 
+This configuration uses the `negate: true` and `match: after` settings to specify that any line that does not match the
 specified pattern belongs to the previous line.
 
 

filebeat/docs/reference/configuration/filebeat-options.asciidoc

@@ -390,11 +390,11 @@ somewhat from the patterns supported by Logstash. See <<regexp-support>> for a l
 +
 [options="header"]
 |=======================
-|Setting for `negate` | Setting for `match` | Result
-|`false`              | `after`             | Consecutive lines that match the pattern are appended to the previous line that doesn't match.
-|`false`              | `before`            | Consecutive lines that match the pattern are prepended to the next line that doesn't match.
-|`true`               | `after`             | Consecutive lines that don't match the pattern are appended to the previous line that does match.
-|`true`               | `before`            | Consecutive lines that don't match the pattern are prepended to the next line that does match.
+|Setting for `negate` | Setting for `match` | Result | Example `pattern: ^b`
+|`false`              | `after`             | Consecutive lines that match the pattern are appended to the previous line that doesn't match. | image:./images/false-after-multi.png[Lines a b b c b b become "abb" and "cbb"]
+|`false`              | `before`            | Consecutive lines that match the pattern are prepended to the next line that doesn't match. | image:./images/false-before-multi.png[Lines b b a b b c become "bba" and "bbc"]
+|`true`               | `after`             | Consecutive lines that don't match the pattern are appended to the previous line that does match. | image:./images/true-after-multi.png[Lines b a c b d e become "bac" and "bde"]
+|`true`               | `before`            | Consecutive lines that don't match the pattern are prepended to the next line that does match. | image:./images/true-before-multi.png[Lines a c b d e b become "acb" and "deb"]
 |=======================
 +
 NOTE: The `after` setting is equivalent to `previous` in https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html[Logstash], and `before` is equivalent to `next`.
@@ -433,7 +433,7 @@ Because this option may lead to data loss, it is disabled by default.
 
 ===== backoff
 
-The backoff options specify how aggressively Filebeat crawls new files for updates.
+The backoff options specify how aggressively Filebeat crawls open files for updates.
 You can use the default values in most cases.
 
 The `backoff` option defines how long Filebeat

filebeat/docs/running-on-docker.asciidoc

@@ -0,0 +1 @@
+include::../../libbeat/docs/shared-docker.asciidoc[]

filebeat/filebeat.template-es2x.json

@@ -303,9 +303,6 @@
           "index": "not_analyzed",
           "type": "string"
         },
-        "fields": {
-          "properties": {}
-        },
         "fileset": {
           "properties": {
             "module": {

filebeat/filebeat.template-es6x.json

@@ -253,9 +253,6 @@
           "ignore_above": 1024,
           "type": "keyword"
         },
-        "fields": {
-          "properties": {}
-        },
         "fileset": {
           "properties": {
             "module": {

filebeat/filebeat.template.json

@@ -256,9 +256,6 @@
           "ignore_above": 1024,
           "type": "keyword"
         },
-        "fields": {
-          "properties": {}
-        },
         "fileset": {
           "properties": {
             "module": {

glide.yaml

@@ -25,7 +25,7 @@ import:
   subpackages:
   - /difflib
 - package: github.com/elastic/gosigar
-  version: v0.2.0
+  version: v0.2.1
 - package: github.com/elastic/procfs
   version: abf152e5f3e97f2fafac028d2cc06c1feb87ffa5
 - package: github.com/samuel/go-parser

heartbeat/docs/command-line.asciidoc

@@ -1,4 +1,4 @@
-[[heartbeat-command-line]]
+[[command-line-options]]
 === Command Line Options
 
 Heartbeat does not have any Heartbeat-specific command line options.

heartbeat/docs/getting-started.asciidoc

@@ -18,7 +18,7 @@ install, configure, and run Heartbeat:
 * <<heartbeat-configuration>>
 * <<heartbeat-template>>
 * <<heartbeat-starting>>
-* <<heartbeat-command-line>>
+* <<command-line-options>>
 * <<directory-layout>>
 
 
@@ -32,17 +32,7 @@ monitor are running.
 
 //TODO: Add a separate topic that explores deployment scenarios in more detail (like installing on a sub-network where there's a firewall etc.
 
-To download and install Heartbeat, use the commands that work with your
-system (<<deb, deb>> for Debian/Ubuntu, <<rpm, rpm>> for Redhat/Centos/Fedora,
-<<mac,mac>> for OS X, and <<win, win>> for Windows).
-
-[NOTE]
-==================================================
-If you use Apt or Yum, you can <<setup-repositories,install Heartbeat from our repositories>> to update to the newest version more easily.
-
-See our https://www.elastic.co/downloads/beats/heartbeat[download page] for other installation options, such as 32-bit images.
-
-==================================================
+include::../../libbeat/docs/shared-download-and-install.asciidoc[]
 
 [[deb]]
 *deb:*
@@ -101,6 +91,25 @@ tar xzvf heartbeat-{version}-darwin-x86_64.tar.gz
 
 endif::[]
 
+
+[[docker]]
+*docker:*
+
+ifeval::["{release-state}"=="unreleased"]
+
+Version {stack-version} of {beatname_uc} has not yet been released.
+
+endif::[]
+
+ifeval::["{release-state}"!="unreleased"]
+
+["source", "shell", subs="attributes"]
+------------------------------------------------
+docker pull {dockerimage}
+------------------------------------------------
+
+endif::[]
+
 [[win]]
 *win:*
 
@@ -147,15 +156,7 @@ options, see <<heartbeat-configuration-details>>.
 [[heartbeat-configuration]]
 === Step 2: Configuring Heartbeat
 
-To configure Heartbeat, you edit the configuration file. For rpm and deb,
-you'll find the configuration file at +/etc/heartbeat/heartbeat.yml+.
-For mac and win, look in the archive that you just extracted. There’s also a
-full example configuration file called `heartbeat.full.yml` that shows all
-non-deprecated options.
-
-See the
-{libbeat}/config-file-format.html[Config File Format] section of the
-_Beats Platform Reference_ for more about the structure of the config file.
+include::../../libbeat/docs/shared-configuring.asciidoc[]
 
 Heartbeat provides monitors to check the status of hosts at set intervals.
 You configure each monitor individually. Heartbeat currently provides monitors
@@ -233,7 +234,7 @@ include::../../libbeat/docs/shared-template-load.asciidoc[]
 Start Heartbeat by issuing the appropriate command for your platform.
 
 NOTE: If you use an init.d script to start Heartbeat on deb or rpm, you can't
-specify command line flags (see <<heartbeat-command-line>>). To specify flags,
+specify command line flags (see <<command-line-options>>). To specify flags,
 start Heartbeat in the foreground.
 
 *deb:*