elastic · tsg · Sep 13, 2016 · Sep 10, 2016
diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc
@@ -7,10 +7,20 @@ After following the <<filebeat-configuration,configuration steps>> in the
 Getting Started, you might want to fine tune the behavior of Filebeat. This section
 describes some common use cases for changing configuration options. 
 
-To configure {beatname_uc}, you edit the configuration file. For rpm and deb, you’ll find the configuration file at
-+/etc/{beatname_lc}/{beatname_lc}.yml+. For mac and win, look in the archive that you extracted.
+To configure {beatname_uc}, you edit the configuration file. For rpm and deb, you’ll find the default configuration file at
++/etc/{beatname_lc}/{beatname_lc}.yml+. There's also a full example configuration file at
++/etc/{beatname_lc}/{beatname_lc}.full.yml+ that shows all non-deprecated options.  For mac and win, look in the archive that you extracted.
 
-For a complete description of all Filebeat configuration options, see <<filebeat-configuration-details>>.
+The following topics describe how to configure Filebeat:
+
+* <<filebeat-configuration-details>>
+* <<filtering-and-enhancing-data>>
+* <<multiline-examples>>
+* <<configuring-ingest-node>>
+* <<using-environ-vars>>
+* <<multiple-prospectors>>
+* <<yaml-tips>>
+* <<regexp-support>>
 
 --
 

diff --git a/filebeat/docs/faq.asciidoc b/filebeat/docs/faq.asciidoc
@@ -31,15 +31,57 @@ it's publishing events successfully:
 ./filebeat -c config.yml -e -d "*"
 ----------------------------------------------------------------------
 
+[float]
+[[open-file-handlers]]
+== Too many open file handlers?
+
+Filebeat keeps the file handler open in case it reaches the end of a file to read new log lines in near real time. If filebeat is harvesting a large number of files, the number of open files can be become an issue. In most environments, the number of files which are actively updated is low. The configuration `close_inactive` should be set accordingly to close files which are not active any more.
+
+There are 4 more configuration options which can be used to close file handlers, but all of them should be used carefully as they can side affects. The options are:
+
+* close_renamed
+* close_removed
+* close_eof
+* close_timeout
+
+`close_renamed` and `close_removed` can be useful on Windows and issues related to file rotation, see <<windows-file-rotation>>. `close_eof` can be useful in environments with a large number of files with only very few entries. `close_timeout` in environments where it is more important to close file handlers then to send all log lines. More details can be found in config options, see <<configuration-filebeat-options>>.
+
+Before using any of these variables, make sure to study the documentation on each.
+
+[float]
+[[reduce-registry-size]]
+=== Registry file is too large?
+
+Filebeat keeps all states of the files and persists the states on disk in the `registry_file`. The states are used to continue file reading at a previous position in case filebeat is restarted. In case every day a large amount of new files is constantly produced, the registry file grows over time. To reduce the size of the registry file, there are two configuration variables: `clean_removed` and `close_inactive`.
+
+In case old files are not touched anymore and fall under `ignore_older`, it is recommended to use `clean_inactive`. If on the other size old files get removed from disk `clean_removed` can be used.
+
+[float]
+[[inode-reuse-issue]]
+=== Inode reuse causes Filebeat to skip lines?
+
+Filebeat uses under linux inode and device to identify files. In case a file is removed from disk, the inode can again be assigned to a new file. In the case of file rotation where and old file is removed and a new one is directly created afterwards, it can happen that the new files has the exact same inode. In this case, Filebeat assumes that the new file is the same as the old and tries to continue reading at the old position which is not correct.
+
+By default states are never removed from the registry file. In case of inode reuse issue it is recommended to use the `clean_*` options, especially `clean_inactive`. In case your files get rotated every 24 hours and the rotated files rotated files are not updated anymore, `ignore_older` could be set to 48 hours and `clean_inactive` 72 hours.
+
+`clean_removed` can be used for files that are removed from disk. Be aware that `clean_removed` also applies if during one scan a file cannot be found anymore. In case the file shows up at a later stage again, it will be sent again from scratch.
+
+[float]
+[[windows-file-rotation]]
+=== Open file handlers cause issues with Windows file rotation?
+
+Under Windows it can happen, that files cannot be renamed / removed as long as filebeat keeps the file handler open. This can lead to issues with the file rotating system. To reduce this issue, the options `close_removed` and `close_renamed` can be used together.
+
+It is important to understand, that these two options mean files are closed before the harvester finished reading the file. In case the file cannot be picked up again by the prospector and the harvester didn't finish reading the file, the missing lines will never be sent to elasticsearch.
+
+
 [float]
 [[filebeat-cpu]]
 === Filebeat is using too much CPU?
 
 Filebeat might be configured to scan for files too frequently. Check the setting for `scan_frequency` in the `filebeat.yml`
 config file. Setting `scan_frequency` to less than 1s may cause Filebeat to scan the disk in a tight loop.
 
-//=== Why is Filebeat keeping old files open?
-
 [float]
 [[dashboard-fields-incorrect-filebeat]]
 === Dashboard in Kibana is breaking up data fields incorrectly?

diff --git a/filebeat/docs/getting-started.asciidoc b/filebeat/docs/getting-started.asciidoc
@@ -9,7 +9,17 @@ To get started with your own Filebeat setup, install and configure these related
 
 See {libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack] for more information.
 
-After you have installed these products, you can start <<filebeat-installation>>.
+After installing the Elastic Stack, read the following topics to learn how to install, configure, and run Filebeat:
+
+* <<filebeat-installation>>
+* <<filebeat-configuration>>
+* <<config-filebeat-logstash>>
+* <<filebeat-template>>
+* <<filebeat-starting>>
+* <<filebeat-index-pattern>>
+* <<filebeat-command-line>>
+* <<directory-layout>>
+
 
 [[filebeat-installation]]
 === Step 1: Installing Filebeat
@@ -147,6 +157,7 @@ include::../../libbeat/docs/shared-logstash-config.asciidoc[]
 :allplatforms:
 include::../../libbeat/docs/shared-template-load.asciidoc[]
 
+[[filebeat-starting]]
 === Step 5: Starting Filebeat
 
 Start Filebeat by issuing the appropriate command for your platform.
@@ -183,6 +194,7 @@ By default, Windows log files are stored in `C:\ProgramData\filebeat\Logs`.
 
 Filebeat is now ready to send log files to your defined output.
 
+[[filebeat-index-pattern]]
 === Step 6: Loading the Kibana Index Pattern
 
 We don't offer prebuilt dashboards for visualizing Filebeat data. However, to make it easier for you to explore Filebeat

diff --git a/filebeat/docs/index.asciidoc b/filebeat/docs/index.asciidoc
@@ -18,12 +18,12 @@ include::./overview.asciidoc[]
 
 include::./getting-started.asciidoc[]
 
-include::./how-filebeat-works.asciidoc[]
-
 include::./command-line.asciidoc[]
 
 include::../../libbeat/docs/shared-directory-layout.asciidoc[]
 
+include::./how-filebeat-works.asciidoc[]
+
 include::./configuring-howto.asciidoc[]
 
 include::./filebeat-filtering.asciidoc[]

diff --git a/filebeat/docs/migration.asciidoc b/filebeat/docs/migration.asciidoc
@@ -4,9 +4,7 @@
 [partintro]
 --
 Filebeat is based on the Logstash Forwarder source code and replaces Logstash Forwarder as the method
-to use for tailing log files and forwarding them to Logstash. This guide contains
-information to help you migrate from
-https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to Filebeat.
+to use for tailing log files and forwarding them to Logstash. 
 
 Filebeat introduces the following major changes:
 
@@ -17,8 +15,18 @@ Filebeat introduces the following major changes:
 * Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference].
 * The https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] is required.
 
+The following topics describe how to migrate from
+https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to Filebeat:
+
+* <<migration-input-plugin>>
+* <<migration-registry-file>>
+* <<migration-configuration>>
+* <<migration-changed-cli>>
+* <<migration-changed-fields>>
+* <<migration-other-changes>>
 --
 
+[[migration-input-plugin]]
 == Migrating to the Beats Input Plugin for Logstash
 
 Filebeat requires the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash].
@@ -45,6 +53,7 @@ What's required?
 version 2.2.8 or later.
 * SSL must be explicitly enabled in the Beats input plugin (`ssl => true`) because SSL is on by default with Logstash Forwarder. The SSL/TLS configs should be the same for both the Logstash Forwarder and Filebeat instances.
 
+[[migration-registry-file]]
 ==  Updating the Registry File
 
 The registry file stores the state and location information that Filebeat uses to track
@@ -68,6 +77,7 @@ Filebeat configuration file to true.
 Using this approach allows you to keep the old Logstash Forwarder running and then
 slowly migrate over to Filebeat.
 
+[[migration-configuration]]
 == Migrating Your Configuration
 
 Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration
@@ -284,6 +294,7 @@ output.elasticsearch:
   hosts: ["http://localhost:5043"]
 -------------------------------------------------------------------------------------
 
+[[migration-changed-cli]]
 == Changes to Command Line Options
 
 Most command line options available in Logstash Forwarder have been removed and
@@ -342,7 +353,7 @@ The `config_dir` option specifies the path to the directory that contains additi
 
 |===
 
-
+[[migration-changed-fields]]
 == Changes to the Output Fields
 
 In the default configuration, Filebeat structures its output documents a little
@@ -403,6 +414,7 @@ The `line` field was removed. This field wasn't correct after restarts, and
 making it correct would have resulted in a performance penalty. We recommend
 using the `offset` field instead.
 
+[[migration-other-changes]]
 == Other Changes
 
 The following list of implementation changes should not affect your experience migrating

diff --git a/filebeat/docs/troubleshooting.asciidoc b/filebeat/docs/troubleshooting.asciidoc
@@ -4,59 +4,19 @@
 [partintro]
 --
 If you have issues installing or running Filebeat, read the
-following tips.
+following tips:
+
+* <<getting-help>>
+* <<enable-filebeat-debugging>>
+* <<faq>>
 
 --
 
+[[getting-help]]
 == Getting Help
 
 include::../../libbeat/docs/getting-help.asciidoc[]
 
-
-== Reduce open file handlers
-
-Filebeat keeps the file handler open in case it reaches the end of a file to read new log lines in near real time. If filebeat is harvesting a large number of files, the number of open files can be become an issue. In most environments, the number of files which are actively updated is low. The configuration `close_inactive` should be set accordingly to close files which are not active any more.
-
-There are 4 more configuration options which can be used to close file handlers, but all of them should be used carefully as they can side affects. The options are:
-
-* close_renamed
-* close_removed
-* close_eof
-* close_timeout
-
-`close_renamed` and `close_removed` can be useful on Windows and issues related to file rotation, see <<windows-file-rotation>>. `close_eof` can be useful in environments with a large number of files with only very few entries. `close_timeout` in environments where it is more important to close file handlers then to send all log lines. More details can be found in config options, see <<configuration-filebeat-options>>.
-
-Before using any of these variables, make sure to study the documentation on each.
-
-
-[[reduce-registry-size]]
-== Reduce Registry File Size
-
-Filebeat keeps all states of the files and persists the states on disk in the `registry_file`. The states are used to continue file reading at a previous position in case filebeat is restarted. In case every day a large amount of new files is constantly produced, the registry file grows over time. To reduce the size of the registry file, there are two configuration variables: `clean_removed` and `close_inactive`.
-
-In case old files are not touched anymore and fall under `ignore_older`, it is recommended to use `clean_inactive`. If on the other size old files get removed from disk `clean_removed` can be used.
-
-[[inode-reuse-issue]]
-== Inode Reuse Issue
-
-Filebeat uses under linux inode and device to identify files. In case a file is removed from disk, the inode can again be assigned to a new file. In the case of file rotation where and old file is removed and a new one is directly created afterwards, it can happen that the new files has the exact same inode. In this case, Filebeat assumes that the new file is the same as the old and tries to continue reading at the old position which is not correct.
-
-By default states are never removed from the registry file. In case of inode reuse issue it is recommended to use the `clean_*` options, especially `clean_inactive`. In case your files get rotated every 24 hours and the rotated files rotated files are not updated anymore, `ignore_older` could be set to 48 hours and `clean_inactive` 72 hours.
-
-`clean_removed` can be used for files that are removed from disk. Be aware that `clean_removed` also applies if during one scan a file cannot be found anymore. In case the file shows up at a later stage again, it will be sent again from scratch.
-
-[[windows-file-rotation]]
-== Windows File Rotation
-
-Under Windows it can happen, that files cannot be renamed / removed as long as filebeat keeps the file handler open. This can lead to issues with the file rotating system. To reduce this issue, the options `close_removed` and `close_renamed` can be used together.
-
-It is important to understand, that these two options mean files are closed before the harvester finished reading the file. In case the file cannot be picked up again by the prospector and the harvester didn't finish reading the file, the missing lines will never be sent to elasticsearch.
-
-
-
-
-
-
 [[enable-filebeat-debugging]]
 == Debugging
 

@@ -4,7 +4,7 @@
 Beats config files are based on http://www.yaml.org[YAML], a file format that is
 easier to read and write than other common data formats like XML or JSON.
 
-In beats all YAML files start with an dictionary, an unordered collection of
+In beats all YAML files start with a dictionary, an unordered collection of
 name/value pairs. In addition to dictionaries, YAML also supports lists, numbers,
 strings, and many other data types. All members of the same list or dictionary must
 have the same indentation level.
@@ -52,6 +52,14 @@ person: \{name: "John Doe", age: 34, country: "Canada"}
 colors: ["Red", "Green", "Blue"]
 ------------------------------------------------------------------------------
 
+The following topics provide more detail to help you understand and work with config files in YAML:
+
+* <<config-file-format-namespacing>>
+* <<config-file-format-type>>
+* <<config-file-format-env-vars>>
+* <<config-gile-format-refs>>
+* <<config-file-format-cli>>
+* <<config-file-format-tips>>
 
 [[config-file-format-namespacing]]
 === Namespacing
@@ -154,7 +162,7 @@ style supports no escaping (use `''` to quote a single quote). Only printable
 characters can be used when using this form.
 
 Unquoted style requires no quotes, but does not support any escaping plus care
-needs to be taken to not use any symbol that has a speacial meaning in YAML.
+needs to be taken to not use any symbol that has a special meaning in YAML.
 
 Note: Single-quoted style is recommended when defining regular expressions,
 event format strings, windows file paths, or non-alphabetical symbolic characters.
@@ -313,7 +321,7 @@ when using plain references.
 === Command line arguments
 
 Config files to load are set using the `-c` flag on command line. If no flag is
-given, a beat and OS pecific default file path will be assumed.
+given, a beat and OS-specific default file path will be assumed.
 
 You can specify multiple configuration files by repeating the `-c` flag. You can
 use this, for example, for setting defaults in a base configuration file, and

@@ -13,7 +13,7 @@ Use the following naming conventions for field names:
 - Group related fields into subdocuments by using dot (.) notation. Groups typically have common prefixes. For example, if you have fields called `CPULoad` and `CPUSystem` in a service, you would convert
 them into `cpu.load` and `cpu.system` in the event. 
 - Avoid repeating the namespace in field names. If a word or abbreviation appears in the namespace, it's not needed in the field name. For example, instead of `cpu.cpu_load`, use `cpu.load`.
-- Use <<abbreviations,standard abbreviations>> and avoid using abbreviations that aren't commonly known.
+- Use <<abbreviations,standardised names>> and avoid using abbreviations that aren't commonly known.
 - Organise the documents from general to specific to allow for namespacing. The type, such as `.pct`, should always be last. For example, `system.core.user.pct`.
 - If two fields are the same, but with different units, remove the less granular one. For example, include `timeout.sec`, but don't include `timeout.min`. If a less granular value is required, you can calculate it later.
 - If a field name matches the namespace used for nested fields, add `.value` to the field name. For example, instead of:

@@ -7,10 +7,10 @@ the Elastic stack for Beats.
 
 A regular _Beats setup_ consists of:
 
- * One or more Beats. You install the Beats on your servers to capture operational data.
  * Elasticsearch for storage and indexing. See <<elasticsearch-installation>>.
  * Logstash (optional) for inserting data into Elasticsearch. See <<logstash-installation,Installing Logstash>>.
  * Kibana for the UI. See <<kibana-installation>>.
+ * One or more Beats. You install the Beats on your servers to capture operational data. See <<installing-beats>>.
  * Kibana dashboards for visualizing the data. See <<load-kibana-dashboards>>.
 
 See the https://www.elastic.co/support/matrix[Elastic Support Matrix] for information 

@@ -15,6 +15,18 @@ an example Beat called Countbeat. The Beat generator creates all the files requi
 for a working Beat. To create your own Beat, you modify the generated
 files and implement the custom logic needed to collect the data you want to ship.
 
+The following topics describe how to build a new Beat:
+
+* <<newbeat-getting-ready>>
+* <<newbeat-overview>>
+* <<newbeat-generate>>
+* <<setting-up-beat>>
+* <<compiling-and-running>>
+* <<beater-interface>>
+* <<ls-beat>>
+* <<newbeat-sharing>>
+* <<event-conventions>>
+
 [[newbeat-getting-ready]]
 === Getting Ready
 
@@ -42,6 +54,7 @@ When you run the command, all source files are downloaded to the
 `$GOPATH/src/github.com/elastic/beats` path. You can ignore the "no buildable Go source files" message because
 you will build the source later.
 
+[[newbeat-overview]]
 === Overview
 
 At the high level, a simple Beat has two main components:
@@ -99,6 +112,7 @@ The following example shows an event object in Lsbeat:
 
 Now that you have the big picture, let's dig into the code.
 
+[[newbeat-generate]]
 === Generating Your Beat
 
 To generate your own Beat, you use the Beat generator available in the beats repo on GitHub. If you haven't
@@ -479,7 +493,7 @@ Lsbeat periodically ships them to Elasticsearch for storage.
 To help you learn how to build a Beat, we've created this http://elastic.co//blog/build-your-own-beat[blog post]
 that describes how to build Lsbeat from scratch. You can refer to the Lsbeat implementation as a working example.
 
-
+[[newbeat-sharing]]
 === Sharing Your Beat with the Community
 
 When you're done with your new Beat, how about letting everyone know? Open

@@ -18,6 +18,14 @@ recommend that you use the virtual environment under
 https://github.com/elastic/beats/tree/master/testing/environments[beats/testing/environments] to import, create, and
 export the Kibana dashboards.
 
+The following topics provide more detail about importing and working with Beats dashboards:
+
+* <<import-dashboards>>
+* <<build-dashboards>>
+* <<generate-index-pattern>>
+* <<export-dashboards>>
+* <<archive-dashboards>>
+
 [[import-dashboards]]
 === Importing Existing Beat Dashboards