Backport: Cherrypicks #4973 and #5068 into 6.0 (#5103)

* [Docs] Document how to use modules.d directory (#4973)

* [Docs] Document how to use modules.d directory

* Add changes from review

* Add upgrading guide docs (#5068)

* Add upgrading guide docs

Part of / related to #4737.

filebeat/docs/configuring-howto.asciidoc

@@ -19,6 +19,7 @@ _Beats Platform Reference_ for more about the structure of the config file.
 
 The following topics describe how to configure Filebeat:
 
+* <<configuration-filebeat-modules>>
 * <<configuration-filebeat-options>>
 * <<multiline-examples>>
 * <<configuration-general-options>>
@@ -39,6 +40,8 @@ The following topics describe how to configure Filebeat:
 
 --
 
+include::./filebeat-modules-options.asciidoc[]
+
 include::./filebeat-options.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]

filebeat/docs/filebeat-modules-options.asciidoc

@@ -1,28 +1,105 @@
 [[configuration-filebeat-modules]]
 == Specify which modules to run
 
-//REVIEWERS: I created this topic because I think it might be confusing to have no mention of this modules config section in the docs. However, I think that adding the info might also be confusing becasue users might try to configure othe things (like prospectors). Note that this topic uses the old command syntax, but would be updated to reflect the new.
+NOTE: Using Filebeat modules is optional. You may decide to
+<<configuration-filebeat-options,set up prospectors manually>> if you are using
+a log file type that isn't supported, or you want to use a different setup.
 
 Filebeat <<filebeat-modules,modules>> provide a quick way for you to get started
 processing common log formats. They contain default configurations,
-Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help
-you implement and deploy a log monitoring solution.
+Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help you
+implement and deploy a log monitoring solution.
 
-Using modules is optional. You may decide to configure Filebeat manaully if
-you are using a log file type that isn't supported or you want to use a
-different setup.
+Filebeat provides a few different ways to enable modules. You can:
 
-To enable specific modules, you can add entries to the `filebeat.modules` list
-in the +{beatname_lc}.yml+ config file. Each entry in the list begins with a dash
-(-) and is followed by settings for that module.
+* <<enable-modules-d-configs>>
+* <<enable-modules-cli>>
+* <<enable-modules-config-file>>
 
-Filebeat also provides command-line options for enabling and disabling modules.
-See <<filebeat-modules-overview>> for more about running modules.
+When you enable modules, you can also
+<<specify-variable-settings,specify variable settings>> to change the default
+behavior of the modules, and you can specify
+<<advanced-settings,advanced settings>> to override prospector settings.
 
-The following example shows a configuration that runs the `nginx`,`mysql`, and
+Before running Filebeat with modules enabled, make sure you also set up the
+environment to use Kibana dashboards. See <<filebeat-modules-quickstart>> for
+more information.
+
+[float]
+[[enable-modules-d-configs]]
+=== Enable module configs in the `modules.d` directory
+
+The `modules.d` directory contains default configurations for all the modules
+available in Filebeat. You can enable or disable specific module configurations
+under `modules.d` by running the
+<<modules-command,`modules enable` or `modules disable`>> commands.
+
+For example, to enable the `apache2` and `mysql` configs in the `modules.d`
+directory, you use:
+
+[source,shell]
+----
+./filebeat modules enable apache2 mysql
+----
+
+Then when you run Filebeat, it loads the corresponding module configurations
+specified in the `modules.d` directory (for example, `modules.d/apache2.yml` and
+`modules.d/mysql.yml`).
+
+To see a list of enabled and disabled modules, run:
+
+[source,shell]
+----
+./filebeat modules list
+----
+
+The default module configurations assume that the logs you’re harvesting are
+in the location expected for your OS and that the behavior of the module is
+appropriate for your environment. To change the default configurations, you need
+to specify variable settings. See <<specify-variable-settings>>.
+
+[float]
+[[enable-modules-cli]]
+=== Enable modules when you run Filebeat
+
+To enable specific <<filebeat-modules,modules>> when you run Filebeat at the
+command line, you can use the `--modules` flag. This approach works well when
+you're getting started and want to specify different modules and settings each
+time you run Filebeat. Any modules specified at the command line will be loaded
+along with any modules that are enabled in the configuration file or `modules.d`
+directory. If there's a conflict, the configuration specified at the command
+line is used.
+
+The following example shows how to enable and run the `nginx`,`mysql`, and
 `system` modules.
 
-//QUESTION: Are we doing something like the modules.d layout for Filebeat?
+[source,shell]
+----
+./filebeat -e --modules nginx,mysql,system
+----
+
+The default module configurations assume that the logs you’re harvesting are
+in the location expected for your OS and that the behavior of the module is
+appropriate for your environment. To change the default configurations, you need
+to specify variable settings. See <<specify-variable-settings>>.
+
+[float]
+[[enable-modules-config-file]]
+=== Enable module configs in the +{beatname_lc}.yml+ file
+
+Enabling <<filebeat-modules,modules>> directly in the config file is a practical
+approach if you have upgraded from a previous version of {beatname_uc} and don't
+want to move your module configs to the `modules.d` directory. You can continue
+to configure modules in the +{beatname_lc}.yml+ file, but you won't be able to
+use the `modules` command to enable and disable configurations because the
+command requires the `modules.d` layout.
+
+To enable specific modules in the +{beatname_lc}.yml+ config file, you can add
+entries to the +{beatname_lc}.modules+ list. Each entry in the list begins with
+a dash (-) and is followed by settings for that module.
+
+The following example shows a configuration that runs the `nginx`,`mysql`, and
+`system` modules.
 
 [source,yaml]
 ----
@@ -32,34 +109,93 @@ filebeat.modules:
 - module: system
 ----
 
-To enable this same configuration from the command line, you use:
+The default module configurations assume that the logs you’re harvesting are
+in the location expected for your OS and that the behavior of the module is
+appropriate for your environment. To change the default configurations, you need
+to specify variable settings. See <<specify-variable-settings>>.
 
-[source,shell]
-----
-./filebeat -e -modules=nginx,mysql,system
-----
+[[specify-variable-settings]]
+=== Specify variable settings
 
-The configuration in the example assumes that the logs you’re harvesting are
-in the location expected for your OS and that the default behavior of Filebeat
-is appropriate for your environment. Each module provides variables that you
-can set to fine tune the behavior of Filebeat, including the location
-where it looks for log files. For example, the following configuration sets
-the path for `nginx` log files:
+Each module and fileset has variables that you can set to change the default
+behavior of the module, including the paths where the module looks for log
+files. For example, the `var.paths` setting in the following example sets the
+path for `nginx` access log files:
 
 [source,yaml]
 ----
-filebeat.modules:
 - module: nginx
   access:
     var.paths: ["/var/log/nginx/access.log*"]
 ----
 
-To set the same configuration from the command line, you use:
+To set the path for Nginx access log files at the command line, you use
+the `-M` flag. For example:
 
 [source,shell]
 ----
-./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
+./filebeat -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
 ----
 
-See the <<filebeat-modules,documentation for the modules>> you are using for
-more information.
+When you set variables at the command line, the variable name needs to include
+the module and fileset name. You can specify multiple overrides. Each override
+must start with `-M`.
+
+Here you see how to use the `-M` flag along with the `--modules` flag. This
+example shows how to set the paths to the access and error logs:
+
+[source,shell]
+----
+./filebeat --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" -M "nginx.error.var.paths=[/var/log/nginx/error.log*]"
+----
+
+For information about specific variables that you can set for each fileset,
+see the <<filebeat-modules,documentation for the modules>>.
+
+[[advanced-settings]]
+=== Advanced settings
+
+Behind the scenes, each module starts a Filebeat prospector. Advanced users
+can add or override any prospector settings. For example, you can set
+<<close-eof,close_eof>> to `true` in the module configuration:
+
+[source,yaml]
+----------------------------------------------------------------------
+- module: nginx
+  access:
+    prospector:
+      close_eof: true
+----------------------------------------------------------------------
+
+Or at the command line like this:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "nginx.access.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+
+Here you see how to use the `-M` flag along with the `--modules` flag:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat --modules nginx -M "nginx.access.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+
+You can use wildcards to change variables or settings for multiple
+modules/filesets at once. For example, the following command enables
+`close_eof` for all the filesets in the `nginx` module:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "nginx.*.prospector.close_eof=true"
+----------------------------------------------------------------------
+
+The following command enables `close_eof` for all prospectors created by any of
+the modules:
+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -M "*.*.prospector.close_eof=true"
+----------------------------------------------------------------------

filebeat/docs/filebeat-options.asciidoc

@@ -1,6 +1,12 @@
 [[configuration-filebeat-options]]
 == Set up prospectors
 
+TIP: <<filebeat-modules-overview,Filebeat modules>> provide the fastest getting
+started experience for common log formats. See <<filebeat-modules-quickstart>>
+to learn how to get started with modules. Also see
+<<configuration-filebeat-modules>> for information about enabling and
+configuring modules.
+
 Filebeat uses prospectors to locate and process files. To configure Filebeat,
 you specify a list of prospectors in the `filebeat.prospectors` section of the
 +{beatname_lc}.yml+ config file.

filebeat/docs/getting-started.asciidoc

@@ -264,7 +264,7 @@ sudo chown root filebeat.yml <1>
 sudo ./filebeat -e -c filebeat.yml -d "publish"
 ----------------------------------------------------------------------
 <1> You'll be running Filebeat as root, so you need to change ownership
-of the configuration file, or run Filebeat with `-strict.perms=false`
+of the configuration file, or run Filebeat with `--strict.perms=false`
 specified. See
 {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
 in the _Beats Platform Reference_.

filebeat/docs/modules-getting-started.asciidoc

@@ -40,6 +40,9 @@ sudo bin/elasticsearch-plugin install ingest-user-agent
 ----------------------------------------------------------------------
 +
 You need to restart Elasticsearch after running these commands.
++
+If you are using an https://cloud.elastic.co/[Elastic Cloud] instance, you can
+enable the two plugins from the configuration page.
 
 * Verify that Elasticsearch and Kibana are running and that Elasticsearch is
 ready to receive data from Filebeat.
@@ -58,17 +61,17 @@ the sample dashboards for visualizing the data in Kibana. For example:
 ./filebeat setup -e
 ----------------------------------------------------------------------
 +
-The value that you pass with the `-modules` flag is a comma-separated list of
+The value that you pass with the `--modules` flag is a comma-separated list of
 modules that you want to set up. The `-e` flag is optional and sends output
 to standard error instead of syslog.
 
-. Start Filebeat and use the `-modules` flag to specify the list of modules
+. Start Filebeat and use the `--modules` flag to specify the list of modules
 you want to run. The following example starts Filebeat with the `system` module
 enabled (it's assumed that you've already loaded the sample dashboards):
 +
 [source,shell]
 ----------------------------------------------------------------------
-./filebeat -e -modules=system
+./filebeat -e --modules system
 ----------------------------------------------------------------------
 +
 This command takes care of configuring Filebeat and loading the ingest node
@@ -79,22 +82,30 @@ example:
 +
 [source,shell]
 ----------------------------------------------------------------------
-./filebeat -e -modules=system,nginx,mysql
+./filebeat -e --modules system,nginx,mysql
 ----------------------------------------------------------------------
-
++
+When you start Filebeat, you should see messages indicating that Filebeat
+has started harvesters for all enabled modules. For example:
++
+[source,shell]
+----------------------------------------------------------------------
+2017/08/16 23:39:15.414375 harvester.go:206: INFO Harvester started for file: /var/log/displaypolicyd.stdout.log
+----------------------------------------------------------------------
++
+If you don't see this message for each log file that needs to be read,
+see <<setting-variables>> to find out how to set the path the files.
 
 NOTE: Depending on how you've installed Filebeat, you might see errors
 related to file ownership or permissions when you try to run Filebeat modules.
 See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
 in the _Beats Platform Reference_ if you encounter errors related to file
 ownership or permissions.
 
-//include::system-module-note.asciidoc[]
-
-
-TIP: In a production environment, you'll probably want to use a configuration
-file, rather than command-line flags, to specify which modules to run. See the
-detailed documentation for more about configuring and running modules.
+This getting started guide uses the `--modules` flag to enable modules
+at the command line when you run Filebeat. In a production environment, you'll
+probably want to use the configs in the `modules.d` directory instead. See
+<<configuration-filebeat-modules>> for more information.
 
 [[setting-variables]]
 ==== Set the path variable
@@ -108,11 +119,11 @@ logs:
 
 [source,shell]
 ----
-./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
+./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
 ----
 
-See the <<modules-tutorial>> for more information about setting variables and
-advanced options.
+See <<configuration-filebeat-modules>> for more information about setting
+variables and advanced options.
 
 [[passing-credentials-modules]]
 ==== Pass credentials