From 5b06c2a1d3eed4ce6e16c71b10b2a4d96ecd103d Mon Sep 17 00:00:00 2001 From: DeDe Morton Date: Fri, 21 Apr 2017 00:56:10 -0700 Subject: [PATCH] Backport: Mulitple doc changes into 5.4 (#4073) * Add workaround for Heartbeat install issue * Clarify docs around setting the index and @metadata fields * Add step to change file ownership on mac * Update yaml tips * Clarify module quick start steps * Remove yellow box redirecting users to master for dev docs * Edit config reload topics * Edit config info about dashboard loading * Update curl examples to include content-type * Fix structure of module topics * add kafkabeat (#4017) * add kafkabeat kafkabeat read data from kafka * Update communitybeats.asciidoc --- filebeat/docs/getting-started.asciidoc | 152 +++--------------- filebeat/docs/index.asciidoc | 4 + .../docs/modules-getting-started.asciidoc | 119 ++++++++++++++ filebeat/docs/modules-overview.asciidoc | 32 ++-- filebeat/docs/modules/apache2.asciidoc | 1 + filebeat/docs/modules/auditd.asciidoc | 1 + filebeat/docs/modules/mysql.asciidoc | 3 + filebeat/docs/modules/nginx.asciidoc | 1 + filebeat/docs/modules/system.asciidoc | 1 + .../docs/reference/configuration.asciidoc | 1 + .../configuration/filebeat-options.asciidoc | 5 +- .../reload-configuration.asciidoc | 34 ++-- filebeat/docs/system-module-note.asciidoc | 19 +++ filebeat/module/mysql/_meta/docs.asciidoc | 2 + filebeat/scripts/docs_collector.py | 1 + heartbeat/docs/getting-started.asciidoc | 5 + heartbeat/docs/index.asciidoc | 2 + .../docs/reference/configuration.asciidoc | 1 + .../configuration/heartbeat-options.asciidoc | 2 +- libbeat/docs/communitybeats.asciidoc | 3 +- libbeat/docs/config-file-format.asciidoc | 43 +---- libbeat/docs/dashboardsconfig.asciidoc | 70 +++++--- libbeat/docs/generalconfig.asciidoc | 2 +- libbeat/docs/gettingstarted.asciidoc | 19 ++- libbeat/docs/index.asciidoc | 5 +- libbeat/docs/loggingconfig.asciidoc | 2 +- libbeat/docs/outputconfig.asciidoc | 87 ++++++---- libbeat/docs/repositories.asciidoc | 15 ++ libbeat/docs/shared-config-ingest.asciidoc | 2 +- libbeat/docs/shared-faq.asciidoc | 13 ++ libbeat/docs/shared-logstash-config.asciidoc | 7 +- libbeat/docs/shared-path-config.asciidoc | 2 +- libbeat/docs/shared-template-load.asciidoc | 6 +- libbeat/docs/yaml.asciidoc | 47 ++++-- metricbeat/docs/gettingstarted.asciidoc | 5 + metricbeat/docs/index.asciidoc | 2 + .../docs/reference/configuration.asciidoc | 1 + .../configuration/metricbeat-options.asciidoc | 2 +- .../reload-configuration.asciidoc | 35 ++-- packetbeat/docs/gettingstarted.asciidoc | 8 + packetbeat/docs/index.asciidoc | 3 +- .../docs/reference/configuration.asciidoc | 1 + .../configuration/packetbeat-options.asciidoc | 8 +- .../configuration/runconfig.asciidoc | 2 +- winlogbeat/docs/index.asciidoc | 2 + .../docs/reference/configuration.asciidoc | 1 + .../configuration/winlogbeat-options.asciidoc | 2 +- 47 files changed, 471 insertions(+), 310 deletions(-) create mode 100644 filebeat/docs/modules-getting-started.asciidoc create mode 100644 filebeat/docs/system-module-note.asciidoc diff --git a/filebeat/docs/getting-started.asciidoc b/filebeat/docs/getting-started.asciidoc index 3dc72a2efea7..2f0fc0ab5289 100644 --- a/filebeat/docs/getting-started.asciidoc +++ b/filebeat/docs/getting-started.asciidoc @@ -11,127 +11,16 @@ See {libbeat}/getting-started.html[Getting Started with Beats and the Elastic St After installing the Elastic Stack, read the following topics to learn how to install, configure, and run Filebeat: -* <> * <> * <> * <> * <> * <> * <> +* <> * <> * <> -[[filebeat-modules-quickstart]] -=== Quick Start for Common Log Formats - -beta[] - -Filebeat provides a set of pre-built modules that you can use to rapidly -implement and deploy a log monitoring solution, complete with sample dashboards -and data visualizations, in about 5 minutes. These modules support common log -formats, such as Nginx, Apache2, and MySQL, and can be run by issuing a simple -command. - -This topic shows you how to run the basic modules out of the box without extra -configuration. For detailed documentation and the full list of available -modules, see <>. - -Skip this topic and go to <> if you are using a log file -type that isn't supported by one of the available Filebeat modules. - -==== Prerequisites - -Before running Filebeat with modules enabled, you need to: - -* Install and configure the Elastic stack. See -{libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack]. - -* Complete the Filebeat installation instructions described in -<>. After installing Filebeat, return to this -quick start page. - -* Install the Ingest Node GeoIP and User Agent plugins, which you can do by -running the following commands in the Elasticsearch home path: -+ -[source,shell] ----------------------------------------------------------------------- -sudo bin/elasticsearch-plugin install ingest-geoip -sudo bin/elasticsearch-plugin install ingest-user-agent ----------------------------------------------------------------------- -+ -You need to restart Elasticsearch after running these commands. - -* Verify that Elasticsearch and Kibana are running and that Elasticsearch is -ready to receive data from Filebeat. - -//TODO: Follow up to find out whether ingest-geoip and ingest-user-agent will be bundled with ES. If so, remove the last prepreq. - -[[running-modules-quickstart]] -==== Running Filebeat with Modules Enabled - -To run one or more Filebeat modules, you issue the following command: - -[source,shell] ----------------------------------------------------------------------- -filebeat -e -modules=MODULES -setup ----------------------------------------------------------------------- - -Where `MODULES` is the name of the module (or a comma-separated list of -modules) that you want to enable. The `-e` flag is optional and sends output -to standard error instead of syslog. The `-setup` flag is a one-time setup step. -For subsequent runs of Filebeat, do not specify this flag. - -For example, to start Filebeat with the `system` module enabled and load the -sample Kibana dashboards, run: - -[source,shell] ----------------------------------------------------------------------- -filebeat -e -modules=system -setup ----------------------------------------------------------------------- - -This command takes care of configuring Filebeat, loading the recommended index -template for writing to Elasticsearch, and deploying the sample dashboards -for visualizing the data in Kibana. - -To start Filebeat with the `system`, `nginx`, and `mysql` modules enabled -and load the sample dashboards, run: - -[source,shell] ----------------------------------------------------------------------- -filebeat -e -modules=system,nginx,mysql -setup ----------------------------------------------------------------------- - -To start Filebeat with the `system` module enabled (it's assumed that -you've already loaded the sample dashboards), run: - -[source,shell] ----------------------------------------------------------------------- -filebeat -e -modules=system ----------------------------------------------------------------------- - -TIP: In a production environment, you'll probably want to use a configuration -file, rather than command-line flags, to specify which modules to run. See the -detailed documentation for more about configuring and running modules. - -These examples assume that the logs you're harvesting are in the location -expected for your OS and that the default behavior of Filebeat is appropriate -for your environment. Each module provides a set of variables that you can set -to fine tune the behavior of Filebeat, including the location where it looks -for log files. See <> for more info. - -[[visualizing-data]] -==== Visualizing the Data in Kibana - -After you've confirmed that Filebeat is sending events to Elasticsearch, launch -the Kibana web interface by pointing your browser to port 5601. For example, -http://127.0.0.1:5601[http://127.0.0.1:5601]. - -Open the dashboard and explore the visualizations for your parsed logs. - -Here's an example of the syslog dashboard: - -image:./images/kibana-system.png[Sylog dashboard] - [[filebeat-installation]] === Step 1: Installing Filebeat @@ -239,26 +128,21 @@ NOTE: If script execution is disabled on your system, you need to set the execut endif::[] -If you're using modules to get started with Filebeat, go back to the -<> page. - -Otherwise, continue on to <>. - -Before starting Filebeat, you should look at the configuration options in the configuration -file, for example `C:\Program Files\Filebeat\filebeat.yml` or `/etc/filebeat/filebeat.yml`. For more information about these options, -see <>. - [[filebeat-configuration]] === Step 2: Configuring Filebeat TIP: <> provide the fastest getting -started experience for common log formats. See <> to -learn how to get started with modules. - -To configure Filebeat, you edit the configuration file. For rpm and deb, you'll -find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and win, look in -the archive that you just extracted. There’s also a full example configuration file -called `filebeat.full.yml` that shows all non-deprecated options. +started experience for common log formats. See <> +to learn how to get started with modules. If you use Filebeat modules to get +started, you can skip the content in this section, including the remaining +getting started steps, and go directly to the <> +page. + +To configure Filebeat manually, you edit the configuration file. For rpm and deb, +you'll find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and +win, look in the archive that you just extracted. There’s also a full example +configuration file called `filebeat.full.yml` that shows all non-deprecated +options. See the {libbeat}/config-file-format.html[Config File Format] section of the @@ -315,7 +199,10 @@ options specified: +./filebeat -configtest -e+. Make sure your config files are in the path expected by Filebeat (see <>). If you installed from DEB or RPM packages, run +./filebeat.sh -configtest -e+. -See <> for more details about each configuration option. +Before starting Filebeat, you should look at the configuration options in the +configuration file, for example `C:\Program Files\Filebeat\filebeat.yml` or +`/etc/filebeat/filebeat.yml`. For more information about these options, +see <>. [[config-filebeat-logstash]] === Step 3: Configuring Filebeat to Use Logstash @@ -332,7 +219,7 @@ include::../../libbeat/docs/shared-template-load.asciidoc[] [[filebeat-starting]] === Step 5: Starting Filebeat -Start Filebeat by issuing the appropriate command for your platform. +Start Filebeat by issuing the appropriate command for your platform. NOTE: If you use an init.d script to start Filebeat on deb or rpm, you can't specify command line flags (see <>). To specify flags, @@ -356,8 +243,13 @@ sudo /etc/init.d/filebeat start [source,shell] ---------------------------------------------------------------------- +sudo chown root filebeat.yml <1> sudo ./filebeat -e -c filebeat.yml -d "publish" ---------------------------------------------------------------------- +<1> You'll be running Filebeat as root, so you need to change ownership +of the configuration file (see +{libbeat}config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_). *win:* diff --git a/filebeat/docs/index.asciidoc b/filebeat/docs/index.asciidoc index ab3c69b755e4..f33c2b3059ce 100644 --- a/filebeat/docs/index.asciidoc +++ b/filebeat/docs/index.asciidoc @@ -7,6 +7,7 @@ include::../../libbeat/docs/version.asciidoc[] :metricbeat: http://www.elastic.co/guide/en/beats/metricbeat/{doc-branch} :filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch} :winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch} +:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} :elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :elasticsearch-plugins: https://www.elastic.co/guide/en/elasticsearch/plugins/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 @@ -19,6 +20,8 @@ include::./overview.asciidoc[] include::./getting-started.asciidoc[] +include::./modules-getting-started.asciidoc[] + include::./command-line.asciidoc[] include::../../libbeat/docs/shared-directory-layout.asciidoc[] @@ -43,6 +46,7 @@ include::./multiple-prospectors.asciidoc[] include::./load-balancing.asciidoc[] +:standalone: :allplatforms: include::../../libbeat/docs/yaml.asciidoc[] diff --git a/filebeat/docs/modules-getting-started.asciidoc b/filebeat/docs/modules-getting-started.asciidoc new file mode 100644 index 000000000000..1b3c11564aac --- /dev/null +++ b/filebeat/docs/modules-getting-started.asciidoc @@ -0,0 +1,119 @@ +[[filebeat-modules-quickstart]] +=== Quick Start for Common Log Formats + +beta[] + +Filebeat provides a set of pre-built modules that you can use to rapidly +implement and deploy a log monitoring solution, complete with sample dashboards +and data visualizations, in about 5 minutes. These modules support common log +formats, such as Nginx, Apache2, and MySQL, and can be run by issuing a simple +command. + +This topic shows you how to run the basic modules out of the box without extra +configuration. For detailed documentation and the full list of available +modules, see <>. + +If you are using a log file type that isn't supported by one of the available +Filebeat modules, you'll need to set up and configure Filebeat manually by +following the numbered steps under <>. + +==== Prerequisites + +Before running Filebeat with modules enabled, you need to: + +* Install and configure the Elastic stack. See +{libbeat}/getting-started.html[Getting Started with Beats and the Elastic Stack]. + +* Complete the Filebeat installation instructions described in +<>. After installing Filebeat, return to this +quick start page. + +* Install the Ingest Node GeoIP and User Agent plugins. These plugins are +required to capture the geographical location and browser information used by +some of the visualizations available in the sample dashboards. You can install +these plugins by running the following commands in the Elasticsearch home path: ++ +[source,shell] +---------------------------------------------------------------------- +sudo bin/elasticsearch-plugin install ingest-geoip +sudo bin/elasticsearch-plugin install ingest-user-agent +---------------------------------------------------------------------- ++ +You need to restart Elasticsearch after running these commands. + +* Verify that Elasticsearch and Kibana are running and that Elasticsearch is +ready to receive data from Filebeat. + +[[running-modules-quickstart]] +==== Running Filebeat with Modules Enabled + +To run one or more Filebeat modules, you issue the following command: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -e -modules=MODULES -setup +---------------------------------------------------------------------- + +Where `MODULES` is the name of the module (or a comma-separated list of +modules) that you want to enable. The `-e` flag is optional and sends output +to standard error instead of syslog. The `-setup` flag is a one-time setup step. +For subsequent runs of Filebeat, do not specify this flag. + +The following example starts Filebeat with the `system` module enabled and +loads the sample Kibana dashboards: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -e -modules=system -setup +---------------------------------------------------------------------- + +This command takes care of configuring Filebeat, loading the recommended index +template for writing to Elasticsearch, and deploying the sample dashboards +for visualizing the data in Kibana. + +NOTE: Depending on how you've installed Filebeat, you might see errors +related to file ownership or permissions when you try to run Filebeat modules. +See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_ if you encounter errors related to file +ownership or permissions. + +include::system-module-note.asciidoc[] + +To start Filebeat with the `system`, `nginx`, and `mysql` modules enabled +and load the sample dashboards, run: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -e -modules=system,nginx,mysql -setup +---------------------------------------------------------------------- + +To start Filebeat with the `system` module enabled (it's assumed that +you've already loaded the sample dashboards), run: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -e -modules=system +---------------------------------------------------------------------- + +TIP: In a production environment, you'll probably want to use a configuration +file, rather than command-line flags, to specify which modules to run. See the +detailed documentation for more about configuring and running modules. + +These examples assume that the logs you're harvesting are in the location +expected for your OS and that the default behavior of Filebeat is appropriate +for your environment. Each module provides a set of variables that you can set +to fine tune the behavior of Filebeat, including the location where it looks +for log files. See <> for more info. + +[[visualizing-data]] +==== Visualizing the Data in Kibana + +After you've confirmed that Filebeat is sending events to Elasticsearch, launch +the Kibana web interface by pointing your browser to port 5601. For example, +http://127.0.0.1:5601[http://127.0.0.1:5601]. + +Open the dashboard and explore the visualizations for your parsed logs. + +Here's an example of the syslog dashboard: + +image:./images/kibana-system.png[Sylog dashboard] \ No newline at end of file diff --git a/filebeat/docs/modules-overview.asciidoc b/filebeat/docs/modules-overview.asciidoc index 1d5f600943b5..05e66016934d 100644 --- a/filebeat/docs/modules-overview.asciidoc +++ b/filebeat/docs/modules-overview.asciidoc @@ -6,7 +6,7 @@ beta[] Filebeat modules simplify the collection, parsing, and visualization of common log formats. -A typical module (say, for the Nginx logs) is composed of one ore +A typical module (say, for the Nginx logs) is composed of one or more filesets (in the case of Nginx, `access` and `error`). A fileset contains the following: @@ -37,13 +37,14 @@ Node. This tutorial assumes you have Elasticsearch and Kibana installed and accessible from Filebeat (see the <> section). It also assumes that the Ingest Node GeoIP and User Agent plugins are -installed, which you can do with the following two commands executed in the -Elasticsearch home path: +installed. These plugins are required to capture the geographical location and +browser information used by some of the visualizations available in the sample +dashboards. You can install these plugins by running the following commands in the Elasticsearch home path: [source,shell] ---------------------------------------------------------------------- -$ sudo bin/elasticsearch-plugin install ingest-geoip -$ sudo bin/elasticsearch-plugin install ingest-user-agent +sudo bin/elasticsearch-plugin install ingest-geoip +sudo bin/elasticsearch-plugin install ingest-user-agent ---------------------------------------------------------------------- You need to restart Elasticsearch after running these commands. @@ -59,7 +60,7 @@ You can start Filebeat with the following command: [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx -setup +./filebeat -e -modules=nginx -setup ---------------------------------------------------------------------- The `-e` flag tells Filebeat to output its logs to standard error, instead of @@ -82,9 +83,11 @@ You can also start multiple modules at once: [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx,mysql,system +./filebeat -e -modules=nginx,mysql,system ---------------------------------------------------------------------- +include::system-module-note.asciidoc[] + While enabling the modules from the CLI file is handy for getting started and for testing, you will probably want to use the configuration file for the production setup. The equivalent of the above in the configuration file is: @@ -95,11 +98,12 @@ production setup. The equivalent of the above in the configuration file is: filebeat.modules: - module: nginx - module: mysql -- module: syslog +- module: system ---------------------------------------------------------------------- Then you can start Filebeat simply with: `./filebeat -e`. +[[module-varialbe-overrides]] ==== Variable overrides Each module and fileset has a set of "variables" which allow adjusting their @@ -116,7 +120,7 @@ files are in a custom location: [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---------------------------------------------------------------------- Or via the configuration file: @@ -129,7 +133,7 @@ filebeat.modules: var.paths = ["/var/log/nginx/access.log*"] ---------------------------------------------------------------------- -The Nginx `access` fileset also has a `pipeline` variables which allows +The Nginx `access` fileset also has a `pipeline` variable which allows selecting which of the available Ingest Node pipelines is used for parsing. At the moment, two such pipelines are available, one that requires the two ingest plugins (`ingest-geoip` and `ingest-user-agent`) and one that doesn't. If you @@ -138,7 +142,7 @@ cannot install the plugins, you can use the following: [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx -M "nginx.access.var.pipeline=no_plugins" +./filebeat -e -modules=nginx -M "nginx.access.var.pipeline=no_plugins" ---------------------------------------------------------------------- ==== Advanced settings @@ -162,7 +166,7 @@ Or like this: [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true" +./filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true" ---------------------------------------------------------------------- From the CLI, it's possible to change variables or settings for multiple @@ -171,7 +175,7 @@ modules/fileset at once. For example, the following works and will enable [source,shell] ---------------------------------------------------------------------- -$ filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true" +./filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true" ---------------------------------------------------------------------- The following also works and will enable `close_eof` for all prospectors @@ -179,5 +183,5 @@ created by any of the modules: [source,shell] ---------------------------------------------------------------------- -filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true" +./filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true" ---------------------------------------------------------------------- diff --git a/filebeat/docs/modules/apache2.asciidoc b/filebeat/docs/modules/apache2.asciidoc index b031627b2e72..3255e51e2f5a 100644 --- a/filebeat/docs/modules/apache2.asciidoc +++ b/filebeat/docs/modules/apache2.asciidoc @@ -47,6 +47,7 @@ An array of paths where to look for the log files. If left empty, Filebeat will choose the paths depending on your operating systems. +[float] === Fields For a description of each field in the metricset, see the diff --git a/filebeat/docs/modules/auditd.asciidoc b/filebeat/docs/modules/auditd.asciidoc index beeca663729b..711f9385e8d6 100644 --- a/filebeat/docs/modules/auditd.asciidoc +++ b/filebeat/docs/modules/auditd.asciidoc @@ -34,6 +34,7 @@ An array of paths where to look for the log files. If left empty, Filebeat will choose the paths depending on your operating systems. +[float] === Fields For a description of each field in the metricset, see the diff --git a/filebeat/docs/modules/mysql.asciidoc b/filebeat/docs/modules/mysql.asciidoc index 71491fcebb76..3b20dc908941 100644 --- a/filebeat/docs/modules/mysql.asciidoc +++ b/filebeat/docs/modules/mysql.asciidoc @@ -7,12 +7,14 @@ This file is generated! See scripts/docs_collector.py This module collects and parses the slow logs and error logs created by https://www.mysql.com/[MySQL]. +[float] === Compatibility The MySQL module was tested with logs from versions 5.5 and 5.7. On Windows, the module was tested with MySQL installed from the Chocolatey repository. +[float] === Dashboard This module comes with a sample dashboard. @@ -38,6 +40,7 @@ An array of paths where to look for the log files. If left empty, Filebeat will choose the paths depending on your operating systems. +[float] === Fields For a description of each field in the metricset, see the diff --git a/filebeat/docs/modules/nginx.asciidoc b/filebeat/docs/modules/nginx.asciidoc index 710fcbe85b25..3a764f34ebe8 100644 --- a/filebeat/docs/modules/nginx.asciidoc +++ b/filebeat/docs/modules/nginx.asciidoc @@ -47,6 +47,7 @@ will choose the paths depending on your operating systems. +[float] === Fields For a description of each field in the metricset, see the diff --git a/filebeat/docs/modules/system.asciidoc b/filebeat/docs/modules/system.asciidoc index 944bca571de7..4e93d97e07f3 100644 --- a/filebeat/docs/modules/system.asciidoc +++ b/filebeat/docs/modules/system.asciidoc @@ -33,6 +33,7 @@ An array of paths where to look for the log files. If left empty, Filebeat will choose the paths depending on your operating systems. +[float] === Fields For a description of each field in the metricset, see the diff --git a/filebeat/docs/reference/configuration.asciidoc b/filebeat/docs/reference/configuration.asciidoc index b33da1810124..30fd9ee202f0 100644 --- a/filebeat/docs/reference/configuration.asciidoc +++ b/filebeat/docs/reference/configuration.asciidoc @@ -23,6 +23,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> * <> * <> * <> diff --git a/filebeat/docs/reference/configuration/filebeat-options.asciidoc b/filebeat/docs/reference/configuration/filebeat-options.asciidoc index 9e26fb8b0838..3f05ba61df22 100644 --- a/filebeat/docs/reference/configuration/filebeat-options.asciidoc +++ b/filebeat/docs/reference/configuration/filebeat-options.asciidoc @@ -1,5 +1,5 @@ [[configuration-filebeat-options]] -=== Filebeat Prospectors Configuration +=== Filebeat Prospectors The `filebeat` section of the +{beatname_lc}.yml+ config file specifies a list of `prospectors` that Filebeat uses to locate and process log files. Each prospector item begins with a dash (-) @@ -294,6 +294,7 @@ If you require log lines to be sent in near real time do not use a very low `sca The default setting is 10s. +[[filebeat-document-type]] ===== document_type The event type to use for published lines read by harvesters. For Elasticsearch @@ -474,7 +475,7 @@ by assigning a higher limit of harvesters. The `enabled` option can be used with each prospector to define if a prospector is enabled or not. By default, enabled is set to true. [[configuration-global-options]] -=== Filebeat Global Configuration +=== Filebeat Global You can specify configuration options in the +{beatname_lc}.yml+ config file to control Filebeat behavior at a global level. diff --git a/filebeat/docs/reference/configuration/reload-configuration.asciidoc b/filebeat/docs/reference/configuration/reload-configuration.asciidoc index 1f6842f5425a..aa6f7560f911 100644 --- a/filebeat/docs/reference/configuration/reload-configuration.asciidoc +++ b/filebeat/docs/reference/configuration/reload-configuration.asciidoc @@ -3,11 +3,17 @@ beta[] -Reload configuration allows to dynamically reload prospector configuration files. A glob can be defined which should be watched - for prospector configuration changes. New prospectors will be started / stopped accordingly. This is especially useful in - container environments where 1 container is used to tail logs from services in other containers on the same host. +You can configure Filebeat to dynamically reload prospector configuration files +when there are changes. To do this, you specify a path +(https://golang.org/pkg/path/filepath/#Glob[Glob]) to watch for prospector +configuration changes. When the files found by the Glob change, new prospectors +are started/stopped according to changes in the configuration files. -The configuration in the main filebeat.yml config file looks as following: +This feature is especially useful in container environments where one container +is used to tail logs for services running in other containers on the same host. + +To enable dynamic config reloading, you specify the `path` and `reload` options +in the main `filebeat.yml` config file. For example: [source,yaml] ------------------------------------------------------------------------------ @@ -17,11 +23,16 @@ filebeat.config.prospectors: reload.period: 10s ------------------------------------------------------------------------------ -A path with a glob must be defined on which files should be checked for changes. A period is set on how often -the files are checked for changes. Do not set period below 1s as the modification time of files is often stored in seconds. -Setting it below 1s will cause an unnecessary overhead. +`path`:: A Glob that defines the files to check for changes. +`reload.enabled`:: When set to `true`, enables dynamic config reload. +`reload.period`:: Specifies how often the files are checked for changes. Do not +set the `period` to less than 1s because the modification time of files is often +stored in seconds. Setting the `period` to less than 1s will result in +unnecessary overhead. + +Each file found by the Glob must contain a list of one or more prospector +definitions. For example: -The configuration inside the files which are found by the glob look as following: [source,yaml] ------------------------------------------------------------------------------ - input_type: log @@ -35,7 +46,6 @@ The configuration inside the files which are found by the glob look as following scan_frequency: 5s ------------------------------------------------------------------------------ -Each file directly contains a list of prospectors. Each file can contain one or multiple prospector definitions. - -WARNING: It is critical that two running prospectors DO NOT have overlapping file paths defined. If more then one prospector -harvests the same file at the same time, it can lead to unexpected behaviour. +WARNING: It is critical that two running prospectors DO NOT have overlapping +file paths defined. If more than one prospector harvests the same file at the +same time, it can lead to unexpected behaviour. diff --git a/filebeat/docs/system-module-note.asciidoc b/filebeat/docs/system-module-note.asciidoc new file mode 100644 index 000000000000..1774a041533a --- /dev/null +++ b/filebeat/docs/system-module-note.asciidoc @@ -0,0 +1,19 @@ +[NOTE] +=============================================================================== +Because Filebeat modules are currently in Beta, the default Filebeat +configuration may interfere with the Filebeat `system` module configuration. If +you plan to run the `system` module, edit the Filebeat configuration file, +`filebeat.yml`, and comment out the following lines: + +[source,yaml] +---------------------------------------------------------------------- +#- input_type: log + #paths: + #- /var/log/*.log +---------------------------------------------------------------------- + +For rpm and deb, you'll find the configuration file at +`/etc/filebeat/filebeat.yml`. For mac and win, look in the archive that you +extracted when you installed Filebeat. + +=============================================================================== \ No newline at end of file diff --git a/filebeat/module/mysql/_meta/docs.asciidoc b/filebeat/module/mysql/_meta/docs.asciidoc index cc55e179ac91..7cd68969ddde 100644 --- a/filebeat/module/mysql/_meta/docs.asciidoc +++ b/filebeat/module/mysql/_meta/docs.asciidoc @@ -2,12 +2,14 @@ This module collects and parses the slow logs and error logs created by https://www.mysql.com/[MySQL]. +[float] === Compatibility The MySQL module was tested with logs from versions 5.5 and 5.7. On Windows, the module was tested with MySQL installed from the Chocolatey repository. +[float] === Dashboard This module comes with a sample dashboard. diff --git a/filebeat/scripts/docs_collector.py b/filebeat/scripts/docs_collector.py index 50f7711d52ae..7acf154aac97 100644 --- a/filebeat/scripts/docs_collector.py +++ b/filebeat/scripts/docs_collector.py @@ -44,6 +44,7 @@ def collect(beat_name): module_file += """ +[float] === Fields For a description of each field in the metricset, see the diff --git a/heartbeat/docs/getting-started.asciidoc b/heartbeat/docs/getting-started.asciidoc index 3819317ab45b..05584a34b91d 100644 --- a/heartbeat/docs/getting-started.asciidoc +++ b/heartbeat/docs/getting-started.asciidoc @@ -253,8 +253,13 @@ sudo /etc/init.d/heartbeat start ["source","sh",subs="attributes"] ---------------------------------------------------------------------- +sudo chown root heartbeat.yml <1> sudo ./heartbeat -e -c heartbeat.yml -d "publish" ---------------------------------------------------------------------- +<1> You'll be running Heartbeat as root, so you need to change ownership +of the configuration file (see +{libbeat}config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_). *win:* diff --git a/heartbeat/docs/index.asciidoc b/heartbeat/docs/index.asciidoc index 8470e621f539..680217e1d8f1 100644 --- a/heartbeat/docs/index.asciidoc +++ b/heartbeat/docs/index.asciidoc @@ -7,6 +7,7 @@ include::../../libbeat/docs/version.asciidoc[] :metricbeat: http://www.elastic.co/guide/en/beats/metricbeat/{doc-branch} :filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch} :winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch} +:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} :elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 :downloads: https://artifacts.elastic.co/downloads/beats @@ -42,6 +43,7 @@ include::./configuring-logstash.asciidoc[] include::../../libbeat/docs/shared-env-vars.asciidoc[] +:standalone: :allplatforms: include::../../libbeat/docs/yaml.asciidoc[] diff --git a/heartbeat/docs/reference/configuration.asciidoc b/heartbeat/docs/reference/configuration.asciidoc index 3304e7e4535d..b73f68540405 100644 --- a/heartbeat/docs/reference/configuration.asciidoc +++ b/heartbeat/docs/reference/configuration.asciidoc @@ -21,6 +21,7 @@ configuration settings, you need to restart Heartbeat to pick up the changes. * <> * <> * <> +* <> * <> * <> * <> diff --git a/heartbeat/docs/reference/configuration/heartbeat-options.asciidoc b/heartbeat/docs/reference/configuration/heartbeat-options.asciidoc index a4144a02eb17..f283e6dc5ec6 100644 --- a/heartbeat/docs/reference/configuration/heartbeat-options.asciidoc +++ b/heartbeat/docs/reference/configuration/heartbeat-options.asciidoc @@ -1,5 +1,5 @@ [[configuration-heartbeat-options]] -=== Heartbeat Configuration +=== Heartbeat The `heartbeat` section of the +heartbeat.yml+ config file specifies the list of `monitors` that Heartbeat uses to check your remote hosts to diff --git a/libbeat/docs/communitybeats.asciidoc b/libbeat/docs/communitybeats.asciidoc index fb10c0aff677..ee0e930e6a58 100644 --- a/libbeat/docs/communitybeats.asciidoc +++ b/libbeat/docs/communitybeats.asciidoc @@ -32,6 +32,7 @@ https://github.com/icinga/icingabeat[icingabeat]:: Icingabeat ships events and s https://github.com/devopsmakers/iobeat[iobeat]:: Reads IO stats from /proc/diskstats on Linux. https://github.com/radoondas/jmxproxybeat[jmxproxybeat]:: Reads Tomcat JMX metrics exposed over 'JMX Proxy Servlet' to HTTP. https://github.com/mheese/journalbeat[journalbeat]:: Used for log shipping from systemd/journald based Linux systems. +https://github.com/dearcode/kafkabeat[kafkabeat]:: read data from kafka with Consumer-groups. https://github.com/eskibars/lmsensorsbeat[lmsensorsbeat]:: Collects data from lm-sensors (such as CPU temperatures, fan speeds, and voltages from i2c and smbus). https://github.com/consulthys/logstashbeat[logstashbeat]:: Collects data from Logstash monitoring API (v5 onwards) and indexes them in Elasticsearch. https://github.com/yedamao/mcqbeat[mcqbeat]:: Reads the status of queues from memcacheq. @@ -59,7 +60,7 @@ https://github.com/hartfordfive/udplogbeat[udplogbeat]:: Accept events via local https://github.com/cleesmith/unifiedbeat[unifiedbeat]:: Reads records from Unified2 binary files generated by network intrusion detection software and indexes the records in Elasticsearch. https://github.com/mrkschan/uwsgibeat[uwsgibeat]:: Reads stats from uWSGI. -https://github.com/eskibars/wmibeat[wmibeat]:: Uses WMI to grab your favorite, configurable Windows metrics. +https://github.com/eskibars/wmibeat[wmibeat]:: Uses WMI to grab your favorite, configurable Windows metrics. Have you created a Beat that's not listed? If so, add the name and description of your Beat to the source document for diff --git a/libbeat/docs/config-file-format.asciidoc b/libbeat/docs/config-file-format.asciidoc index 73a859932567..9a08649d824c 100644 --- a/libbeat/docs/config-file-format.asciidoc +++ b/libbeat/docs/config-file-format.asciidoc @@ -375,44 +375,5 @@ individual settings can be overwritten using `-E =`. [[config-file-format-tips]] === YAML Tips and Gotchas -When you edit the configuration file, there are a few things that you should know. - -[float] -==== Use Spaces for Indentation - -Indentation is meaningful in YAML. Make sure that you use spaces, rather than -tab characters, to indent sections. - -In the default configuration files and in all the examples in the documentation, -we use 2 spaces per indentation level. We recommend you do the same. - -[float] -==== Look at the Default Config File for Structure - -The best way to understand where to define a configuration option is by looking -at the provided sample configuration files. The configuration files contain most -of the default configurations that are available per beat. To change a setting, -simply uncomment the line and change the values. - -[float] -==== Test Your Config File - -You can test your configuration file to verify that the structure is valid. -Simply change to the directory where the binary is installed, and run -your Beat in the foreground with the `-configtest` flag specified. For example: - -["source","yaml",subs="attributes,callouts"] ----------------------------------------------------------------------- -filebeat -c filebeat.yml -configtest ----------------------------------------------------------------------- - -You'll see a message if an error in the configuration file is found. - -[float] -==== Wrap Regular Expressions in Single Quotation Marks - -If you need to specify a regular expression in a YAML file, it's a good idea to -wrap the regular expression in single quotation marks to work around YAML's -tricky rules for string escaping. - -For more information about YAML, see http://yaml.org/. +:allplatforms: +include::yaml.asciidoc[] diff --git a/libbeat/docs/dashboardsconfig.asciidoc b/libbeat/docs/dashboardsconfig.asciidoc index b894683007f5..7d4dfee9f2c1 100644 --- a/libbeat/docs/dashboardsconfig.asciidoc +++ b/libbeat/docs/dashboardsconfig.asciidoc @@ -11,22 +11,24 @@ ////////////////////////////////////////////////////////////////////////// [[configuration-dashboards]] -=== Dashboards Configuration +=== Dashboards beta[] The `dashboards` section of the +{beatname_lc}.yml+ config file contains options -for the automatic loading of the sample Beats dashboards. The loading of the -dashboards is disabled by default, but can be enabled either from the configuration +for automatically loading the sample Beats dashboards. Automatic dashboard +loading is disabled by default, but can be enabled either from the configuration file or by using the `-setup` CLI flag. If dashboard loading is enabled, {beatname_uc} attempts to configure Kibana by writing directly in the Elasticsearch index for the Kibana configuration (by -default, `.kibana`). To connect to Elasticsearch, it uses the settings defined -in the Eleasticsearch output. If the Elasticsearch output is not configured or -not enabled, {beatname_uc} will stop with an error. Loading the dashboards is -only attempted at the Beat start, if Elasticsearch is not available when the -Beat starts, {beatname_uc} will stop with an error. +default, `.kibana`). To connect to Elasticsearch, {beatname_uc} uses the +settings defined in the Elasticsearch output. If the Elasticsearch output is +not configured or not enabled, {beatname_uc} will stop with an error. Dashboard +loading is only attempted at Beat startup. If Elasticsearch is not available when +the Beat starts, {beatname_uc} will stop with an error. + +Here is an example configuration: [source,yaml] ------------------------------------------------------------------------------ @@ -40,48 +42,64 @@ You can specify the following options in the `dashboards` section of the ===== enabled -If enabled, load the sample Kibana dashboards on startup. If no other options -are set, the dashboards archive is downloaded from the elastic.co website. +If this option is set to true, {beatname_uc} loads the sample Kibana dashboards +automatically on startup. If no other options are set, the dashboard archive is +downloaded from the elastic.co website. + +To load dashboards from a different location, you can +configure one of the following options: <>, +<>, or <>. + +To load dashboards from a snapshot URL, use the <> +option and optionally <>. +[[url-option]] ===== url -The URL from where to download the dashboards archive. By default this URL has a -value which is computed based on the Beat name and version. For released -versions, this URL points to the dashboard archive on the artifacts.elastic.co +The URL to use for downloading the dashboard archive. By default this URL +is computed based on the Beat name and version. For released versions, +this URL points to the dashboard archive on the artifacts.elastic.co website. +[[directory-option]] ===== directory -The directory from where to read the dashboards. It is used instead of the URL -when it has a value. +The directory that contains the dashboards to load. If this option is set, +{beatname_uc} looks for dashboards in the specified directory instead of +downloading an archive from a URL. +[[file-option]] ===== file -The file archive (zip file) from where to read the dashboards. It is used -instead of the URL when it has a value. +The file archive (zip file) that contains the dashboards to load. If this option +is set, {beatname_uc} looks for a dashboard archive in the specified path +instead of downloading the archive from a URL. +[[snapshot-option]] ===== snapshot If this option is set to true, the snapshot URL is used instead of the default URL. +[[snapshot-url-option]] ===== snapshot_url -The URL from where to download the snapshot version of the dashboards. By -default this has a value which is computed based on the Beat name and version. +The URL to use for downloading the snapshot version of the dashboards. By +default the snapshot URL is computed based on the Beat name and version. ===== beat -In case the archive contains the dashboards from multiple Beats, this lets you -select which one to load. You can load all the dashboards in the archive by -setting this to the empty string. The default is "{beatname_lc}". +In case the archive contains the dashboards for multiple Beats, this setting +lets you select the Beat for which you want to load dashboards. To load all the +dashboards in the archive, set this option to an empty string. The default is ++"{beatname_lc}"+. ===== kibana_index -The name of the Kibana index to use for setting the configuration. Default is -".kibana" +The name of the Kibana index to use for setting the configuration. The default +is `".kibana"` ===== index -The Elasticsearch index name. This overwrites the index name defined in the -dashboards and index pattern. Example: "testbeat-*" +The Elasticsearch index name. This setting overwrites the index name defined +in the dashboards and index pattern. Example: `"testbeat-*"` diff --git a/libbeat/docs/generalconfig.asciidoc b/libbeat/docs/generalconfig.asciidoc index 08c406519f57..c1a7c54082cf 100644 --- a/libbeat/docs/generalconfig.asciidoc +++ b/libbeat/docs/generalconfig.asciidoc @@ -11,7 +11,7 @@ ////////////////////////////////////////////////////////////////////////// [[configuration-general]] -=== General Configuration +=== General The general section of the +{beatname_lc}.yml+ config file contains configuration options for the Beat and some general settings that control its behaviour. diff --git a/libbeat/docs/gettingstarted.asciidoc b/libbeat/docs/gettingstarted.asciidoc index 96c81fce5e5f..e508c5fe8827 100644 --- a/libbeat/docs/gettingstarted.asciidoc +++ b/libbeat/docs/gettingstarted.asciidoc @@ -267,9 +267,8 @@ endif::[] ==== Setting Up Logstash In this setup, the Beat sends events to Logstash. Logstash receives -these events by using the -https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] and then sends the transaction to Elasticsearch by using the -http://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html[Elasticsearch +these events by using the {logstashdoc}/plugins-inputs-beats.html[Beats input plugin for Logstash] and then sends the transaction to Elasticsearch by using the +{logstashdoc}/plugins-outputs-elasticsearch.html[Elasticsearch output plugin for Logstash]. The Elasticsearch output plugin uses the bulk API, making indexing very efficient. @@ -305,6 +304,7 @@ and to index into Elasticsearch. You configure Logstash by creating a configuration file. For example, you can save the following example configuration to a file called `logstash.conf`: + +-- [source,ruby] ------------------------------------------------------------------------------ input { @@ -317,15 +317,22 @@ output { elasticsearch { hosts => "localhost:9200" manage_template => false - index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" - document_type => "%{[@metadata][type]}" + index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" <1> + document_type => "%{[@metadata][type]}" <2> } } ------------------------------------------------------------------------------ -+ +<1> `%{[@metadata][beat]}` sets the first part of the index name to the value +of the `beat` metadata field, and `%{+YYYY.MM.dd}` sets the second part of the +name to a date based on the Logstash `@timestamp` field. For example: ++{beatname_lc}-2017.03.29+. +<2> `%{[@metadata][type]}` sets the document type based on the value of the `type` +metadata field. + Logstash uses this configuration to index events in Elasticsearch in the same way that the Beat would, but you get additional buffering and other capabilities provided by Logstash. +-- To use this setup, you'll also need to configure your Beat to use Logstash. For more information, see the documentation for the Beat. diff --git a/libbeat/docs/index.asciidoc b/libbeat/docs/index.asciidoc index eb4b3c82676d..d702c77d80dc 100644 --- a/libbeat/docs/index.asciidoc +++ b/libbeat/docs/index.asciidoc @@ -9,6 +9,7 @@ include::./version.asciidoc[] :winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch} :heartbeat: http://www.elastic.co/guide/en/beats/heartbeat/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 +:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} :beatname_lc: beatname :beatname_uc: a Beat :security: X-Pack Security @@ -33,14 +34,10 @@ include::./upgrading.asciidoc[] include::./config-file-format.asciidoc[] -pass::[] - include::./newbeat.asciidoc[] include::./event-conventions.asciidoc[] include::./newdashboards.asciidoc[] -pass::[] - include::./release.asciidoc[] diff --git a/libbeat/docs/loggingconfig.asciidoc b/libbeat/docs/loggingconfig.asciidoc index c03201b2b829..df7d7e11f12b 100644 --- a/libbeat/docs/loggingconfig.asciidoc +++ b/libbeat/docs/loggingconfig.asciidoc @@ -11,7 +11,7 @@ ////////////////////////////////////////////////////////////////////////// [[configuration-logging]] -=== Logging Configuration +=== Logging The `logging` section of the +{beatname_lc}.yml+ config file contains options for configuring the Beats logging output. The logging system can write logs to diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index da8336a83169..de0e55a2668f 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -11,7 +11,7 @@ ////////////////////////////////////////////////////////////////////////// [[elasticsearch-output]] -=== Elasticsearch Output Configuration +=== Elasticsearch Output When you specify Elasticsearch for the output, the Beat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API. @@ -359,33 +359,54 @@ See <> for more information. [[logstash-output]] -=== Logstash Output Configuration +=== Logstash Output -The Logstash output sends the events directly to Logstash by using the lumberjack -protocol, which runs over TCP. To use this option, you must +*Prerequisite:* To use Logstash as an output, you must {libbeat}/logstash-installation.html#logstash-setup[install and configure] the Beats input -plugin for Logstash. Logstash allows for additional processing and routing of +plugin for Logstash. + +The Logstash output sends the events directly to Logstash by using the lumberjack +protocol, which runs over TCP. Logstash allows for additional processing and routing of generated events. -Every event sent to Logstash contains additional metadata for indexing and filtering: +Here is an example of how to configure {beatname_uc} to use Logstash: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.logstash: + hosts: ["localhost:5044"] +------------------------------------------------------------------------------ + +==== Accessing Metadata Fields + +Every event sent to Logstash contains the following metadata fields that you can +use in Logstash for indexing and filtering: -[source,json] +["source","json",subs="attributes"] ------------------------------------------------------------------------------ { ... - "@metadata": { - "beat": "", - "type": "" + "@metadata": { <1> + "beat": "{beatname_lc}", <2> + "type": "" <3> } } ------------------------------------------------------------------------------ - -In Logstash, you can configure the Elasticsearch output plugin to use the -metadata and event type for indexing. - -The following Logstash configuration file for the versions 2.x and 5.x sets Logstash to -use the index and document type reported by Beats for indexing events into Elasticsearch. -The index used will depend on the `@timestamp` field as identified by Logstash. +<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. The +contents of the `@metadata` field only exist in Logstash and are not part of any +events sent from Logstash. See the +{logstashdoc}/event-dependent-configuration.html#metadata[Logstash documentation] +for more about the `@metadata` field. +<2> The default is {beatname_lc}. To change this value, set the +<> option in the {beatname_uc} config file. +<3> The value of `type` varies depending on the event type. + +You can access this metadata from within the Logstash config file to set values +dynamically based on the contents of the metadata. + +For example, the following Logstash configuration file for versions 2.x and +5.x sets Logstash to use the index and document type reported by Beats for +indexing events into Elasticsearch: [source,logstash] ------------------------------------------------------------------------------ @@ -399,24 +420,21 @@ input { output { elasticsearch { hosts => ["http://localhost:9200"] - index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" - document_type => "%{[@metadata][type]}" + index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" <1> + document_type => "%{[@metadata][type]}" <2> } } ------------------------------------------------------------------------------ +<1> `%{[@metadata][beat]}` sets the first part of the index name to the value +of the `beat` metadata field, and `%{+YYYY.MM.dd}` sets the second part of the +name to a date based on the Logstash `@timestamp` field. For example: ++{beatname_lc}-2017.03.29+. +<2> `%{[@metadata][type]}` sets the document type based on the value of the `type` +metadata field. Events indexed into Elasticsearch with the Logstash configuration shown here will be similar to events directly indexed by Beats into Elasticsearch. -Here is an example of how to configure the Beat to use Logstash: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.logstash: - hosts: ["localhost:5044"] - index: {beatname_lc} ------------------------------------------------------------------------------- - ==== Compatibility This output works with all compatible versions of Logstash. See "Supported Beats Versions" in the https://www.elastic.co/support/matrix#show_compatibility[Elastic Support Matrix]. @@ -516,6 +534,7 @@ The `proxy_use_local_resolver` option determines if Logstash hostnames are resolved locally when using a proxy. The default value is false which means that when a proxy is used the name resolution occurs on the proxy server. +[[logstash-index]] ===== index The index root name to write events to. The default is the Beat name. @@ -562,7 +581,7 @@ Elasticsearch. Beats that publish data in batches (such as Filebeat) send events spooler size. [[kafka-output]] -=== Kafka Output Configuration +=== Kafka Output The Kafka output sends the events to Apache Kafka. @@ -760,7 +779,7 @@ Configuration options for SSL parameters like the root CA for Kafka connections. <> for more information. [[redis-output]] -=== Redis Output Configuration +=== Redis Output The Redis output inserts the events into a Redis list or a Redis channel. This output plugin is compatible with @@ -981,7 +1000,7 @@ This option determines whether Redis hostnames are resolved locally when using a The default value is false, which means that name resolution occurs on the proxy server. [[file-output]] -=== File Output Configuration +=== File Output The File output dumps the transactions into a file where each transaction is in a JSON format. Currently, this output is used for testing, but it can be used as input for @@ -1036,7 +1055,7 @@ Output codec configuration. If the `codec` section is missing, events will be js See <> for more information. [[console-output]] -=== Console Output Configuration +=== Console Output The Console output writes events in JSON format to stdout. @@ -1079,7 +1098,7 @@ Setting `bulk_max_size` to 0 disables buffering in libbeat. [[configuration-output-ssl]] -=== SSL Configuration +=== SSL You can specify SSL options for any output that supports SSL. @@ -1215,7 +1234,7 @@ The following elliptic curve types are available: * P-521 [[configuration-output-codec]] -=== Output Codec Configuration +=== Output Codec For outputs that do not require a specific encoding, you can change the encoding by using the codec configuration. You can specify either the `json` or `format` diff --git a/libbeat/docs/repositories.asciidoc b/libbeat/docs/repositories.asciidoc index 47853efe7ab7..963ea52c8467 100644 --- a/libbeat/docs/repositories.asciidoc +++ b/libbeat/docs/repositories.asciidoc @@ -70,6 +70,21 @@ the following: Simply delete the `deb-src` entry from the `/etc/apt/sources.list` file, and the installation should work as expected. ================================================== +ifeval::["{beatname_uc}"=="Heartbeat"] + +. On Debian or Ubuntu, pin the repository before installing to ensure that the +correct Elastic Heartbeat package is installed. To do this, edit +`/etc/apt/preferences` (or `/etc/apt/preferences.d/heartbeat`) as follows: ++ +[source,shell] +-------------------------------------------------- +Package: heartbeat +Pin: origin artifacts.elastic.co +Pin-Priority: 700 +-------------------------------------------------- + +endif::[] + . Run `apt-get update`, and the repository is ready for use. For example, you can install {beatname_uc} by running: + diff --git a/libbeat/docs/shared-config-ingest.asciidoc b/libbeat/docs/shared-config-ingest.asciidoc index 99282db6a5f6..a86db6769001 100644 --- a/libbeat/docs/shared-config-ingest.asciidoc +++ b/libbeat/docs/shared-config-ingest.asciidoc @@ -50,7 +50,7 @@ To add the pipeline in Elasticsearch, you would run: [source,shell] ------------------------------------------------------------------------------ -curl -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@pipeline.json +curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@pipeline.json ------------------------------------------------------------------------------ Then in the +{beatname_lc}.yml+ file, you would specify: diff --git a/libbeat/docs/shared-faq.asciidoc b/libbeat/docs/shared-faq.asciidoc index a287878c77e5..dde20cc2826b 100644 --- a/libbeat/docs/shared-faq.asciidoc +++ b/libbeat/docs/shared-faq.asciidoc @@ -21,6 +21,19 @@ You may encounter errors loading the config file on POSIX operating systems if: See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] for more about resolving these errors. +[float] +[[error-found-unexpected-character]] +=== Found Unexpected or Unknown Characters? + +Either there is a problem with the structure of your config file, or you have +used a path or expression that the YAML parser cannot resolve because the config +file contains characters that aren't properly escaped. + +If the YAML file contains paths with spaces or unusual characters, wrap the +paths in single quotation marks (see <>). + +Also see the general advice under <>. + [float] [[connection-problem]] === Logstash connection doesn't work? diff --git a/libbeat/docs/shared-logstash-config.asciidoc b/libbeat/docs/shared-logstash-config.asciidoc index 51a40b8b885f..3024d04ddec3 100644 --- a/libbeat/docs/shared-logstash-config.asciidoc +++ b/libbeat/docs/shared-logstash-config.asciidoc @@ -9,6 +9,10 @@ //// include::../../libbeat/docs/shared-logstash-config.asciidoc[] ////////////////////////////////////////////////////////////////////////// +*Prerequisite:* To use Logstash as an output, you must also +{libbeat}/logstash-installation.html#logstash-setup[set up Logstash] to receive events +from Beats. + If you want to use Logstash to perform additional processing on the data collected by {beatname_uc}, you need to configure {beatname_uc} to use Logstash. @@ -47,7 +51,4 @@ options specified: +.\winlogbeat.exe -c .\winlogbeat.yml -configtest -e+. endif::win[] -To use this configuration, you must also -{libbeat}/logstash-installation.html#logstash-setup[set up Logstash] to receive events -from Beats. diff --git a/libbeat/docs/shared-path-config.asciidoc b/libbeat/docs/shared-path-config.asciidoc index 7aac4ca428e2..eaadea3179bc 100644 --- a/libbeat/docs/shared-path-config.asciidoc +++ b/libbeat/docs/shared-path-config.asciidoc @@ -11,7 +11,7 @@ ////////////////////////////////////////////////////////////////////////// [[configuration-path]] -=== Paths Configuration +=== Paths The `path` section of the +{beatname_lc}.yml+ config file contains configuration options that define where the Beat looks for its files. For example, all Beats diff --git a/libbeat/docs/shared-template-load.asciidoc b/libbeat/docs/shared-template-load.asciidoc index 78462f860f91..3ce8def274c5 100644 --- a/libbeat/docs/shared-template-load.asciidoc +++ b/libbeat/docs/shared-template-load.asciidoc @@ -67,7 +67,7 @@ ifdef::allplatforms[] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- -curl -XPUT 'http://localhost:9200/_template/{beatname_lc}' -d@/etc/{beatname_lc}/{beatname_lc}.template.json +curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/{beatname_lc}' -d@/etc/{beatname_lc}/{beatname_lc}.template.json ---------------------------------------------------------------------- *mac:* @@ -75,7 +75,7 @@ curl -XPUT 'http://localhost:9200/_template/{beatname_lc}' -d@/etc/{beatname_lc} ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- cd {beatname_lc}-{version}-darwin-x86_64 -curl -XPUT 'http://localhost:9200/_template/{beatname_lc}' -d@{beatname_lc}.template.json +curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/{beatname_lc}' -d@{beatname_lc}.template.json ---------------------------------------------------------------------- *win:* @@ -84,7 +84,7 @@ endif::allplatforms[] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- -PS C:\Program Files{backslash}{beatname_uc}> Invoke-WebRequest -Method Put -InFile {beatname_lc}.template.json -Uri http://localhost:9200/_template/{beatname_lc}?pretty +PS C:\Program Files{backslash}{beatname_uc}> Invoke-WebRequest -Method Put -InFile {beatname_lc}.template.json -Uri http://localhost:9200/_template/{beatname_lc}?pretty -ContentType application/json ---------------------------------------------------------------------- where `localhost:9200` is the IP and port where Elasticsearch is listening. diff --git a/libbeat/docs/yaml.asciidoc b/libbeat/docs/yaml.asciidoc index 62739625a977..7a24bdab9a37 100644 --- a/libbeat/docs/yaml.asciidoc +++ b/libbeat/docs/yaml.asciidoc @@ -9,31 +9,38 @@ //// include::../../libbeat/docs/yaml.asciidoc[] ////////////////////////////////////////////////////////////////////////// +ifdef::standalone[] + [[yaml-tips]] == YAML Tips and Gotchas -The {beatname_uc} configuration file uses http://yaml.org/[YAML] for its syntax. When you edit the +endif::[] + +The configuration file uses http://yaml.org/[YAML] for its syntax. When you edit the file to modify configuration settings, there are a few things that you should know. [float] === Use Spaces for Indentation -Indentation is meaningful in YAML. Make sure that you use spaces, rather than tab characters, to indent sections. +Indentation is meaningful in YAML. Make sure that you use spaces, rather than tab characters, to indent sections. + +In the default configuration files and in all the examples in the documentation, +we use 2 spaces per indentation level. We recommend you do the same. [float] === Look at the Default Config File for Structure -The best way to understand where to define a configuration option is by looking at -the {beatname_lc}.yml configuration file. The configuration file contains most of the -configuration options that are available for {beatname_uc}. To change a configuration setting, -simply uncomment the line and change the setting. +The best way to understand where to define a configuration option is by looking +at the provided sample configuration files. The configuration files contain most +of the default configurations that are available for the Beat. To change a setting, +simply uncomment the line and change the values. [float] === Test Your Config File You can test your configuration file to verify that the structure is valid. Simply change to the directory where the binary is installed, and run -{beatname_uc} in the foreground with the `-configtest` flag specified. For example: +the Beat in the foreground with the `-configtest` flag specified. For example: ifdef::allplatforms[] @@ -53,11 +60,33 @@ ifdef::win[] endif::win[] -You'll see a message if {beatname_uc} finds an error in the file. +You'll see a message if the Beat finds an error in the file. [float] === Wrap Regular Expressions in Single Quotation Marks If you need to specify a regular expression in a YAML file, it's a good idea to wrap the regular expression in single quotation marks to work around YAML's tricky rules for string escaping. -For more information about YAML, see http://yaml.org/. \ No newline at end of file +For more information about YAML, see http://yaml.org/. + +[float] +[[wrap-paths-in-quotes]] +=== Wrap Paths in Single Quotation Marks + +Windows paths in particular sometimes contain spaces or characters, such as drive +letters or triple dots, that may be misinterpreted by the YAML parser. + +To avoid this problem, it's a good idea to wrap paths in single quotation marks. + +[float] +[[avoid-leading-zeros]] +=== Avoid Using Leading Zeros in Numeric Values + +If you use a leading zero (for example, `09`) in a numeric field without +wrapping the value in single quotation marks, the value may be interpreted +incorrectly by the YAML parser. If the value is a valid octal, it's converted +to an integer. If not, it's converted to a float. + +To prevent unwanted type conversions, avoid using leading zeros in field values, +or wrap the values in single quotation marks. + diff --git a/metricbeat/docs/gettingstarted.asciidoc b/metricbeat/docs/gettingstarted.asciidoc index 6d2ae881d35d..c3b737d0bdb5 100644 --- a/metricbeat/docs/gettingstarted.asciidoc +++ b/metricbeat/docs/gettingstarted.asciidoc @@ -259,8 +259,13 @@ sudo /etc/init.d/metricbeat start [source,shell] ---------------------------------------------------------------------- +sudo chown root metricbeat.yml <1> sudo ./metricbeat -e -c metricbeat.yml -d "publish" ---------------------------------------------------------------------- +<1> You'll be running Metricbeat as root, so you need to change ownership +of the configuration file (see +{libbeat}config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_). *win:* diff --git a/metricbeat/docs/index.asciidoc b/metricbeat/docs/index.asciidoc index 536a76b0b9bb..9454e1709310 100644 --- a/metricbeat/docs/index.asciidoc +++ b/metricbeat/docs/index.asciidoc @@ -4,6 +4,7 @@ include::../../libbeat/docs/version.asciidoc[] :libbeat: http://www.elastic.co/guide/en/beats/libbeat/{doc-branch} :filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch} +:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} :elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 :version: {stack-version} @@ -37,6 +38,7 @@ include::./configuring-logstash.asciidoc[] include::../../libbeat/docs/shared-env-vars.asciidoc[] +:standalone: :allplatforms: include::../../libbeat/docs/yaml.asciidoc[] diff --git a/metricbeat/docs/reference/configuration.asciidoc b/metricbeat/docs/reference/configuration.asciidoc index 5a36941570ce..5133ada7a2cf 100644 --- a/metricbeat/docs/reference/configuration.asciidoc +++ b/metricbeat/docs/reference/configuration.asciidoc @@ -20,6 +20,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> * <> * <> * <> diff --git a/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc b/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc index 8521a9c0dba6..199f4ab4ac0d 100644 --- a/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc +++ b/metricbeat/docs/reference/configuration/metricbeat-options.asciidoc @@ -1,5 +1,5 @@ [[configuration-metricbeat]] -=== Modules Configuration +=== Modules The `metricbeat.modules` section of the +{beatname_lc}.yml+ config file contains an array with the enabled <>. The following example shows a configuration where the Apache and MySQL modules are enabled: diff --git a/metricbeat/docs/reference/configuration/reload-configuration.asciidoc b/metricbeat/docs/reference/configuration/reload-configuration.asciidoc index 6c45764e6251..e0e37d3e7ee7 100644 --- a/metricbeat/docs/reference/configuration/reload-configuration.asciidoc +++ b/metricbeat/docs/reference/configuration/reload-configuration.asciidoc @@ -1,14 +1,22 @@ +[[metricbeat-configuration-reloading]] === Reload Configuration beta[] -Reload configuration allows to dynamically reload module configuration files. A glob can be defined which should be watched - for module configuration changes. New modules will started / stopped accordingly. This is especially useful in - container environments where 1 container is used to monitor all services in other containers on the same host. - New containers appear and disappear dynamically which requires changes to which modules - are needed and which hosts must be monitored. +You can configure Metricbeat to dynamically reload configuration files when +there are changes. To do this, you specify a path +(https://golang.org/pkg/path/filepath/#Glob[Glob]) to watch for module +configuration changes. When the files found by the Glob change, new modules are +started/stopped according to changes in the configuration files. -The configuration in the main metricbeat.yml config file looks as following: +This feature is especially useful in container environments where one container +is used to monitor all services running in other containers on the same host. +Because new containers appear and disappear dynamically, you may need to change +the Metricbeat configuration frequently to specify which modules are needed and +which hosts must be monitored. + +To enable dynamic config reloading, you specify the `path` and `reload` options +in the main `metricbeat.yml` config file. For example: [source,yaml] ------------------------------------------------------------------------------ @@ -18,11 +26,16 @@ metricbeat.config.modules: reload.period: 10s ------------------------------------------------------------------------------ -A path with a glob must be defined on which files should be checked for changes. A period is set on how often -the files are checked for changes. Do not set period below 1s as the modification time of files is often stored in seconds. -Setting it below 1s will have an unnecessary overhead. +`path`:: A Glob that defines the files to check for changes. +`reload.enabled`:: When set to `true`, enables dynamic config reload. +`reload.period`:: Specifies how often the files are checked for changes. Do not +set the `period` to less than 1s because the modification time of files is often +stored in seconds. Setting the `period` to less than 1s will result in +unnecessary overhead. + +Each file found by the Glob must contain a list of one or more module +definitions. For example: -The configuration inside the files which are found by the glob look as following: [source,yaml] ------------------------------------------------------------------------------ - module: system @@ -35,5 +48,3 @@ The configuration inside the files which are found by the glob look as following enabled: true period: 10s ------------------------------------------------------------------------------ - -Each file directly contains a list of modules. Each file can contain one or multiple module definitions. diff --git a/packetbeat/docs/gettingstarted.asciidoc b/packetbeat/docs/gettingstarted.asciidoc index 671a784e6f58..fc0ee462f6ac 100644 --- a/packetbeat/docs/gettingstarted.asciidoc +++ b/packetbeat/docs/gettingstarted.asciidoc @@ -239,6 +239,9 @@ binary is installed, and run Packetbeat in the foreground with the following options specified: +sudo ./packetbeat -configtest -e+. Make sure your config files are in the path expected by Packetbeat (see <>). If you installed from DEB or RPM packages, run +sudo ./packetbeat.sh -configtest -e+. +Depending on your OS, you might run into file ownership issues when you run this +test. See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_ for more information. [[packetbeat-template]] === Step 3: Loading the Index Template in Elasticsearch @@ -273,8 +276,13 @@ sudo /etc/init.d/packetbeat start [source,shell] ---------------------------------------------------------------------- +sudo chown root packetbeat.yml <1> sudo ./packetbeat -e -c packetbeat.yml -d "publish" ---------------------------------------------------------------------- +<1> You'll be running Packetbeat as root, so you need to change ownership +of the configuration file (see +{libbeat}config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_). *win:* diff --git a/packetbeat/docs/index.asciidoc b/packetbeat/docs/index.asciidoc index a58d044278c9..c3628c56b8f2 100644 --- a/packetbeat/docs/index.asciidoc +++ b/packetbeat/docs/index.asciidoc @@ -7,8 +7,8 @@ include::../../libbeat/docs/version.asciidoc[] :metricbeat: http://www.elastic.co/guide/en/beats/metricbeat/{doc-branch} :filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch} :winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch} -:elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} +:elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 :kibanadoc: https://www.elastic.co/guide/en/kibana/{doc-branch} :plugindoc: https://www.elastic.co/guide/en/elasticsearch/plugins/{doc-branch} @@ -48,6 +48,7 @@ include::./thrift.asciidoc[] include::./maintaining-topology.asciidoc[] +:standalone: :allplatforms: include::../../libbeat/docs/yaml.asciidoc[] diff --git a/packetbeat/docs/reference/configuration.asciidoc b/packetbeat/docs/reference/configuration.asciidoc index f5320deb5623..d661da0b6ee5 100644 --- a/packetbeat/docs/reference/configuration.asciidoc +++ b/packetbeat/docs/reference/configuration.asciidoc @@ -23,6 +23,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> * <> * <> * <> diff --git a/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc b/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc index 317e60b47928..8267357974b7 100644 --- a/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc +++ b/packetbeat/docs/reference/configuration/packetbeat-options.asciidoc @@ -1,5 +1,5 @@ [[configuration-interfaces]] -=== Network Device Configuration +=== Network Device (Interfaces) The `interfaces` section of the +{beatname_lc}.yml+ config file configures the sniffer. Here is an example configuration: @@ -198,7 +198,7 @@ see the following published transactions (when `ignore_outgoing` is true): [[configuration-flows]] -=== Flows Configuration +=== Flows The `flows` section of the +{beatname_lc}.yml+ config file contains configuration options for bidirectional network flows. If section is missing from configuration file, network flows are @@ -236,7 +236,7 @@ disabled, flows are still reported once being timed out. The default value is [[configuration-protocols]] -=== Transaction Protocols Configuration +=== Transaction Protocols The `protocols` section of the +{beatname_lc}.yml+ config file contains configuration options for each supported protocol, including common options like `enabled`, `ports`, `send_request`, `send_response`, and options that are protocol-specific. @@ -731,7 +731,7 @@ formatted JSON objects. [[configuration-processes]] -=== Monitored Processes Configuration +=== Monitored Processes This section of the +{beatname_lc}.yml+ config file is optional, but configuring the processes enables Packetbeat to show you not only the servers that the diff --git a/packetbeat/docs/reference/configuration/runconfig.asciidoc b/packetbeat/docs/reference/configuration/runconfig.asciidoc index 9f5f2dfe6396..8f1e70e4959c 100644 --- a/packetbeat/docs/reference/configuration/runconfig.asciidoc +++ b/packetbeat/docs/reference/configuration/runconfig.asciidoc @@ -1,5 +1,5 @@ [[configuration-run-options]] -=== Run Options Configuration +=== Run Options The Beat can drop privileges after creating the sniffing socket. Root access is required for opening the socket, but everything else requires no diff --git a/winlogbeat/docs/index.asciidoc b/winlogbeat/docs/index.asciidoc index 53437fbccc69..bb6c2073f22e 100644 --- a/winlogbeat/docs/index.asciidoc +++ b/winlogbeat/docs/index.asciidoc @@ -7,6 +7,7 @@ include::../../libbeat/docs/version.asciidoc[] :metricbeat: http://www.elastic.co/guide/en/beats/metricbeat/{doc-branch} :filebeat: http://www.elastic.co/guide/en/beats/filebeat/{doc-branch} :winlogbeat: http://www.elastic.co/guide/en/beats/winlogbeat/{doc-branch} +:logstashdoc: https://www.elastic.co/guide/en/logstash/{doc-branch} :elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/{doc-branch} :securitydoc: https://www.elastic.co/guide/en/x-pack/5.2 :version: {stack-version} @@ -32,6 +33,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[] include::../../libbeat/docs/shared-env-vars.asciidoc[] +:standalone: :win: include::../../libbeat/docs/yaml.asciidoc[] diff --git a/winlogbeat/docs/reference/configuration.asciidoc b/winlogbeat/docs/reference/configuration.asciidoc index 0404226e06a3..d4f1db33f988 100644 --- a/winlogbeat/docs/reference/configuration.asciidoc +++ b/winlogbeat/docs/reference/configuration.asciidoc @@ -21,6 +21,7 @@ configuration settings, you need to restart {beatname_uc} to pick up the changes * <> * <> * <> +* <> * <> * <> * <> diff --git a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc index 531c839f9d31..0c7e7c3232ef 100644 --- a/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc +++ b/winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc @@ -2,7 +2,7 @@ supporting the Windows Event Log API (Microsoft Windows Vista and newer). [[configuration-winlogbeat-options]] -=== Winlogbeat Configuration +=== Winlogbeat The `winlogbeat` section of the +{beatname_lc}.yml+ config file specifies all options that are specific to Winlogbeat. Most importantly, it contains the list of event logs to monitor.