From 06714be79ed6e171bf42a6eecfa119e0a36d67fb Mon Sep 17 00:00:00 2001 From: DeDe Morton Date: Thu, 7 Sep 2017 06:07:52 -0700 Subject: [PATCH] Backport: Cherrypicks #4973 and #5068 into 6.0 (#5103) * [Docs] Document how to use modules.d directory (#4973) * [Docs] Document how to use modules.d directory * Add changes from review * Add upgrading guide docs (#5068) * Add upgrading guide docs Part of / related to #4737. --- filebeat/docs/configuring-howto.asciidoc | 3 + .../docs/filebeat-modules-options.asciidoc | 194 ++++++++++++--- filebeat/docs/filebeat-options.asciidoc | 6 + filebeat/docs/getting-started.asciidoc | 2 +- .../docs/modules-getting-started.asciidoc | 39 +-- filebeat/docs/modules-overview.asciidoc | 37 ++- libbeat/docs/breaking.asciidoc | 10 + libbeat/docs/command-reference.asciidoc | 8 +- libbeat/docs/upgrading.asciidoc | 234 +++++++----------- 9 files changed, 317 insertions(+), 216 deletions(-) diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc index 745b8ab45f5a..3a2d4d2f4a3f 100644 --- a/filebeat/docs/configuring-howto.asciidoc +++ b/filebeat/docs/configuring-howto.asciidoc @@ -19,6 +19,7 @@ _Beats Platform Reference_ for more about the structure of the config file. The following topics describe how to configure Filebeat: +* <> * <> * <> * <> @@ -39,6 +40,8 @@ The following topics describe how to configure Filebeat: -- +include::./filebeat-modules-options.asciidoc[] + include::./filebeat-options.asciidoc[] include::./multiple-prospectors.asciidoc[] diff --git a/filebeat/docs/filebeat-modules-options.asciidoc b/filebeat/docs/filebeat-modules-options.asciidoc index 865abec98bc1..a680464a574e 100644 --- a/filebeat/docs/filebeat-modules-options.asciidoc +++ b/filebeat/docs/filebeat-modules-options.asciidoc @@ -1,28 +1,105 @@ [[configuration-filebeat-modules]] == Specify which modules to run -//REVIEWERS: I created this topic because I think it might be confusing to have no mention of this modules config section in the docs. However, I think that adding the info might also be confusing becasue users might try to configure othe things (like prospectors). Note that this topic uses the old command syntax, but would be updated to reflect the new. +NOTE: Using Filebeat modules is optional. You may decide to +<> if you are using +a log file type that isn't supported, or you want to use a different setup. Filebeat <> provide a quick way for you to get started processing common log formats. They contain default configurations, -Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help -you implement and deploy a log monitoring solution. +Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help you +implement and deploy a log monitoring solution. -Using modules is optional. You may decide to configure Filebeat manaully if -you are using a log file type that isn't supported or you want to use a -different setup. +Filebeat provides a few different ways to enable modules. You can: -To enable specific modules, you can add entries to the `filebeat.modules` list -in the +{beatname_lc}.yml+ config file. Each entry in the list begins with a dash -(-) and is followed by settings for that module. +* <> +* <> +* <> -Filebeat also provides command-line options for enabling and disabling modules. -See <> for more about running modules. +When you enable modules, you can also +<> to change the default +behavior of the modules, and you can specify +<> to override prospector settings. -The following example shows a configuration that runs the `nginx`,`mysql`, and +Before running Filebeat with modules enabled, make sure you also set up the +environment to use Kibana dashboards. See <> for +more information. + +[float] +[[enable-modules-d-configs]] +=== Enable module configs in the `modules.d` directory + +The `modules.d` directory contains default configurations for all the modules +available in Filebeat. You can enable or disable specific module configurations +under `modules.d` by running the +<> commands. + +For example, to enable the `apache2` and `mysql` configs in the `modules.d` +directory, you use: + +[source,shell] +---- +./filebeat modules enable apache2 mysql +---- + +Then when you run Filebeat, it loads the corresponding module configurations +specified in the `modules.d` directory (for example, `modules.d/apache2.yml` and +`modules.d/mysql.yml`). + +To see a list of enabled and disabled modules, run: + +[source,shell] +---- +./filebeat modules list +---- + +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. + +[float] +[[enable-modules-cli]] +=== Enable modules when you run Filebeat + +To enable specific <> when you run Filebeat at the +command line, you can use the `--modules` flag. This approach works well when +you're getting started and want to specify different modules and settings each +time you run Filebeat. Any modules specified at the command line will be loaded +along with any modules that are enabled in the configuration file or `modules.d` +directory. If there's a conflict, the configuration specified at the command +line is used. + +The following example shows how to enable and run the `nginx`,`mysql`, and `system` modules. -//QUESTION: Are we doing something like the modules.d layout for Filebeat? +[source,shell] +---- +./filebeat -e --modules nginx,mysql,system +---- + +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. + +[float] +[[enable-modules-config-file]] +=== Enable module configs in the +{beatname_lc}.yml+ file + +Enabling <> directly in the config file is a practical +approach if you have upgraded from a previous version of {beatname_uc} and don't +want to move your module configs to the `modules.d` directory. You can continue +to configure modules in the +{beatname_lc}.yml+ file, but you won't be able to +use the `modules` command to enable and disable configurations because the +command requires the `modules.d` layout. + +To enable specific modules in the +{beatname_lc}.yml+ config file, you can add +entries to the +{beatname_lc}.modules+ list. Each entry in the list begins with +a dash (-) and is followed by settings for that module. + +The following example shows a configuration that runs the `nginx`,`mysql`, and +`system` modules. [source,yaml] ---- @@ -32,34 +109,93 @@ filebeat.modules: - module: system ---- -To enable this same configuration from the command line, you use: +The default module configurations assume that the logs you’re harvesting are +in the location expected for your OS and that the behavior of the module is +appropriate for your environment. To change the default configurations, you need +to specify variable settings. See <>. -[source,shell] ----- -./filebeat -e -modules=nginx,mysql,system ----- +[[specify-variable-settings]] +=== Specify variable settings -The configuration in the example assumes that the logs you’re harvesting are -in the location expected for your OS and that the default behavior of Filebeat -is appropriate for your environment. Each module provides variables that you -can set to fine tune the behavior of Filebeat, including the location -where it looks for log files. For example, the following configuration sets -the path for `nginx` log files: +Each module and fileset has variables that you can set to change the default +behavior of the module, including the paths where the module looks for log +files. For example, the `var.paths` setting in the following example sets the +path for `nginx` access log files: [source,yaml] ---- -filebeat.modules: - module: nginx access: var.paths: ["/var/log/nginx/access.log*"] ---- -To set the same configuration from the command line, you use: +To set the path for Nginx access log files at the command line, you use +the `-M` flag. For example: [source,shell] ---- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---- -See the <> you are using for -more information. +When you set variables at the command line, the variable name needs to include +the module and fileset name. You can specify multiple overrides. Each override +must start with `-M`. + +Here you see how to use the `-M` flag along with the `--modules` flag. This +example shows how to set the paths to the access and error logs: + +[source,shell] +---- +./filebeat --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" -M "nginx.error.var.paths=[/var/log/nginx/error.log*]" +---- + +For information about specific variables that you can set for each fileset, +see the <>. + +[[advanced-settings]] +=== Advanced settings + +Behind the scenes, each module starts a Filebeat prospector. Advanced users +can add or override any prospector settings. For example, you can set +<> to `true` in the module configuration: + +[source,yaml] +---------------------------------------------------------------------- +- module: nginx + access: + prospector: + close_eof: true +---------------------------------------------------------------------- + +Or at the command line like this: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "nginx.access.prospector.close_eof=true" +---------------------------------------------------------------------- + + +Here you see how to use the `-M` flag along with the `--modules` flag: + +[source,shell] +---------------------------------------------------------------------- +./filebeat --modules nginx -M "nginx.access.prospector.close_eof=true" +---------------------------------------------------------------------- + + +You can use wildcards to change variables or settings for multiple +modules/filesets at once. For example, the following command enables +`close_eof` for all the filesets in the `nginx` module: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "nginx.*.prospector.close_eof=true" +---------------------------------------------------------------------- + +The following command enables `close_eof` for all prospectors created by any of +the modules: + +[source,shell] +---------------------------------------------------------------------- +./filebeat -M "*.*.prospector.close_eof=true" +---------------------------------------------------------------------- diff --git a/filebeat/docs/filebeat-options.asciidoc b/filebeat/docs/filebeat-options.asciidoc index 7bfdcfd8bd20..daf8fef7dd43 100644 --- a/filebeat/docs/filebeat-options.asciidoc +++ b/filebeat/docs/filebeat-options.asciidoc @@ -1,6 +1,12 @@ [[configuration-filebeat-options]] == Set up prospectors +TIP: <> provide the fastest getting +started experience for common log formats. See <> +to learn how to get started with modules. Also see +<> for information about enabling and +configuring modules. + Filebeat uses prospectors to locate and process files. To configure Filebeat, you specify a list of prospectors in the `filebeat.prospectors` section of the +{beatname_lc}.yml+ config file. diff --git a/filebeat/docs/getting-started.asciidoc b/filebeat/docs/getting-started.asciidoc index e09a06a545fc..9406d7241449 100644 --- a/filebeat/docs/getting-started.asciidoc +++ b/filebeat/docs/getting-started.asciidoc @@ -264,7 +264,7 @@ sudo chown root filebeat.yml <1> sudo ./filebeat -e -c filebeat.yml -d "publish" ---------------------------------------------------------------------- <1> You'll be running Filebeat as root, so you need to change ownership -of the configuration file, or run Filebeat with `-strict.perms=false` +of the configuration file, or run Filebeat with `--strict.perms=false` specified. See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] in the _Beats Platform Reference_. diff --git a/filebeat/docs/modules-getting-started.asciidoc b/filebeat/docs/modules-getting-started.asciidoc index 6191105c6df7..cb387a57c1d3 100644 --- a/filebeat/docs/modules-getting-started.asciidoc +++ b/filebeat/docs/modules-getting-started.asciidoc @@ -40,6 +40,9 @@ sudo bin/elasticsearch-plugin install ingest-user-agent ---------------------------------------------------------------------- + You need to restart Elasticsearch after running these commands. ++ +If you are using an https://cloud.elastic.co/[Elastic Cloud] instance, you can +enable the two plugins from the configuration page. * Verify that Elasticsearch and Kibana are running and that Elasticsearch is ready to receive data from Filebeat. @@ -58,17 +61,17 @@ the sample dashboards for visualizing the data in Kibana. For example: ./filebeat setup -e ---------------------------------------------------------------------- + -The value that you pass with the `-modules` flag is a comma-separated list of +The value that you pass with the `--modules` flag is a comma-separated list of modules that you want to set up. The `-e` flag is optional and sends output to standard error instead of syslog. -. Start Filebeat and use the `-modules` flag to specify the list of modules +. Start Filebeat and use the `--modules` flag to specify the list of modules you want to run. The following example starts Filebeat with the `system` module enabled (it's assumed that you've already loaded the sample dashboards): + [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=system +./filebeat -e --modules system ---------------------------------------------------------------------- + This command takes care of configuring Filebeat and loading the ingest node @@ -79,9 +82,19 @@ example: + [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=system,nginx,mysql +./filebeat -e --modules system,nginx,mysql ---------------------------------------------------------------------- - ++ +When you start Filebeat, you should see messages indicating that Filebeat +has started harvesters for all enabled modules. For example: ++ +[source,shell] +---------------------------------------------------------------------- +2017/08/16 23:39:15.414375 harvester.go:206: INFO Harvester started for file: /var/log/displaypolicyd.stdout.log +---------------------------------------------------------------------- ++ +If you don't see this message for each log file that needs to be read, +see <> to find out how to set the path the files. NOTE: Depending on how you've installed Filebeat, you might see errors related to file ownership or permissions when you try to run Filebeat modules. @@ -89,12 +102,10 @@ See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions in the _Beats Platform Reference_ if you encounter errors related to file ownership or permissions. -//include::system-module-note.asciidoc[] - - -TIP: In a production environment, you'll probably want to use a configuration -file, rather than command-line flags, to specify which modules to run. See the -detailed documentation for more about configuring and running modules. +This getting started guide uses the `--modules` flag to enable modules +at the command line when you run Filebeat. In a production environment, you'll +probably want to use the configs in the `modules.d` directory instead. See +<> for more information. [[setting-variables]] ==== Set the path variable @@ -108,11 +119,11 @@ logs: [source,shell] ---- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---- -See the <> for more information about setting variables and -advanced options. +See <> for more information about setting +variables and advanced options. [[passing-credentials-modules]] ==== Pass credentials diff --git a/filebeat/docs/modules-overview.asciidoc b/filebeat/docs/modules-overview.asciidoc index 0f9c363e58fb..f182f48af2cf 100644 --- a/filebeat/docs/modules-overview.asciidoc +++ b/filebeat/docs/modules-overview.asciidoc @@ -34,9 +34,20 @@ Node. Filebeat modules require Elasticsearch 5.2 or later. +[float] +=== Get started + +To learn how to configure and run Filebeat modules: + +* Get started by reading <>. +* Try out the extended <> for a more in-depth introduction. +* Learn about the different ways to enable modules in <>. + [[modules-tutorial]] === Tutorial +//TODO: Replace this content with a more in-depth tutorial or remove it. + This tutorial assumes you have Elasticsearch and Kibana installed and accessible from Filebeat (see the <> section). It also assumes that the Ingest Node GeoIP and User Agent plugins are @@ -63,13 +74,13 @@ You can start Filebeat with the following command: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -setup +./filebeat -e --modules nginx -setup ---------------------------------------------------------------------- The `-e` flag tells Filebeat to output its logs to standard error, instead of syslog. -The `-modules=nginx` flag loads the Nginx module. +The `--modules nginx` flag loads the Nginx module. The `-setup` flag tells Filebeat to load the associated sample Kibana dashboards. This setup phase, in which the dashboards are loaded, doesn't have @@ -86,10 +97,9 @@ You can also start multiple modules at once: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx,mysql,system +./filebeat -e --modules nginx,mysql,system ---------------------------------------------------------------------- -include::system-module-note.asciidoc[] While enabling the modules from the CLI file is handy for getting started and for testing, you will probably want to use the configuration file for the @@ -123,7 +133,7 @@ files are in a custom location: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" +./filebeat -e --modules nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]" ---------------------------------------------------------------------- Or via the configuration file: @@ -136,17 +146,6 @@ filebeat.modules: var.paths: ["/var/log/nginx/access.log*"] ---------------------------------------------------------------------- -The Nginx `access` fileset also has a `pipeline` variable which allows -selecting which of the available Ingest Node pipelines is used for parsing. At -the moment, two such pipelines are available, one that requires the two ingest -plugins (`ingest-geoip` and `ingest-user-agent`) and one that doesn't. If you -cannot install the plugins, you can use the following: - - -[source,shell] ----------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.var.pipeline=no_plugins" ----------------------------------------------------------------------- ==== Advanced settings @@ -169,7 +168,7 @@ Or like this: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true" +./filebeat -e --modules nginx -M "nginx.access.prospector.close_eof=true" ---------------------------------------------------------------------- From the CLI, it's possible to change variables or settings for multiple @@ -178,7 +177,7 @@ modules/fileset at once. For example, the following works and will enable [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true" +./filebeat -e --modules nginx -M "nginx.*.prospector.close_eof=true" ---------------------------------------------------------------------- The following also works and will enable `close_eof` for all prospectors @@ -186,5 +185,5 @@ created by any of the modules: [source,shell] ---------------------------------------------------------------------- -./filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true" +./filebeat -e --modules nginx,mysql -M "*.*.prospector.close_eof=true" ---------------------------------------------------------------------- diff --git a/libbeat/docs/breaking.asciidoc b/libbeat/docs/breaking.asciidoc index 6264ba2578db..b1727397cec9 100644 --- a/libbeat/docs/breaking.asciidoc +++ b/libbeat/docs/breaking.asciidoc @@ -100,6 +100,16 @@ with a CLI flag: filebeat --modules=system ---- +==== Other settings changed or moved + +The `outputs.elasticsearch.template.*` settings have been moved under +`setup.template.*`, but are otherwise unchanged. + +The `dashboards.*` settings have been moved under `setup.dashboards.*`. + +The Filebeat deprecated options `force_close_files` and `close_older` are +removed. + [[breaking-changes-import-dashboards]] ==== Changes for importing the Kibana dashboards diff --git a/libbeat/docs/command-reference.asciidoc b/libbeat/docs/command-reference.asciidoc index 3fe3759c1e6e..a13c612ef428 100644 --- a/libbeat/docs/command-reference.asciidoc +++ b/libbeat/docs/command-reference.asciidoc @@ -159,9 +159,11 @@ ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat")] ==== `modules` command {modules-command-short-desc}. You can use this command to enable and disable -specific modules. The changes you make with this command are persisted and -used for subsequent runs of {beatname_uc}. To see which modules are enabled -and disabled, run the `list` subcommand. +specific module configurations defined in the `modules.d` directory. The +changes you make with this command are persisted and used for subsequent +runs of {beatname_uc}. + +To see which modules are enabled and disabled, run the `list` subcommand. *SYNOPSIS* diff --git a/libbeat/docs/upgrading.asciidoc b/libbeat/docs/upgrading.asciidoc index 57c5aeae2674..76811c4b0e9f 100644 --- a/libbeat/docs/upgrading.asciidoc +++ b/libbeat/docs/upgrading.asciidoc @@ -4,214 +4,148 @@ This section gives general recommendations for upgrading the Beats: * <> -* <> +* <> * <> [[upgrading-minor-versions]] === Upgrade between minor versions -As a general rule, upgrading between minor versions (e.g. 5.x to 5.y, where x < +As a general rule, upgrading between minor versions (e.g. 6.x to 6.y, where x < y) can be done by simply installing the new release and restarting the Beat process, because the Beats keep backwards compatibility when it comes to configuration and exported fields. Please review the <> for potential exceptions. -While we keep the backwards compatibility when it comes to the exported fields, -a new minor version can add new fields (e.g. a new Metricbeat module is added). -This can be problematic if the Elasticsearch mapping settings are not updated as -well. +Upgrading between non-consecutive major versions (e.g. 1.x to 6.x) is not +supported. If you are on version 1.x of the Beats, we recommend first upgrading +to 5.6. -Please see the <> section for possible strategies in -upgrading the Elasticsearch mapping templates. - -[[upgrading-1-to-5]] -=== Upgrade from 1.x to 5.x +[[upgrading-5-to-6]] +=== Upgrade from 5.x to 6.x Before upgrading your Beats, please review the <> doc. -In addition to the subsections here, please read <>. - -==== Migrate configuration files +// TODO: link to the stack upgrade docs +// +// If you are planning an upgrade of the full stack (Elasticsearch, Kibama, or +// Logstash, in addition to Beats), please read the stack upgrade guide. -Beats 5.0 comes with several backwards incompatible configuration changes. -However, we provide a script that can automatically fixup the most common -settings in your configuration files. +We recommend that you fully upgrade Elasticsearch and Kibana to version 6.0 +before upgrading Beats. If you are on a Beats 5.x version lower than 5.6, please +follow the <> section *before* doing the Elasticsearch +upgrade. -You can find this script in the Beats download packages. For the `.tar.gz` and -`.zip` archives, look under: +If you're not ready to upgrade Elasticsearch and Kibana to 6.0, that's alright. +Beats version 6.0 works with Elasticsearch and Kibana version 5.6, so you can +upgrade Beats now and the rest of the stack later. -[source,shell] ------------------------------------------------------------------------------- -./scripts/migrate_beat_config_1_x_to_5_0.py ------------------------------------------------------------------------------- +[[upgrading-to-5.6]] +==== Upgrade to 5.6 -For the `.deb` and `.rpm` packages, look under the -`/usr/share//scripts/` folder, for example: +The upgrade procedure assumes that you have Beats version 5.6 installed. If you +are on a previous 5.x version of Beats, please upgrade to version 5.6 first. +The reason is that the Elasticsearch mapping template in 5.6 was modified to be +compatible with Elasticsearch 6.0 (by removing the `_all` settings). +For this reason, after upgrading to 5.6, you need to make sure that the 5.6 +template is loaded. You can do this by temporarily enabling the +`output.elasticsearch.template.overwrite` setting, for example with Metricbeat: [source,shell] ------------------------------------------------------------------------------ -/usr/share/packetbeat/scripts/migrate_beat_config_1_x_to_5_0.py +metricbeat -e -E output.elasticsearch.template.overwrite=true ------------------------------------------------------------------------------ -Python 2.7 or later is required for executing the migration script. - -To upgrade your configuration file automatically, simply pass it as an argument -to the migration script, for example: +Alternatively, you can manually force loading the template: [source,shell] ------------------------------------------------------------------------------ -> ./scripts/migrate_beat_config_1_x_to_5_0.py packetbeat.yml -Backup file created: packetbeat.yml.bak +curl -XPUT -H'Content-Type: application/json' http://localhost:9200/_template/metricbeat -d @metricbeat.template.json ------------------------------------------------------------------------------ -The script will automatically create a backup file and overwrite the input file -with the upgraded version. You can also use the `--dry` option, which doesn't -modify the input file, but outputs the upgraded version to stdout instead. - -==== Filebeat registry file location and format +To check which version of the template is loaded, open Kibana Console, call `GET +/_template/metricbeat`, and look for the version string. Note that you need to +do this for each Beat type that you are running (e.g. Filebeat, Metricbeat, +Packetbeat). -Filebeat stores the read offset in a registry file that it persists to disk. -The format of this file has changed in version 5.0, but Filebeat automatically -upgrades the format on the first run, so there's normally no action required on -upgrading. +==== Migrate configuration files -If you have installed Filebeat from the DEB or RPM packages, the location of the -file stays the same as well (`/var/lib/filebeat/registry`). Also if you've -installed Filebeat as a service on Windows, the location stays the same -(`C:\ProgramData\filebeat\registry`). +Beats 6.0 comes with several backwards incompatible configuration changes. +Please review the <> document. Where possible, we kept the +old configuration options working, but deprecated them. However, deprecation +was not always possible, so if you use any of the settings mentioned in the +Breaking Changes section of the release notes, make sure you understand the +alternatives that we provide. -If you are running Filebeat from the `tar.gz` archive, however, the location of -the registry file has changed. In version 1.x it was by default named -`.filebeat` in the current working directory. In version 5.0, it is -`data/registry` relative to the binary location. For upgrading in this case, we -recommend copying the `.filebeat` file to `data/registry`. +===== modules.d configuration layout -==== Upgrade Topbeat to Metricbeat +Starting with the 6.0 version, Filebeat and Metricbeat are moving to a directory +layout for configuration, where each module is configured in its own +configuration file. -NOTE: When upgrading to Metricbeat, keep in mind that any data you've collected -with Topbeat is not compatible with the 5.0 version of the Beats dashboards -because the underlying event data structures have changed. +While the all-in-one configuration is still fully supported, we recommend moving +to the new layout at upgrade time. This typically means starting off with the new +default configuration and modifying it with the custom settings that you had in +your old files. -With the Beats 5.0 release, Topbeat is replaced by Metricbeat, which offers more -functionality. More precisely, the Topbeat functionality is roughly equivalent -to the "system" module in Metricbeat. +// TODO: link to the docs on modules.d -While there is no automatic way to convert a Topbeat configuration file to the -equivalent of a Metricbeat configuration file, both configurations are simple -enough that migrating shouldn't take long. +===== New command `test config` command -For example, this is the input section of the Topbeat configuration file: +Beats 6.0 introduces a new test command for testing the configuration file. For +example: -[source,yaml] +[source,shell] ------------------------------------------------------------------------------ -input: - # In seconds, defines how often to read server statistics - period: 10 - - # Regular expression to match the processes that are monitored - # By default, all the processes are monitored - procs: [".*"] - - # Statistics to collect (all enabled by default) - stats: - # per system statistics, by default is true - system: true - - # per process statistics, by default is true - process: true - - # file system information, by default is true - filesystem: true - - # cpu usage per core, by default is false - cpu_per_core: false +metricbeat test config ------------------------------------------------------------------------------ -And here is the equivalent configuration for the system module in Metricbeat: +The old `-configtest` flag is still available, but deprecated. -[source,yaml] ------------------------------------------------------------------------------- -metricbeat.modules: +===== Reference configuration files -#------------------------------- System Module ------------------------------- -- module: system - metricsets: - # CPU stats - - cpu +The `.full.yml` file, which contains all the non-deprecated +configuration options is renamed to `.reference.yml` starting with +Beats 6.0. We recommend using this file as a reference only. It's not intended +to be used in production. - # System Load stats - - load +==== Dashboard upgrades - # Per filesystem stats - - filesystem +We recommend re-importing the Kibana dashboards after the Beats and Kibana +upgrades are complete. This way, you will take advantage of the new Beats +dashboards created for the 6.0 release. - # Memory stats - - memory +See <> in the Breaking Changes +documentation. - # Network stats - - network +NOTE: If you've modified the Beats dashboards manually, reimporting them will +overwrite your changes. Consider saving your dashboards under new IDs, so +that the reimport doesn't overwrite them. - # Per process stats - - process - enabled: true - period: 10s - processes: ['.*'] ------------------------------------------------------------------------------- +==== Heartbeat package names -The remaining configuration options (outputs, name, tags) stay the same or can -be upgraded using the migration script. +The DEB and RPM packages for Heartbeat are now named `heartbeat-elastic` to +avoid conflicts with a different `heartbeat` project. The `.deb` and `.rpm` +file names haven't changed, just the package name in the repositories. [[upgrade-mapping-template]] === Upgrade the Elasticsearch mapping template -By default, the Elasticsearch mapping template that comes with the Beats is not -re-installed if it already exists. This means that if the new Beats version -adds new fields (e.g. a new module in Metricbeat), they won't be reflected in -the Elasticsearch mapping, which can result in visualization problems or -incorrect data. - -You can set the `output.elasticsearch.template.overwrite` option to `true` to -make the Beats install the new mapping template even if a template with the same -name already exists. You can set this option in the configuration file or as a -CLI flag, for example: +Starting with Beats 6.0, the mapping templates and the default index names are +versioned. For example, Metricbeat 6.0.0 creates indices like this: [source,shell] ------------------------------------------------------------------------------ -packetbeat -e -E output.elasticsearch.template.overwrite=true ------------------------------------------------------------------------------- - -However, the mapping template is applied on index creation, so it won't affect -the current index if it was already created with the old version. - -You can force the start of a new index by renaming the old index before starting -the new version of the Beat. As of Elasticsearch 5.0, this can be done via the -re-index API: - - -[source,json] ------------------------------------------------------------------------------- -POST /_reindex -{ - "source": { - "index": "packetbeat-2016.09.20" - }, - "dest": { - "index": "packetbeat-2016.09.20-old" - } -} -DELETE /packetbeat-2016.09.20 +metricbeat-6.0.0-2017-08-31 ------------------------------------------------------------------------------ -Note that the reindex API command can take a long time, depending on the size of -the index. It is recommended that you stop the Beat during this time, so the -order of operations should be: +And the corresponding Elasticsearch template is named `metricbeat-6.0.0`. -1. Stop the old version of the Beat -2. Rename the index of the current day -3. Start the new version of the Beat +This means that each version of the Beat creates a new index, and it is +guaranteed that the correct template for that version is applied. With these +changes in place, you generally don't have to do anything to upgrade the mapping +template when you move to a new version. -If downtime is not acceptable, another possible approach is to configure a -different index pattern in the new Beat version, but this will likely require -adjustments to your Kibana dashboards. +One exception is when upgrading from 5.x, in which case you should make sure to +read the <> section.