From 057c1cc55795dcef8060caf4958e524945b4268a Mon Sep 17 00:00:00 2001 From: Tudor Golubenco Date: Thu, 10 Aug 2017 23:06:47 +0300 Subject: [PATCH 1/2] Added breaking changes doc for 6.0 (#4760) * Added breaking changes doc for 6.0 Part of #4737, but there are some TODOs left. * Add section for multiple outputs Also, addressed commits and removed the 5.0 breaking changes. --- libbeat/docs/breaking.asciidoc | 214 ++++++++++++++++++++------------- 1 file changed, 133 insertions(+), 81 deletions(-) diff --git a/libbeat/docs/breaking.asciidoc b/libbeat/docs/breaking.asciidoc index 781566274710..6264ba2578db 100644 --- a/libbeat/docs/breaking.asciidoc +++ b/libbeat/docs/breaking.asciidoc @@ -2,112 +2,164 @@ == Breaking changes As a general rule, we strive to keep backwards compatibility between minor -versions (e.g. 5.x to 5.y) so you can upgrade without any configuration file +versions (e.g. 6.x to 6.y) so you can upgrade without any configuration file changes, but there are breaking changes between major versions (e.g. 5.x to -6.y). +6.y). Migrating directly between non consecutive major versions (e.g. 1.x to +6.x) is not recommended. -* <> +[[breaking-changes-6.0]] +=== Breaking changes in 6.0 -We also list the breaking changes at the top of the <> -for each version. +This section discusses the main changes that you should be aware of if you +upgrade the Beats from version 5.x to 6.x. Please also review the relevant +Breaking Changes sections of the <>. -[[breaking-changes-5.0]] -=== Breaking changes in 5.0 +// TODO: better link to the consolidated release notes for 6.0.0. -This section discusses the changes that you should be aware of if you upgrade -the Beats from version 1.x to 5.x. -==== URL changes for DEB/RPM packages +[[breaking-changes-spooler-removed]] +==== Filebeat spooler removed -The previous `packages.elastic.co` URL has been altered to `artifacts.elastic.co`. -Ensure you update your repository files before running the upgrade process, or -your operating system may not see the new packages. +Version 6.0 comes with a new architecture for the internal pipeline of all +Beats. This architecture refactoring is mostly internal, but one of the more +visible effects is that the Spooler component of Filebeat is removed. The +functionality of the Spooler was similar to the one of the publisher queue from +libbeat (the code shared by all Beats), and the presence of multiple queues +made the performance tuning of Filebeat more complex than it needed to be. -==== Configuration file changes +As a result, the following options are removed: -Beats 5.0 comes with a more powerful and more robust <>. While this necessarily causes a few breaking changes, we -provide a migration script that should make upgrading easy. Please see the -<> section. +- `filebeat.spool_size` +- `filebeat.publish_async` +- `filebeat.idle_timeout` +- `queue_size` +- `bulk_queue_size` -For consistency and clarity, all configuration options that are specific to a -particular Beat are now prefixed by the name of the Beat. For example, the -`interfaces.device` option from Packetbeat becomes -`packetbeat.interfaces.device`. This change is made automatically by the -migration script. +The first three are specific to Filebeat, while `queue_size` and +`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat +6.0 will refuse to start. -Similarly, common settings available in all Beats are now top level. As part of -this effort, the options under the `shipper` section are moved to the top level. For -example, the `shipper.name` settings becomes `name`. This change is made -automatically by the migration script. +Instead of the settings above, the `queue.mem` settings are introduced. If you +had to tune the `spool_size` or the `queue_size` before, you might want to tune +the `queue.mem.events` when upgrading. However, it is best to leave the rest of +the `queue.mem` settings to their default values, as they are appropriate for +all loads. -For consistency with the rest of the Elastic stack, several TLS/SSL related -options where changed: +The `publish_async` option (which was deprecated since 5.3) is removed because +the new pipeline already works asynchronously by default. -* the name of the section was renamed from `tls` to `ssl` -* `certificate_key` was renamed to `key` -* `insecure` was replaced by `verification_mode` -* `min_version` and `max_version` were replaced with `supported_protocols` +// TODO: for the above new settings, link to their configuration settings. -These changes are made automatically by the migration script. +[[breaking-changes-single-output]] +==== Only one enabled output -In Filebeat, we refactored the configuration around when to close and when to -start ignoring files. Options like `close_older` and `force_close_older` were -changed or replaced with options from the `close_*` group. See the -{filebeat}/how-filebeat-works.html[How Filebeat Works] docs for details. These -changes are not made automatically by the migration script. +In versions prior to 6.0, you could enabled multiple outputs at the same time, +but only of different types. For example, you were able to enable the +Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback +of enabling multiple outputs was that the Beats that wait for acknowledgments +(Filebeat and Winlogbeat) before proceeding slowed down to the slowest output. +This implication was not obvious and hindered the use cases where multiple +outputs would have been useful. -==== Field renames +As part of the pipeline re-architecture that we did for 6.0, we removed the +option to enable multiple outputs at the same time. This helps with keeping the +pipeline simple and with clarifying the scope of outputs in Beats. -Beats 5.0 also comes with several changes in the exported fields. If you are -using these fields in Kibana dashboards, you will likely need to adjust the -dashbaords. +If you require multiple outputs, you have the following options: -Changes: +* use the Logstash output and then use Logstash to pipe the events to multiple + outputs +* run multiple instances of the same Beat -[options="header",] -|========================================================== -| Location | Old name | New Name -| Packetbeat HTTP module | http.phrase | http.response.phrase -| Packetbeat HTTP module | http.code | http.response.code -| Packetbeat HTTP module | http.response_headers | http.response.headers -| Packetbeat HTTP module | http.request_headers | http.request.headers -| Packetbeat HTTP module | http.params | http.request.params -| Packetbeat HTTP module | http.content_length | http.response.headers.content-length -| Packetbeat HTTP module | http.content_length | http.response.headers.content-length -| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available -| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available -|========================================================== +If you used the `file` or `console` outputs for debugging purposes, in addition +to the main output, we recommend using the `-d "publish"` option which logs the +published events in the Filebeat logs. -In addition, the unused `count` field was removed from all Packetbeat modules. +[[breaking-changes-types]] +==== Filebeat prospector type and document type changes -==== Installation paths +The `document_type` setting, from the prospector configuration, was removed +because the `_type` concept is being removed from Elasticsearch. Instead of the +`document_type` setting, you can use a custom field. -For 5.0 we have refactored the {filebeat}/directory-layout.html[directory -layout] of the Beats, so they are more consistent and in-line with the best -practices of each operating system. +This has led also to the rename of the `input_type` configuration setting to +`type`. This change is backwards compatible because the old setting still +works. However, the `input_type` output field was renamed to `prospector_type`. -A notable change is that on the DEB/RPM installations, the binary is no longer -installed in `/usr/bin` but in `/usr/share//bin`. Instead, a simple -shell script is placed under `/usr/bin` that is helpful for executing the Beat -in foreground. It also sets the path settings to the same values as the init -script would use. +[[breaking-changes-default-config]] +==== Filebeat default prospector disabled in the configuration file -For example, here are the contents of the `/usr/bin/filebeat.sh` script: +The default startup behaviour (based on the included sample configuration) of +Filebeat was to read all the files matching the `/var/log/*.log` pattern. +Starting with version 6.0, Filebeat doesn't read any files in its default +configuration. However, you can easily enable the `system` module, for example +with a CLI flag: +["source","sh",subs="attributes"] +---- +filebeat --modules=system +---- -[source,shell] ------------------------------------------------------------------------------- -> cat /usr/bin/filebeat.sh -#!/bin/bash +[[breaking-changes-import-dashboards]] +==== Changes for importing the Kibana dashboards -# Script to run filebeat in foreground with the same path settings that -# the init script / systemd unit file would do. +The `import_dashboards` program, used to load the Kibana dashboards in previous +versions of Beats, is replaced by the `setup` command. For example, the +following command: + +["source","sh",subs="attributes"] +---- +./scripts/import_dashboards -user elastic -pass changeme +---- + +Can be replaced with: + +["source","sh",subs="attributes"] +---- +./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password=changeme" +---- + +Note that the `-E` flags are only required if the Elasticsearch output is not +already configured in the configuration file. + +Besides the change in the commands, it's important to note that loading the +Kibana dashboards works differently in the 6.0 version of the stack. Prior to +6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch +index. Starting with 6.0, the Beats use a Kibana server API. This means that +the Beat that loads the dashboards needs direct access to Kibana and that the +Kibana URL needs to be set. The option to set the Kibana URL is +`setup.kibana.host`, which you can set in the configuration file or via the +`-E` CLI flag: + + +["source","sh",subs="attributes"] +---- +./filebeat setup -E "setup.kibana.host=http://kibana-host:5601" +---- + +The default value for the Kibana host is `localhost:5601`. + +[[breaking-changes-filters]] +==== Metricbeat filters renamed to processors + +The "local" processors, which are configured at the module level, used to be +called `filters` in Metricbeat, but were offering similar functionality with +the global `processors`. A notable difference between the two was that the +filters accessed fields relatively to the metricset (for example, +`mount_point`), while the processors referred to fields by their fully +qualified name (for example, `system.filesystem.mount_point`). + +Starting with version 6.0, the `filters` are renamed to `processors` and they +can access the fields only by using the fully qualified names. + +[[breaking-changes-cgo]] +==== Binaries are dynamically compiled against libc + +Prior to 6.0, Metricbeat and Packetbeat were compiled using +https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat +were compiled using the pure Go compiler. One of the side-effects of compiling +with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats +are compiled using Cgo and therefore dynamically compiled against libc. This +can reduce the portability of the binaries, but none of the supported platforms +is affected. -/usr/share/filebeat/bin/filebeat \ - -path.home /usr/share/filebeat \ - -path.config /etc/filebeat \ - -path.data /var/lib/filebeat \ - -path.logs /var/log/filebeat \ - $@ ------------------------------------------------------------------------------- From acd43e7bccdec8ebe45eb7b48e18fe1cbe446d0b Mon Sep 17 00:00:00 2001 From: Kyriakos Kentzoglanakis Date: Tue, 8 Aug 2017 10:15:16 +0300 Subject: [PATCH 2/2] add cloudwatchlogsbeat to community beats (#4829) --- libbeat/docs/communitybeats.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/libbeat/docs/communitybeats.asciidoc b/libbeat/docs/communitybeats.asciidoc index 6c96aa53a3b1..e977a9c441d7 100644 --- a/libbeat/docs/communitybeats.asciidoc +++ b/libbeat/docs/communitybeats.asciidoc @@ -26,6 +26,7 @@ https://github.com/goomzee/burrowbeat[burrowbeat]:: Monitors Kafka consumer lag https://github.com/goomzee/cassandrabeat[cassandrabeat]:: Uses Cassandra's nodetool cfstats utility to monitor Cassandra database nodes and lag. https://github.com/hartfordfive/cloudflarebeat[cloudflarebeat]:: Indexes log entries from the Cloudflare Enterprise Log Share API. https://github.com/aidan-/cloudtrailbeat[cloudtrailbeat]:: Reads events from Amazon Web Services' https://aws.amazon.com/cloudtrail/[CloudTrail]. +https://github.com/e-travel/cloudwatchlogsbeat[cloudwatchlogsbeat]:: Reads log events from Amazon Web Services' https://aws.amazon.com/cloudwatch/details/#log-monitoring[CloudWatch Logs]. https://github.com/raboof/connbeat[connbeat]:: Exposes metadata about TCP connections. https://github.com/Pravoru/consulbeat[consulbeat]:: Reads services health checks from consul and pushes them to Elastic. https://github.com/Ingensi/dockbeat[dockbeat]:: Reads Docker container