Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport to 6.0: #4760 and #4829 #4897

Merged
merged 2 commits into from
Aug 18, 2017
Merged

Backport to 6.0: #4760 and #4829 #4897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
057c1cc
Added breaking changes doc for 6.0 (#4760)
tsg Aug 10, 2017
acd43e7
add cloudwatchlogsbeat to community beats (#4829)
kkentzo Aug 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 133 additions & 81 deletions libbeat/docs/breaking.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,112 +2,164 @@
== Breaking changes

As a general rule, we strive to keep backwards compatibility between minor
versions (e.g. 5.x to 5.y) so you can upgrade without any configuration file
versions (e.g. 6.x to 6.y) so you can upgrade without any configuration file
changes, but there are breaking changes between major versions (e.g. 5.x to
6.y).
6.y). Migrating directly between non consecutive major versions (e.g. 1.x to
6.x) is not recommended.

* <<breaking-changes-5.0>>
[[breaking-changes-6.0]]
=== Breaking changes in 6.0

We also list the breaking changes at the top of the <<release-notes,release notes>>
for each version.
This section discusses the main changes that you should be aware of if you
upgrade the Beats from version 5.x to 6.x. Please also review the relevant
Breaking Changes sections of the <<release-notes,release notes>>.

[[breaking-changes-5.0]]
=== Breaking changes in 5.0
// TODO: better link to the consolidated release notes for 6.0.0.

This section discusses the changes that you should be aware of if you upgrade
the Beats from version 1.x to 5.x.

==== URL changes for DEB/RPM packages
[[breaking-changes-spooler-removed]]
==== Filebeat spooler removed

The previous `packages.elastic.co` URL has been altered to `artifacts.elastic.co`.
Ensure you update your repository files before running the upgrade process, or
your operating system may not see the new packages.
Version 6.0 comes with a new architecture for the internal pipeline of all
Beats. This architecture refactoring is mostly internal, but one of the more
visible effects is that the Spooler component of Filebeat is removed. The
functionality of the Spooler was similar to the one of the publisher queue from
libbeat (the code shared by all Beats), and the presence of multiple queues
made the performance tuning of Filebeat more complex than it needed to be.

==== Configuration file changes
As a result, the following options are removed:

Beats 5.0 comes with a more powerful and more robust <<config-file-format,file
configuration format>>. While this necessarily causes a few breaking changes, we
provide a migration script that should make upgrading easy. Please see the
<<upgrading>> section.
- `filebeat.spool_size`
- `filebeat.publish_async`
- `filebeat.idle_timeout`
- `queue_size`
- `bulk_queue_size`

For consistency and clarity, all configuration options that are specific to a
particular Beat are now prefixed by the name of the Beat. For example, the
`interfaces.device` option from Packetbeat becomes
`packetbeat.interfaces.device`. This change is made automatically by the
migration script.
The first three are specific to Filebeat, while `queue_size` and
`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat
6.0 will refuse to start.

Similarly, common settings available in all Beats are now top level. As part of
this effort, the options under the `shipper` section are moved to the top level. For
example, the `shipper.name` settings becomes `name`. This change is made
automatically by the migration script.
Instead of the settings above, the `queue.mem` settings are introduced. If you
had to tune the `spool_size` or the `queue_size` before, you might want to tune
the `queue.mem.events` when upgrading. However, it is best to leave the rest of
the `queue.mem` settings to their default values, as they are appropriate for
all loads.

For consistency with the rest of the Elastic stack, several TLS/SSL related
options where changed:
The `publish_async` option (which was deprecated since 5.3) is removed because
the new pipeline already works asynchronously by default.

* the name of the section was renamed from `tls` to `ssl`
* `certificate_key` was renamed to `key`
* `insecure` was replaced by `verification_mode`
* `min_version` and `max_version` were replaced with `supported_protocols`
// TODO: for the above new settings, link to their configuration settings.

These changes are made automatically by the migration script.
[[breaking-changes-single-output]]
==== Only one enabled output

In Filebeat, we refactored the configuration around when to close and when to
start ignoring files. Options like `close_older` and `force_close_older` were
changed or replaced with options from the `close_*` group. See the
{filebeat}/how-filebeat-works.html[How Filebeat Works] docs for details. These
changes are not made automatically by the migration script.
In versions prior to 6.0, you could enabled multiple outputs at the same time,
but only of different types. For example, you were able to enable the
Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback
of enabling multiple outputs was that the Beats that wait for acknowledgments
(Filebeat and Winlogbeat) before proceeding slowed down to the slowest output.
This implication was not obvious and hindered the use cases where multiple
outputs would have been useful.

==== Field renames
As part of the pipeline re-architecture that we did for 6.0, we removed the
option to enable multiple outputs at the same time. This helps with keeping the
pipeline simple and with clarifying the scope of outputs in Beats.

Beats 5.0 also comes with several changes in the exported fields. If you are
using these fields in Kibana dashboards, you will likely need to adjust the
dashbaords.
If you require multiple outputs, you have the following options:

Changes:
* use the Logstash output and then use Logstash to pipe the events to multiple
outputs
* run multiple instances of the same Beat

[options="header",]
|==========================================================
| Location | Old name | New Name
| Packetbeat HTTP module | http.phrase | http.response.phrase
| Packetbeat HTTP module | http.code | http.response.code
| Packetbeat HTTP module | http.response_headers | http.response.headers
| Packetbeat HTTP module | http.request_headers | http.request.headers
| Packetbeat HTTP module | http.params | http.request.params
| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
|==========================================================
If you used the `file` or `console` outputs for debugging purposes, in addition
to the main output, we recommend using the `-d "publish"` option which logs the
published events in the Filebeat logs.

In addition, the unused `count` field was removed from all Packetbeat modules.
[[breaking-changes-types]]
==== Filebeat prospector type and document type changes

==== Installation paths
The `document_type` setting, from the prospector configuration, was removed
because the `_type` concept is being removed from Elasticsearch. Instead of the
`document_type` setting, you can use a custom field.

For 5.0 we have refactored the {filebeat}/directory-layout.html[directory
layout] of the Beats, so they are more consistent and in-line with the best
practices of each operating system.
This has led also to the rename of the `input_type` configuration setting to
`type`. This change is backwards compatible because the old setting still
works. However, the `input_type` output field was renamed to `prospector_type`.

A notable change is that on the DEB/RPM installations, the binary is no longer
installed in `/usr/bin` but in `/usr/share/<beatname>/bin`. Instead, a simple
shell script is placed under `/usr/bin` that is helpful for executing the Beat
in foreground. It also sets the path settings to the same values as the init
script would use.
[[breaking-changes-default-config]]
==== Filebeat default prospector disabled in the configuration file

For example, here are the contents of the `/usr/bin/filebeat.sh` script:
The default startup behaviour (based on the included sample configuration) of
Filebeat was to read all the files matching the `/var/log/*.log` pattern.
Starting with version 6.0, Filebeat doesn't read any files in its default
configuration. However, you can easily enable the `system` module, for example
with a CLI flag:

["source","sh",subs="attributes"]
----
filebeat --modules=system
----

[source,shell]
------------------------------------------------------------------------------
> cat /usr/bin/filebeat.sh
#!/bin/bash
[[breaking-changes-import-dashboards]]
==== Changes for importing the Kibana dashboards

# Script to run filebeat in foreground with the same path settings that
# the init script / systemd unit file would do.
The `import_dashboards` program, used to load the Kibana dashboards in previous
versions of Beats, is replaced by the `setup` command. For example, the
following command:

["source","sh",subs="attributes"]
----
./scripts/import_dashboards -user elastic -pass changeme
----

Can be replaced with:

["source","sh",subs="attributes"]
----
./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password=changeme"
----

Note that the `-E` flags are only required if the Elasticsearch output is not
already configured in the configuration file.

Besides the change in the commands, it's important to note that loading the
Kibana dashboards works differently in the 6.0 version of the stack. Prior to
6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch
index. Starting with 6.0, the Beats use a Kibana server API. This means that
the Beat that loads the dashboards needs direct access to Kibana and that the
Kibana URL needs to be set. The option to set the Kibana URL is
`setup.kibana.host`, which you can set in the configuration file or via the
`-E` CLI flag:


["source","sh",subs="attributes"]
----
./filebeat setup -E "setup.kibana.host=http://kibana-host:5601"
----

The default value for the Kibana host is `localhost:5601`.

[[breaking-changes-filters]]
==== Metricbeat filters renamed to processors

The "local" processors, which are configured at the module level, used to be
called `filters` in Metricbeat, but were offering similar functionality with
the global `processors`. A notable difference between the two was that the
filters accessed fields relatively to the metricset (for example,
`mount_point`), while the processors referred to fields by their fully
qualified name (for example, `system.filesystem.mount_point`).

Starting with version 6.0, the `filters` are renamed to `processors` and they
can access the fields only by using the fully qualified names.

[[breaking-changes-cgo]]
==== Binaries are dynamically compiled against libc

Prior to 6.0, Metricbeat and Packetbeat were compiled using
https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat
were compiled using the pure Go compiler. One of the side-effects of compiling
with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats
are compiled using Cgo and therefore dynamically compiled against libc. This
can reduce the portability of the binaries, but none of the supported platforms
is affected.

/usr/share/filebeat/bin/filebeat \
-path.home /usr/share/filebeat \
-path.config /etc/filebeat \
-path.data /var/lib/filebeat \
-path.logs /var/log/filebeat \
$@
------------------------------------------------------------------------------
1 change: 1 addition & 0 deletions libbeat/docs/communitybeats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ https://github.com/goomzee/burrowbeat[burrowbeat]:: Monitors Kafka consumer lag
https://github.com/goomzee/cassandrabeat[cassandrabeat]:: Uses Cassandra's nodetool cfstats utility to monitor Cassandra database nodes and lag.
https://github.com/hartfordfive/cloudflarebeat[cloudflarebeat]:: Indexes log entries from the Cloudflare Enterprise Log Share API.
https://github.com/aidan-/cloudtrailbeat[cloudtrailbeat]:: Reads events from Amazon Web Services' https://aws.amazon.com/cloudtrail/[CloudTrail].
https://github.com/e-travel/cloudwatchlogsbeat[cloudwatchlogsbeat]:: Reads log events from Amazon Web Services' https://aws.amazon.com/cloudwatch/details/#log-monitoring[CloudWatch Logs].
https://github.com/raboof/connbeat[connbeat]:: Exposes metadata about TCP connections.
https://github.com/Pravoru/consulbeat[consulbeat]:: Reads services health checks from consul and pushes them to Elastic.
https://github.com/Ingensi/dockbeat[dockbeat]:: Reads Docker container
Expand Down