Backport 2631: Breaking changes guide + fixes

libbeat/docs/breaking.asciidoc

@@ -0,0 +1,105 @@
+[[breaking-changes]]
+== Breaking changes
+
+As a general rule, we strive to keep backwards compatibility between minor
+versions (e.g.  5.x to 5.y) so you can upgrade without any configuration file
+changes, but there are breaking changes between major versions (e.g. 5.x to
+6.y).
+
+We list the breaking changes at the top of the <<release-notes,release notes>>
+for each version.
+
+[[breaking-changes-5.0]]
+=== Breaking changes in 5.0
+
+This section discusses the changes that you should be aware of if you upgrade
+the Beats from version 1.x to 5.x.
+
+==== Configuration file changes
+
+Beats 5.0 come with a more powerful and more robust <<config-file-format,file
+configuration format>>. While this necessarily causes a few breaking changes, we
+provide a migration script that should make upgrading easy. Please see the
+<<upgrading>> section.
+
+For consistency and clarity, all configuration options that are specific to a
+particular Beat are now prefixed by the name of the Beat. For example, the
+`interfaces.device` option from Packetbeat becomes
+`packetbeat.interfaces.device`. This change is made automatically by the
+migration script.
+
+Similarly, common settings available in all Beats are now top level. As part of
+this, the options under the `shipper` section are moved to the top level. For
+example, the `shipper.name` settings becomes `name`. This change is made
+automatically by the migration script.
+
+For consistency with the rest of the Elastic stack, several TLS/SSL related
+options where changed:
+
+* the name of the section was renamed from `tls` to `ssl`
+* `certificate_key` was renamed to `key`
+* `insecure` was replaced by `verification_mode`
+* `min_version` and `max_version` were replaced with `supported_protocols`
+
+These changes are made automatically by the migration script.
+
+In Filebeat, we refactored the configuration around when to close and when to
+start ignoring files. Options like `close_older` and `force_close_older` were
+changed or replaced with options from the `close_*` group. See the
+{filebeat}/how-filebeat-works.html[How Filebeat works] docs for details. These
+changes are not made automatically by the migration script.
+
+==== Field renames
+
+Beats 5.0 also come with several changes in the exported fields. If you are
+using these fields in Kibana dashboards, you will likely need to adjust the
+dashbaords.
+
+Changes:
+
+[options="header",]
+|==========================================================
+| Location | Old name | New Name
+| Packetbeat HTTP module | http.phrase | http.response.phrase
+| Packetbeat HTTP module | http.code | http.response.code
+| Packetbeat HTTP module | http.response_headers | http.response.headers
+| Packetbeat HTTP module | http.request_headers | http.request.headers
+| Packetbeat HTTP module | http.params | http.request.params
+| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
+| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
+| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
+| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
+|==========================================================
+
+In addition, the unused `count` field was removed from all Packetbeat modules.
+
+==== Installation paths
+
+For 5.0 we have refactored the {filebeat}/directory-layout.html[directory
+layout] of the Beats, so they are more consistent and in-line with the best
+practices of each operating system.
+
+A notable change is that on the DEB/RPM installations, the binary is no longer
+installed in `/usr/bin` but in `/usr/share/<beatname>/bin`. Instead, a simple
+shell script is placed under `/usr/bin` that is helpful for executing the Beat
+in foreground. It also sets the path settings to the same values as the init
+script would use.
+
+For example, here are the contents of the `/usr/bin/filebeat.sh` script:
+
+
+[source,shell]
+------------------------------------------------------------------------------
+> cat /usr/bin/filebeat.sh
+#!/bin/bash
+
+# Script to run filebeat in foreground with the same path settings that
+# the init script / systemd unit file would do.
+
+/usr/share/filebeat/bin/filebeat \
+  -path.home /usr/share/filebeat \
+  -path.config /etc/filebeat \
+  -path.data /var/lib/filebeat \
+  -path.logs /var/log/filebeat \
+  $@
+------------------------------------------------------------------------------

libbeat/docs/index.asciidoc

@@ -23,6 +23,10 @@ include::./installing-beats.asciidoc[]
 
 include::./repositories.asciidoc[]
 
+include::./breaking.asciidoc[]
+
+include::./upgrading.asciidoc[]
+
 include::./config-file-format.asciidoc[]
 
 // include::./visualizing-data.asciidoc[]

libbeat/docs/upgrading.asciidoc

@@ -0,0 +1,209 @@
+[[upgrading]]
+== Upgrading
+
+This section gives general recommendations for upgrading the Beats.
+
+=== Upgrading between minor versions
+
+As a general rule, upgrading between minor versions (e.g. 5.x to 5.y, where x <
+y) can be done by simply installing the new release and restarting the Beat
+process, because the Beats keep backwards compatibility when it comes to
+configuration and exported fields. Please review the <<release-notes,release
+notes>> for potential exceptions.
+
+While we keep the backwards compatibility when it comes to the exported fields,
+a new minor version can add new fields (e.g. a new Metricbeat module is added).
+This can be problematic if the Elasticsearch mapping settings are not updated as
+well.
+
+Please see the <<upgrade-mapping-template>> section for possible strategies in
+upgrading the Elasticsearch mapping templates.
+
+
+=== Upgrading from 1.x to 5.x
+
+Before upgrading your Beats, please review the <<breaking-changes, breaking
+changes>> doc.
+
+In addition to the subsections here, please read the
+<<upgrade-mapping-template>> 
+
+==== Configuration file migration
+
+Beats 5.0 come with several backwards incompatible configuration changes.
+However, we provide a script that can automatically fixup the most common
+settings in your configuration file.
+
+This script can be found in the Beats download packages. For the `.tar.gz` and
+`.zip` archives, it is found under:
+
+[source,shell]
+------------------------------------------------------------------------------
+./scripts/migrate_beat_config_1_x_to_5_0.py
+------------------------------------------------------------------------------
+
+For the `.deb` and `.rpm` packages it is found under the
+`/usr/share/<beatname>/scripts/` folder, for example:
+
+
+[source,shell]
+------------------------------------------------------------------------------
+/usr/share/packetbeat/scripts/migrate_beat_config_1_x_to_5_0.py
+------------------------------------------------------------------------------
+
+Python 2.7 or later is required for executing the migration script.
+
+To upgrade your configuration file automatically, simply pass it as an argument
+to the migration script, e.g.:
+
+[source,shell]
+------------------------------------------------------------------------------
+> ./scripts/migrate_beat_config_1_x_to_5_0.py packetbeat.yml
+Backup file created: packetbeat.yml.bak
+------------------------------------------------------------------------------
+
+The script will automatically create a backup file and overwrite it with the
+upgraded version. You can also use the `--dry` option, which doesn't modify the
+input file, but outputs the upgraded version to stdout instead.
+
+==== Filebeat registry file location and format
+
+Filebeat stores the read offset in a registry file that it persist to disk. The
+format of this file has changed in version 5.0, but Filebeat automatically
+upgrades the format on first run, so there's normally no action required on
+upgrading.
+
+If you have installed Filebeat from the DEB or RPM packages, the location of the
+file stays the same as well (`/var/lib/filebeat/registry`). Also if you've
+installed Filebeat as a service on Windows the location stays the same
+(`C:\ProgramData\filebeat\registry`).
+
+If you are running Filebeat from the `tar.gz` archive, however, the location of
+the registry file has changed. In version 1.x it was by default named
+`.filebeat` in the current working directory. In version 5.0, it is
+`data/registry` relative to the binary location. For upgrading in this case, we
+recommend copying the `.filebeat` file to `data/registry`.
+
+==== Upgrading Topbeat to Metricbeat
+
+With the Beats 5.0 release, Topbeat is replaced by Metricbeat, which offers more
+functionality. More precisely, the Topbeat functionality is roughly equivalent
+with the "system" module in Metricbeat.
+
+While there is no automatic way to convert a Topbeat configuration file in the
+equivalent Metricbeat configuration file, both configuration are simple enough
+that migrating shouldn't take long.
+
+For example, this is the input section of the Topbeat configuration file:
+
+[source,yaml]
+------------------------------------------------------------------------------
+input:
+  # In seconds, defines how often to read server statistics
+  period: 10
+
+  # Regular expression to match the processes that are monitored
+  # By default, all the processes are monitored
+  procs: [".*"]
+
+  # Statistics to collect (all enabled by default)
+  stats:
+    # per system statistics, by default is true
+    system: true
+
+    # per process statistics, by default is true
+    process: true
+
+    # file system information, by default is true
+    filesystem: true
+
+    # cpu usage per core, by default is false
+    cpu_per_core: false
+------------------------------------------------------------------------------
+
+And here is the equivalent configuration for the system module in Metricbeat:
+
+[source,yaml]
+------------------------------------------------------------------------------
+metricbeat.modules:
+
+#------------------------------- System Module -------------------------------
+- module: system
+  metricsets:
+    # CPU stats
+    - cpu
+
+    # System Load stats
+    - load
+
+    # Per filesystem stats
+    - filesystem
+
+    # Memory stats
+    - memory
+
+    # Network stats
+    - network
+
+    # Per process stats
+    - process
+  enabled: true
+  period: 10s
+  processes: ['.*']
+------------------------------------------------------------------------------
+
+The rest of the configuration options (outputs, name, tags) stay the same or can
+be upgraded using the migration script.
+
+[[upgrade-mapping-template]]
+=== Upgrade the Elasticsearch mapping template
+
+By default, the Elasticsearch mapping template which comes with the Beats is not
+re-installed if it already exists.  This means that if the new Beats version
+adds new fields (e.g. a new module in Metricbeat), they won't be reflected in
+the Elasticsearch mapping, which can result in visualization problems or
+incorect data.
+
+You can set the `output.elasticsearch.template.overwrite` option to `true` to
+make the Beats install the new mapping template even if a template with the same
+name already exists. You can set this option in the configuration file or as a
+CLI flag, for example:
+
+[source,shell]
+------------------------------------------------------------------------------
+packetbeat -e -E output.elasticsearch.template.overwrite=true
+------------------------------------------------------------------------------
+
+However, the mapping template is applied on index creation, so it won't affect
+the current index if it was already created with the old version.
+
+You can force the start of a new index by renaming the old index before starting
+the new version of the Beat. As of Elasticsearch 5.0, this can be done via the
+re-index API:
+
+
+[source,json]
+------------------------------------------------------------------------------
+POST /_reindex
+{
+  "source": {
+    "index": "packetbeat-2016.09.20"
+  },
+  "dest": {
+    "index": "packetbeat-2016.09.20-old"
+  }
+}
+DELETE /packetbeat-2016.09.20
+------------------------------------------------------------------------------
+
+Note that the reindex API command can take a long time, depending on the size of
+the index. It is recommended that the Beat is stopped during this time, so the
+order of operations should be:
+
+1. Stop the old version of the Beat
+2. Rename the index of the current day
+3. Start the new version of the Beat
+
+If downtime is not acceptable, another possible approach is to configure a
+different index pattern in the new Beat version, but this will likely require
+adjustments to your Kibana dashboards.