Document breaking changes in 6.0

[tsg]

From the CHANGELOG:

Affecting all Beats

• Rename kubernetes processor to add_kubernetes_metadata. Rename kubernetes processor -> add_kubernetes_metadata #4473 <-- kubernetes is new anyway, and in beta
• Rename *.full.yml config files to *.reference.yml. Rename .full.yml to .reference.yml #4563 <-- mention it in the upgrade guide
• The scripts/import_dashboards is removed from packages. Use the setup command instead. Remove import_dashboards from packaging #4586 <-- done
• Change format of the saved kibana dashboards to have a single JSON file for each dashboard Use import/export API for loading dashboards and index pattern #4413 <-- not really important from the user POV
• Rename configtest command to test config. Test command #4590 <-- configtest still works so not really a breaking change? Perhaps mention it in the upgrade guide?
• Remove setting queue_size and bulk_queue_size. Update broker/pipeline setup #4650 <-- done
• Remove setting dashboard.snapshot and dashboard.snapshot_url. They are no longer needed because the
  dashboards are included in the packages by default. Include Kibana dashboards in the individual Beats packages #4675 <-- minor
• Introduce beat version in the Elasticsearch index and mapping template Introduce beat version in index and template #3527 <-- is this breaking in any way?
• Usage of field _type is now ignored and hardcoded to doc. elasticsearch: set _type=doc #3757 <-- done in 5.x already
• Change vendor manager from glide to govendor. Moving from glide to govendor #3851 <-- not user facing
• Rename error field to error.message. Change error to error.message #3987 <-- minor
• Change dashboards.* config options to setup.dashboards.*. Improve and extend template and dashboard configurations #3921 <-- minor, since the main way of importing the dashboards in 5.x was import_dashboards
• Change outputs.elasticsearch.template.* to setup.template.*` Change template loading configuration to be part of setup #4080

Filebeat

• Rename input_type field to prospector.type Rename input_type to type in config and input_type to prospector.type in event #4294 <-- done
• The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. Fix panic due to no type #4331. <-- minor
• Remove code to convert states from 1.x. Remove convert old states code #3767 <-- i think only relevant in the sense that we should mention that upgrades have to be done major by major
• Remove deprecated config options force_close_files and close_older. Remove deprecated config options force_close_files and close_older #3768 <-- minor
• Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. Modify clean_removed handling #3827 <-- minor
• Remove document_type config option. Use fields instead. Remove document_type from Filebeat #4204 <-- done
• Move json_error under error.message and error.key. Move json_error to error.message and error.type #4167 <-- minor

Heartbeat

• Renamed the heartbeat RPM/DEB name to heartbeat-elastic. Rename heartbeat packages #4601 <-- mention in the upgrade guide

Metricbeat

• Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores.
  This will make the CPU usage percentages from the system cpu metricset consistent
  with the system process metricset. The documentation for these metrics already
  stated that on multi-core systems the percentages could be greater than 100%. Scale system.cpu.*.pct metrics by the number of cores #4544 <-- this was actually already done in 5.5
• Remove filters setting from metricbeat modules. Update metricbeat to use new publisher API #4699 <-- done

Packetbeat

• Remove the already unsupported pf_ring sniffer option. Delete pfring code #4608 <-- mionor
• Remove deprecated geoip. Remove deprecated geoip feature #3766 <-- minor
• Replace waitstop command line argument by shutdown_timeout in configuration file. Unify -waitstop and shutdown_timeout #3588 <-- minor

Winlogbeat

• Remove metrics endpoint. Replaced by http endpoint in libbeat (see Expose metrics through http endpoint #3717). Remove metrics endpoint in winlogbeat #3901 <-- this functionality doesn't exist, stop asking questions

Another one, from #4540:

• Also, there is another breaking change I am aware of that does not appear to be documented today. Which is the fact that filebeat will not start now unless you have a module or prospector defined (and the default prospector for /var/log/*.log is disabled by default). <-- done

[urso]

missing in filebeat:

• removed: filebeat.spool_size, filebeat.publish_async, filebeat.idle_timeout

[tsg]

Another one we shouldn't forget:

• cgo is enabled in all beats now

[tsg]

Almost forgot:

• It is no longer possible to enable two outputs (of different kind) at the same time.

We should also add a deprecation log in 5.6 for this.

[tsg]

There's one more that we missed:

• The module level processors in metricbeat need to specify the fields by their full name, not relative the the module.

[dedemorton]

@tsg I think we can close this now, right? Looks like the requirement for using the fully qualified name is mentioned here: https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-6.0.html#breaking-changes-filters

[dedemorton]

Closing because the work on this for 6.0 is done. If the breaking changes need additional updates, let's open a new issue.