Beats doc updates in 6.0

[dedemorton]

GA

• [DeDe] Document index setting change in the breaking changes. See Document change to LS index setting in Breaking Changes #5614
• [DeDe] Add link that points from release notes to breaking changes doc.
• [DeDe] Change getting started guides to show how to pass credentials by updating the config file - we do not want to recommend passing credentials at the command line.
• [DeDe] Add docs about manually loading the template (LS users need to do this). See Docs to manually loading template in 6.0 #5615 and Improve Beats + Logstash getting started steps #5080
• [DeDe] Update docs to reflect Set default credentials for Kibana #4867 (by default, the user doesn't need to configure the credentials for Kibana as they are taken from the Elasticsearch).
• [DeDe] Deprecate document_type in LS examples.
• [DeDe] Verify that the latest version of the breaking changes documentation reflects Pius's comments below.
• For auditbeat kernel metricset:
  • Document how to check if auditd is running, how to stop it, and how to disable it from starting at boot. service auditd status, service auditd stop, chkconfig auditd off
  • Document how to stop journald from listening to audit messages to save CPU and disk. systemctl mask systemd-journald-audit.socket
• Add info about setting a custom index name to the GS doc. See [Filebeat][Doc] Elasticsearch output - loading index template missing index name option #4254
• Set up redirects for topics that have moved in 6.0.
• For metricbeat:
  • We removed cpu_ticks and replaced it with cpu.metrics and core.metrics. diff Update and reorganize system module config docs #4909
• Docs around how to import/export dashboards are changing significantly to use Kibana API for importing dashboards.
• Change doc structure to accommodate prospector types. See Create Redis Prospector Configuration #4398
• Add/update compatibility statements for all modules in all Filebeat and Metricbeat (and others?); make compatibility sections consistent, including for 3rd party Outputs (Kafka/Redit).
• Add Beats usage with Cloud
• Add a link to the ES doc regarding removal of _types

RC1 and RC2

• [DeDe] Document modules.d layout for Metricbeat. See Document modules.d layout for Metricbeat #5207
• Solution pages - link off to docs - Beats doc landing pages need to be enhanced to make the user experience better.
• Remove import_dashboards from packaging. See Remove import_dashboards from packaging #4586
• [Monica] Update docs about how to create new dashboards in the dev guide: Updating docs about how to create new dashboards #5229 https://www.elastic.co/guide/en/beats/devguide/current/new-dashboards.html
  https://www.elastic.co/guide/en/beats/devguide/6.x/export-dashboards.html
• [DeDe] Add screen shot of Icinga dashboard.
• Verify that compatibility statements in the Filebeat module documentation are correct.

Beta 1 and 2

• [DeDe] Document new command line with sub-commands. See Document new command line interface #4490.
• [DeDe] Add docs for auditbeat. Implement a structure for config options that's similar to the one described here: Simplify the section about Configuring Filebeat #4422
• [DeDe] Update shared gs topics about template loading and importing dashboards
• [Tudor] Add docs around importing ML jobs
• [Tudor] Update breaking changes section of doc (see note from Pius below). See Document breaking changes in 6.0 #4737
• [Tudor] Write the upgrade guide for 5.x to 6.x Add upgrading guide docs #5068
• [Tudor] Heartbeat packages were renamed to heartbeat-elastic. This affects the command to start the service too.
• [DeDe] Update getting started docs about dashboard loading. See Include Kibana dashboards in the individual Beats packages #4675
• [kvch] Update Filbeat module development guide
• Add docs for PRs labeled as needs_docs:
  • [DeDe] Metricbeat test modules. Metricbeat test modules #4656
  • Update broker/pipeline setup. Update broker/pipeline setup #4650 Urso added in Add queue settings to docs #4884
  • [DeDe] Add filebeat modules config reloading. See Add filebeat modules config reloading #4566
  • [DeDe] Move to a modules.d layout. See Move to a modules.d layout #4537
  • [DeDe] Add support for json logging to file. See Add support for json logging to file #4523
  • [DeDe] UDP prospector type with plain harvester. See UDP prospector type with plain harvester #4452
  • Add Fields and Tag to monitor config. Add Fields and Tag to monitor config #4141

Moved backlog to #5632

[ppf2]

We will also want to update the breaking changes section of the doc. For example, there are breaking changes here and here, but these are not in the breaking changes section of the documentation. Naturally, users will be looking at the breaking changes section of the documentation for breaking changes, so we should move (or copy) all the breaking changes items from the release notes of all alphas, betas, and GA to the breaking changes guide.

Also, there is another breaking change I am aware of that does not appear to be documented today. Which is the fact that filebeat will not start now unless you have a module or prospector defined (and the default prospector for /var/log/*.log is disabled by default).

[ppf2]

Adding: #4633

[urso]

Pipeline changes will require doc updates as well: #4598

[andrewkroh]

For auditbeat kernel metricset:

• Document how to check if auditd is running, how to stop it, and how to disable it from starting at boot. service auditd status, service auditd stop, chkconfig auditd off
• Document how to stop journald from listening to audit messages to save CPU and disk. systemctl mask systemd-journald-audit.socket

For metricbeat:

• We removed cpu_ticks and replaced it with cpu.metrics and core.metrics. diff Update and reorganize system module config docs #4909

[tsg]

I checked the heartbeat one because heartbeat-elastic is already used here: https://www.elastic.co/guide/en/beats/heartbeat/6.0/setup-repositories.html and I also mentioned it briefly in the Upgrading guide. I think that covers it.

[brandonmensing]

@dedemorton Can we add a link to the ES doc regarding removal of _types?
From here: https://www.elastic.co/guide/en/beats/libbeat/6.0/breaking-changes-6.0.html#breaking-changes-types
To here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

[dedemorton]

@dedemorton Can we add a link to the ES doc regarding removal of _types?

@brandonmensing Yup will do. Your timing is good because I'm must now making some remaining doc updates related to removal of _types.

[brandonmensing]

@dedemorton Reminder comment for the types link :)

[dedemorton]

Re the breaking changes content: I think it's fairly clear from the wording (now) that "Breaking changes in 6.0" only describes the main changes that affect upgrade. Plus there is a link to the release notes for additional breaking changes. I think we're covered. It's an extra hop for users, but they might actually like that we distill the most important changes into a topic with more detail.

Plus...it's not a simple copy/paste to move the content. We pull in the changelog for the release notes. I don't like the idea of duplicating content, so I am planning to leave the relnotes as-is. I have added more links though to help people navigate between the BC and relnotes.

[dedemorton]

I've moved any pending items to #5632. Closing out this issue.