Simplify the section about Configuring Filebeat

[dedemorton]

Right now, the section about configuring Filebeat contains a nested section called Configuration Options (Reference).

There are a few problems with this section:

• The section is meant to provide reference-style info, but because the options aren't alphabetized and the search is not always very, um, robust, users can't always find what they're looking for in the reference info. Plus the reference info contains more guidance info than you would typically find in pure ref content.
• As we've enhanced the book with more how-to type info, we keep encountering situations where we have content about a specific topic in two different places because we don't want to duplicate content. Expecting users to look in two places for configuration info is not ideal.
• We also have a full config file that provides short descriptions of every option, so it's possible that some users don't even use the reference section to understand the configuration options.
• Now that we want to add info about different types of prospectors, the current structure doesn't scale well because we can't add sub-levels in the navigation under Prospectors.

I'd like to propose that we remove the Configuration Options (Reference) container, pop all the configuration topics up one level in the nav, rename them using task/goal-oriented headings, and merge duplicated and repeated content so that we have one place to go for info about subjects like managing multiline messages. So the TOC would look something like this:

Configuring Filebeat
  Setting up prosectors
    Logs
    Redis slowlogs
    Multiple prospectors
  Managing multiline messages
  Specifying global and general options
  Configuring the output
    Elasticsearch
    Logstash
    Kafka
    Redis
    File
    Console
  Filtering and enhancing the exported data
    Defining processors
    Adding cloud metadata
    Adding the local timezone
    Decoding JSON fields
    Dropping events
    Dropping fields from events
    Adding additional fields to events
    Adding Kubernetes metadata
  Using ingest node
  Load balancing
  Reloading the config file
  Changing the output codec
  Setting paths
  Loading dashboards
  Loading the index template
  Logging
  YAML tips and gotchas
  Regular expression support

If we want to provide pure reference info, I suggest that we take the content that we currently use to generate the beat.full.yml file and generate a Quick Reference document in asciidoc that we can include in the docs. The benefit of doing so is that the content would be discoverable through google and provide a formatted version of the content that's in the full yml file. @ruflin I think you proposed something similar last year.

[dedemorton]

Of course, whatever we decide for Filebeat will need to be implemented across the docs for Beats.

[tsg]

Generally I like the idea, but seems like a pretty big undertaking :).

If we want to provide pure reference info, I suggest that we take the content that we currently use to generate the beat.full.yml file and generate a Quick Reference document in asciidoc that we can include in the docs.

The beat.full.yml is not really generated, more like multiple parts are concatenated together, but we write those comments manually in that form. We could probably turn it into asciidoc via some hacks, or better, we make a beat_config.yml from which we generate both the full config and the asciidoc.

It would be good to get rid of the duplication between the full config and the reference, I agree.

[monicasarbu]

I would replace:

• Load Balancing with Load Balancing the output hosts or something similar.
• Using Ingest Node with Parse logs with Ingest Node or something similar.
• Loading dashboards with Loading Kibana dashboards
• Loading the index template with Loading the Elasticsearch index template

I think we should have a section about prospector configuration where we can include the Setting paths section and the input types we support. I also think we should rename the Specifying global and general options to General Filebeat configuration options (or similar) and include the options from https://www.elastic.co/guide/en/beats/filebeat/current/configuration-global-options.html and from https://www.elastic.co/guide/en/beats/filebeat/current/configuration-general.html.

[dedemorton]

but seems like a pretty big undertaking :).

The changes mostly involve renaming topics and reorganizing the TOC, so it's not as much work as it seems on the surface.