Review Multiline Docs

[ruflin]

With the implementation of multiline #570 the docs for multiline were added. Our current configuration documentation page gets larger and larger and it gets harder to find specific config options. We should discuss on how the page could be split up or restructured to make it better readable. An additional goal should be to be able to add more detailed documentation for multiline with examples. Where would such a page be?

In addition, currently the documentation for Regexp is not clear. This includes exclude / include line as they use the same regexp. We should separately document the regexp which can be used or link to a good documentation.

[dedemorton]

Let's discuss in more detail when I get back on Jan 4. But I'll add some notes here for when I get back:

I agree that the configuration topic is getting quite long. I've given this a bit of thought.

I considered breaking the configuration doc into a separate file for each section of the the config file. Ideally, users would be able to jump to a specific option description by clicking a link (the link would either be one of the generated links on the right of the page--my preference--or a jump table at the top of the page).

But implementing this change requires some rethinking of the current book structure to add nesting to the TOC. Because of how ASCIIDOC handles the "chunking" of content, the current structure would require the addition of quite a few "float" tags to create a nested TOC that is property structured. IMHO, the "float" tag is a hack that should be used sparingly, so I would want to look at restructuring the other content in our reference guides.

Also, we have to consider the fact that breaking the topic down into multiple topics means that users might need to click a couple of times to find what they're looking for. The advantage of the current structure is that everything is searchable on a single page.

I'd also like to consider organizing the options alphabetically within each section. The decision about whether to do this, though, hinges on how we expect people to use our doc. If we expect them to use it solely for reference info, then alphabetical is the way to go. If we think they'll be looking at the doc side-by-side with the config file, then maybe the current order (the order that options appear in the config file) is preferable.

Another issue that we need to consider: in the best of all possible worlds, we would have reference topics that describe what each option does, and then task-based "usage" topics that describe in more detail how to use the most common options together to configure specific behaviors. I haven't looked very closely at the docs added for multiline, but perhaps some of the detail could be teased out of the reference info and added to a usage topic. We have a few topics like this already (for example, https://www.elastic.co/guide/en/beats/packetbeat/current/_https_and_basic_authentication.html), which right now appear at the same level in the TOC as other topics. Over time, I'd like to provide more usage information (probably organized under a topic called Configuring Beatname), where we don't just throw users at a file containing all the options they could possibly want to use and instead describe how to configure behaviors that we expect users will want to configure, especially when the configuration relies on setting multiple related options.

[ruflin]

Lets discuss it in detail when you are back. I like to idea of organising it alphabetically. The challenge here is what we do with options that exist multiple times but nested in different hierarchies?

[dedemorton]

@ruflin Are you thinking about the repeated options under Output or something else? For the different types of output, I would probably create a separate topic for each, and organize the options alphabetically within the topic. This would also help users who are searching the doc for an option because they would have more context in the search results.

Another possibility is to keep the current organization (by order in config file) so that related options appear together, but then provide a jump list or table at the top of each topic where the options are organized alphabetically. This might seem a bit redundant with the auto-generated list of links that appears on the right side of the page, but we can experiment and see how it goes.

[ruflin]

Here some examples:

Multiline has different options which should be discussed together with examples of the different combinations:

      # Mutiline can be used for log messages spanning multiple lines. This is common
      # for Java Stack Traces or C-Line Continuation
      #multiline:

        # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
        #pattern: ^\[

        # Defines if the pattern set under pattern should be negated or not. Default is false.
        #negate: false

        # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
        # that was (not) matched before or after or as long as a pattern is not matched based on negate.
        # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
        #match: after

        # The maximum number of lines that are combined to one event.
        # In case there are more the max_lines the additional lines are discarded.
        # Default is 500
        #max_lines: 500

        # After the defined timeout, an multiline event is sent even if no new pattern was found to start a new event
        # Default is 5s.
        #timeout: 5s

Exclude lines and include lines are very similar and should be discussed together.

      # Exclude lines. A list of regular expressions to match. It drops the lines that are
      # matching any regular expression from the list. The include_lines is called before
      # exclude_lines. By default, no lines are dropped.
      # exclude_lines: ["^DBG"]

      # Include lines. A list of regular expressions to match. It exports the lines that are
      # matching any regular expression from the list. The include_lines is called before
      # exclude_lines. By default, all the lines are exported.
      # include_lines: ["^ERR", "^WARN"]

What multiline and *_lines have in common is the usage of regexp. As they both use the same regexp implementation, we should have a place where we discuss which regexp structures can be used to link to.

I think this is very similar to #652

[dedemorton]

I'm taking the v1.1.0 tag off of this, but keeping the issue open because I made some structural improvements to the multiline doc that make the doc acceptable for v1.1. However, the config topic is still very long and needs to be broken down into sub-topics.

Before I can do that, though, I need to restructure the doc content to support a nested TOC structure. I

[dedemorton]

I'm closing this because the doc restructuring work is complete. The remaining changes required to support the multiline doc are tracked in #740.