TCP input with parsers

[jlind23]

Goal

Parsers are the successors of readers of the log input. They provide a more flexible message parsing. At the moment they are only available in the filestream input. But the functionality could be useful in other inputs.

We have to inspect all available inputs in Filebeat to see if it makes sense to add parsers. If the answer is yes, then the feature should be added.

Parsers

• ndjson
• multiline
• container Add new parser to filestream input: container #26115
• syslog

How to add parsers to an input

1. Add parsers code to the input

The parser.Config object has its own Unpack function, so it is enough for you to add it as an attribute to your configuration.

type myInputConfiguration struct {
   // other attributes
    parsers parser.Config `config:",inline"`
}

Then create the parser based on the configuration. The Create function expects a reader.Reader as a parameter it can read from.

var reader reader.Reader
p = inp.config.parsers.Create(reader)

Example configuration accepted by the code above

parsers:
  - multiline:
      type: count
      lines_count: 3

2. Add parsers to the reference configuration

Add the reference configuration to either

• filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
• x-pack/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl

3. Add parsers to the documentation of the input

There is no common parsers section at the moment because it might not make sense to document all parsers for every input.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!