Filebeat Prospector Processors Improvements

[ruflin]

This is a follow up issue for #3823

• Add documentation for prospector processors
• Add system tests
• Align naming / logic of EventHolder and outputs.Data: https://github.com/elastic/beats/blob/master/libbeat/outputs/outputs.go#L20 Potentially move it to common package.

[vjsamuel]

Should we also consider retiring the custom filters in the Harvester config?

	ExcludeLines    []match.Matcher         `config:"exclude_lines"`
	IncludeLines    []match.Matcher         `config:"include_lines"`

[ruflin]

@vjsamuel And use processors for it instead? I like the idea to reduce complexity / feature duplication here. @urso ?

[vjsamuel]

@ruflin : yes. This will help standardize the offering and also when we start supporting go plug-ins, users can plug in their own processors per file type and assign it to the prospector which would be awesome.

[ruflin]

Sounds like we will have beats specific processors in the future ...

[tsg]

I would leave the two for the time being, as they are implemented in the most efficient possible manner and are straight forward to use. Generally I think processors are harder to understand for the user, so I would reserve for them the less common use cases.

Sure, you can also use drop_events already as an alternative to exclude_lines, but I think most (new) users will find the exclude_lines more intuitive.

[ruflin]

+1 on leaving the config options as they are. Often these config options are simpler then using processors. I was more thinking of the "background" implementation that it should also be a processor. I'm kind of hoping we could make it as efficient in processors are with the direct implementation.

[tsg]

Closing as old.