Follow up kubernetes processor

[exekias]

After initial merge from #3888, some things to review:

• Add missing unit tests
• Explore integration tests strategies
• Add documentation / tutorial on how to use the processor
• Add beta flags
• Unify field names across all k8s modules
• Complete fields.yml
• Move generic logic from mb / fb to libbeat, only leave activation in specific beats
• Clean up vendoring. Move as much as possible to the module itself.
• Add support for include_lables and include_annotations
• Add docs on how to write and register custom indexers/matchers.

[dedemorton]

We haven't really defined what it means for content to be a "tutorial." I'd like to reserve that label for real tutorials based around specific scenarios. If you think it's important to have a separate how-to section about using this processor, I suggest that we add a topic under Configuring called something like "Enhancing Events with Kubernetes Data". The file would live in libbeat and be included in the doc through asciidoc includes (so the content could be shared across the docs for multiple Beats). I did a quick review of the reference info, and it looks mostly OK.

[vjsamuel]

• Figure out vendoring strategry
• Add support for include_labels and include_annotations
• Improve documentation on how to write and register custom indexers/matchers.

[monicasarbu]

@dedemorton Regarding documentation, currently we have

I would say it's a bit confusing, as there are two sections that define the same thing: one is "Processors" and the other is "Filtering and Enhancing the exported data". To make it more visible, I would suggest moving the processors description under "Filtering and Enhancing the exported data", and use the description of the processor instead of its name. For example:

Filtering and Enhancing the Exported data

• Enhancing events with Cloud metadata
• Drop events
• Drop fields from events
• Add additional fields into your events
• Decode json fields
• Enhancing events with Kubernetes metadata

[dedemorton]

@monicasarbu This is a good idea. Originally the processors were under Configuration Options (Reference), so the structure made a bit more sense. But with the current structure, I agree...it's a bit confusing. I also like the idea of organizing the content around user goals. I think users are more likely to look for the task they are trying to complete (rather than looking for reference info). I'll create a separate issue to track this change because it's not really part of the Kubernetes work.

[dedemorton]

@exekias I've created a new issue to track the structural changes that Monica proposed, so you don't have worry about those: #4058

[vjsamuel]

#4152 should help cleanup vendoring

[dedemorton]

@exekias RE docs: With the restructuring of the content about processors (#4149), I've moved the content around a bit. If there's more info that you want to add about using the Kubernetes processor, I'd suggest adding it to the section called "Adding Kubernetes Metadata" in libbeat/docs/processors-using.asciidoc. The topic is no longer meant to be pure reference content, so it's appropriate to add more how-to type info there.

[exekias]

perfect @dedemorton I'll open a PR for that!

[jstangroome]

The documentation for the Kubernetes processor currently doesn't show any examples or explanation of how to use the exclude_labels, include_labels, or include_annotations configuration settings added by PRs #4043 and #4757.

• Config settings here: https://github.com/elastic/beats/blob/master/libbeat/processors/add_kubernetes_metadata/config.go#L22-L24
• Likely documentation location: https://github.com/elastic/beats/blob/master/libbeat/docs/processors-using.asciidoc#add-kubernetes-metadata

[elasticmachine]

Pinging @elastic/infrastructure

[dedemorton]

@exekias This issue has been open quite awhile. I'm wondering if I can do something to help resolve the remaining work. I'm not familiar with how these settings work and need some guidance.

[exekias]

All the points are done by now, closing this