Automatic tag generation for ingest pipeline processors

[chrisberkhout]

Manually maintaining tag values for processors in ingest pipelines is time consuming and can lead to confusion if processor settings drift away from the meaning captured in a tag value.

Good tag values can be generated automatically by combining the processor name with the values of certain processor options (best chosen based on the processor type), normalizing that, and adding a number if necessary for uniqueness within a pipeline.

An example with a generated tag name:

- set:
    tag: set_destination_ip
    field: destination.ip
    copy_from: parsed_json.ip

A rough POC can be found in tag-processors.py.

This could be made part of elastic-package format, or it could be a separate command.

[jsoriano]

So the idea is that all processors should have this tag set?

Adding this as part of elastic-package format could make sense, yes.

[chrisberkhout]

Yes, that's it.

When error messages include it (like below) it can help to locate the problem.

on_failure:
  - append:
      field: error.message
      value: >
        Processor "{{{ _ingest.on_failure_processor_type }}}"
        with tag "{{{ _ingest.on_failure_processor_tag }}}"
        in pipeline "{{{ _ingest.on_failure_pipeline }}}"
        failed with message "{{{ _ingest.on_failure_message }}}"