Support for setting the document ID

[urso]

For helping users to reduce duplicates in Elasticsearch, Beats should have some opt-in support to configure the document ID. If the field @metadata.id is set, Beats will already use said ID as document ID. But this feature is still well hidden, because we currently have no support for setting this field.

If the document ID is set, Beats will publish using the op_type create.

• Add support to read IDs from existing JSON logs Add support to set the document ID in the filebeat json reader #5844
• fingerprint processor Fingerprint processor #14205
• processor adding a randomized UUID like ID (for example for metrics data) Add ID processor #14524

[urso]

@ycombinator Can we close this issue?

[ycombinator]

Yep! Resolved via #5844, #14205, and #14524.

[urso]

Testing:
While testing I found that we are missing support for setting the document ID via the json_decode_fields processor. The processor is used when JSON documents/events are embedded as strings. Fix is available in #15859

[mdelapenya]

Hi! 👋 Does this issue need to be added to the v7.6.0 test plan? Then I'd see convenient to add that label. @andresrc wdyt?

Thanks!

[urso]

Added missing v7.6.0 label. All found issues have been resolved in 7.6.