Beats type as prefix: Change default behavior to disable

[waynekearns]

Expected Behavior

The fields sent by the Beats agent should be sent as they are ingested rather than adding an annotation to the field name (prefix).

Unless the customer explicitly desires this behavior.

Context

If I load a custom Beats configuration that normalized the data fields, or if I want to normalize my data based on the already standardized fields of the Beats agents, adding a prefix be default is not useful or desired.

This also incurs a higher metadata cost to the customer as the number of fields in a single message (especially for Winlogbeat) can be very numerous. Adding 7 bytes per field per message multiplied across the total number of logs can add up to a lot of data wasted.

Your Environment

• Graylog Version:3.0.2
• Elasticsearch Version: 6.7.2
• MongoDB Version: 4.0.9
• Operating System: Linux (Whatever)
• Browser version: All of them

[mpfz0r]

Ref #5542
on why we changed that.
The main reason was the conflicting source field.

Possible routes we could go:

• Rewrite the field in every beats configuration: Fields changed for new Beats input in Graylog 3  #5905 (comment)
• Rewrite it in the Beats Input (Either always, or with another option)

[bernd]

We also have to think about backwards compatibility if we should change the default. For example if there is anything that will break for existing users.

[waynekearns]

re: #5542 - I agree with the comment in #5905, the assignment of source is wrong in the default filebeat behavior. It muddies the waters when trying to separate out hosts, files, and services within logs. Providing a template with actions that make the agent behave in a more civilized manner is nice.

Re: the standard reassignment of the field. I would agree with defaulting the source field to something like source_file for filebeat and source_host for winlogbeat. Maybe source_type would be generic enough for both?

[tjanson]

Maybe this is old news to you, but I noticed that Beats 7 doesn’t have the source field anymore; that was changed back November in elastic/beats#8902. (The new fields have been backported to some 6.x versions but those still have source.)

So the old versions need to handled somehow, but apart from that it further supports the proposal to drop the prefix.

[mattpr]

Beats 7 doesn’t have the source field anymore

Yeah and it seems that all messages on hosts that are running Beats 7+ are now coming in source:unknown. Whereas beats 6 senders are still showing source:hostname.

Been reading docs and searching issues trying to figure out how to make graylog happy with the beats 7 senders...but haven't found anything explaining how to address this source:unknown issue. Would be nice to have this topic documented on the graylog end (how to deal with beats 6 vs 7 senders in terms of getting source set right).

The workaround I came up for here was just to manually re-add the source field on the beats sender end (e.g. in filebeat.yml or journalbeat.yml or whatever).

processors:
  - add_fields:
      target: '' # top level
      fields:
          source: <insert hostname or whatever here>

Since these files are templated by our infra provisioning anyway, inserting the hostname or whatever we want is not a big deal. This resolves the issue of many of our hosts being grouped under source:unknown in graylog.

Would be cool if on the beats (or other) input config on graylog we could specify an arbitrary key/field to use as the source.