Add the ability to set custom fields #1092

andrewkroh · 2016-03-02T05:24:23Z

This pull request implements the feature requested in #726.

Changes Affecting All Beats

Added a fields and fields_under_root as options available
under the shipper configuration. These settings are analogous to settings
currently available in Filebeat. fields defined in the prospector
configuration take precedence over fields define in the shipper
configuration.

Changes Affecting Filebeat

Added the ability to set a list of tags for each prospector. These tags will
be appended to the list of tags specified in the shipper configuration.
Scalar values under fields are not longer automatically converted to strings.

Changes Affecting Winlogbeat

Added the ability to set tags, fields, and fields_under_root as options
for each event log.

Examples

Topbeat

shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
output:
  console:
    pretty: true

{
  "@timestamp": "2016-03-02T03:53:49.793Z",
  "aws": {
    "instance_id": "i-33458498",
    "region": "us-east-1"
  },
  "beat": {
    "hostname": "x",
    "name": "x"
  },
  "count": 1,
  "proc": {
    "cmdline": "/usr/bin/agentX",
    "cpu": {
      "start_time": "Feb05",
      "system": 7178,
      "total": 10789,
      "total_p": 0,
      "user": 3611
    },
    "mem": {
      "rss": 2162688,
      "rss_p": 0,
      "share": 0,
      "size": 2593021952
    },
    "name": "agentX",
    "pid": 395,
    "ppid": 1,
    "state": "running",
    "username": "someuser"
  },
  "type": "process"
}

Filebeat

filebeat:
  prospectors:
    - paths: ["/var/log/myapp/log.json"]
      fields: {app_id: 456789}
      tags: [json]
output:
  logstash:
    hosts: ["localhost:5044"]
shipper:
  tags: ['digital ocean', centos7]
  fields:
    customer_id: 1234567

{
  "@timestamp": "2016-03-02T04:12:32.491Z",
  "beat": {
    "hostname": "x",
    "name": "x"
  },
  "count": 1,
  "fields": {
    "app_id": 456789,
    "customer_id": 1234567
  },
  "input_type": "log",
  "message": "{ \"threadName\": \"MainThread\", \"name\": \"root\", \"thread\": 140735202359648, \"created\": 1336281068.506248, \"process\": 41937, \"processName\": \"MainProcess\", \"relativeCreated\": 9.100914001464844, \"module\": \"tests\", \"funcName\": \"testFormatKeys\", \"levelno\": 20, \"msecs\": 506.24799728393555, \"pathname\": \"tests/tests.py\", \"lineno\": 60, \"asctime\": [\"12-05-05 22:11:08,506248\"], \"message\": \"testing logging format\", \"filename\": \"tests.py\", \"levelname\": \"INFO\", \"special\": \"value\", \"run\": 12 }",
  "offset": 0,
  "source": "/var/log/myapp/log.json",
  "tags": [
    "digital ocean",
    "centos7",
    "json"
  ],
  "type": "log"
}

Winlogbeat

winlogbeat:
  event_logs:
    - name: MyCustomEventLog
      tags: [web]
      fields_under_root: true
      fields: {service_id: web01}
shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
output:
  console:
    pretty: true

See the test cases for more examples.

Open Questions

Are you happy with keeping this called fields? (same as it's called in Filebeat)

andrewkroh · 2016-03-02T06:26:16Z

The Jenkins failure is caused by #1083.

ruflin · 2016-03-02T06:56:22Z

filebeat/config/config.go

@@ -57,11 +57,11 @@ type ProspectorConfig struct {
 }

 type HarvesterConfig struct {
+	common.EventMetadata `config:",inline"` // Fields and tags to add to events.


+1 on approaching this directly with an eventMetadata object. That makes it easy to extend it later with other fields.

ruflin · 2016-03-02T07:02:30Z

LGTM.

Even though fields has its issue, as far as a I know we haven't found a better option yet and people got used to call it fields, so I would stay with it.

What happens if inside a prospector fields_under_root:true is set and in the shipper fields_under_root:false?

monicasarbu · 2016-03-02T10:11:43Z

Great work! LGTM

urso · 2016-03-02T13:50:27Z

we still need tags, if we got fields?

urso · 2016-03-02T13:52:29Z

filebeat/docs/reference/configuration/filebeat-options.asciidoc

+Optional fields that you can specify to add additional information to the
+output. For example, you might add fields that you can use for filtering log
+data. Fields can be scalar values, arrays, dictionaries, or any nested
+combination of these. All scalar values will be interpreted as strings. By


All scalar values will be interpreted as strings

This is not true anymore. Types are currently preserved.

I removed this sentence from the docs. I didn't see your comment prior to asking you the question on Slack.

ruflin · 2016-03-02T13:55:24Z

@urso About tags vs fields: I was thinking the same first but I get the feeling people use them differently. I would keep both if it doesn't add much complexity.

ruflin · 2016-03-02T14:28:56Z

@andrewkroh

There seems to be an issue on windows: https://ci.appveyor.com/project/elastic-beats/beats/build/1568/job/ls2h81bajqmoaghk
I didn't manage to reproduce the race condition from filebeat locally yet: http://build-eu-00.elastic.co/job/Beats-PR/769/console

andrewkroh · 2016-03-02T17:16:13Z

We could do away with tags since you could implement them with fields. But you couldn't merge the "global" and "local" tags if you used fields to implement tags yourself.

andrewkroh · 2016-03-02T17:19:40Z

What happens if inside a prospector fields_under_root:true is set and in the shipper fields_under_root:false?

The prospector fields will be added to the root of the document and the shipper fields will be under the fields key.

There seems to be an issue on windows: https://ci.appveyor.com/project/elastic-beats/beats/build/1568/job/ls2h81bajqmoaghk

This should be fixed now. It was due to scalars not being converted to strings under ucfg. The docs have been updated to reflect this change in behavior and the test fixed.

ruflin · 2016-03-03T11:49:52Z

@andrewkroh Interesting about the fields_under_root. I like it.

ruflin · 2016-03-03T11:52:45Z

LGTM. @andrewkroh Can you rebase?

ruflin · 2016-03-03T13:32:10Z

libbeat/common/mapstr.go

 	}
-	*ms = cleanUpInterfaceMap(result)
+
+	// Add fields and override.


@andrewkroh Seems like the old comment disappeared. The main reason I was thinking about logging is for debugging reason in case we get the info that some fields disappear. Lets leave it out for now and discuss it in case we hit such a case.

Add the ability to set tags in Filebeat prospector config Add the ability to set tags and fields in Winlogbeat event log config Update docs to remove wording about all scalars being changed to strings. Closes elastic#726

andrewkroh · 2016-03-03T21:07:36Z

It's rebased. Jenkins is failing because of a docker problem.

Add the ability to set custom fields

danielmotaleite · 2016-03-28T17:02:02Z

Any change of back-porting this to the current topbeat and packetbeat releases? i don't want to upgrade live servers with nightly builds, but i need this custom fields to be able to proper filter things on kibana.

andrewkroh · 2016-03-28T17:06:51Z

This feature will be included in the upcoming ~~1.2.0~~ 5.0 release.

andrewkroh · 2016-03-28T17:21:28Z

I amended my previous comment. I was thinking of environment variable replacement (which is in 1.2). The custom fields feature is in 5.0.

PauliniMrtns · 2018-03-21T14:53:07Z

Great work,
It would be great if it could be expanded to other beats as well. I opened an issue regarding Metricbeat for instance, where I need to spin-off some additional fields based on existing fields that already exist using metricbeat.

Thanks!

andrewkroh · 2018-03-21T16:40:50Z

Every Beat has support for fields, fields_under_root, and tags. And most support setting them at the module level too. Checkout the documentation for Metricbeat:

PauliniMrtns · 2018-03-21T17:43:26Z

Yes but this fields are static right? You define them at the metricbeat.yml if I'm not mistaken. Is there a way to make them variable according to another field pulled from a module? For instance vsphere.virtualmachine.name ?

Thank you!

andrewkroh · 2018-03-21T23:44:58Z

These fields are static. You could mutate the event in Logstash.

mehak1bharagava · 2019-11-22T15:39:26Z

@andrewkroh the documentation for adding tags isnt for 7.4 version anywhere. There are too many older versions of addings tags/fields and could you please tell how it is in 7.4 version? Will it be something like this-
filebeat.inputs:
- paths:
- Xxxxxxx\xxxxxxx\logs\dispatcher-scheduler.log
input_type: log
fields:
dispatcher: true
fields_under_root: true

This would be the way to tag each log file with its own tag and then in logstash create an index for this tag? Thanks, I have looked at discussion forum for ES but all are for older versions.

andrewkroh · 2019-11-22T17:15:33Z

The documentation for tags, fields, and fields_under_root can be found here. Also note that there are processors now for this -- add_fields and add_tags. The processors allow for a condition to be attached.

mehak1bharagava · 2019-11-22T18:32:43Z

The documentation for tags, fields, and fields_under_root can be found here. Also note that there are processors now for this -- add_fields and add_tags. The processors allow for a condition to be attached.

thank you so much, but has it been updated for logstash as well? From the add_field example, will index be formed in logstash too? For something like this-
filter {
if[type] =="DispatcherApp"{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
} }}
output {
if [@metadata][beat] == "filebeat"{
elasticsearch {
hosts => ["http://localhost:9200"]
sniffing => true
manage_template => false
index => "dispatcher-%{+YYYY.MM.dd}"

}
}

andrewkroh added enhancement review labels Mar 2, 2016

andrewkroh force-pushed the feature/fields-and-tags branch from 66b1dd0 to 87e1371 Compare March 2, 2016 05:28

ruflin reviewed Mar 2, 2016
View reviewed changes

urso reviewed Mar 2, 2016
View reviewed changes

andrewkroh force-pushed the feature/fields-and-tags branch from 87e1371 to 4b9e118 Compare March 2, 2016 17:13

ruflin reviewed Mar 3, 2016
View reviewed changes

Add the ability to set custom fields in shipper config

1043c19

Add the ability to set tags in Filebeat prospector config Add the ability to set tags and fields in Winlogbeat event log config Update docs to remove wording about all scalars being changed to strings. Closes elastic#726

andrewkroh force-pushed the feature/fields-and-tags branch from 4b9e118 to 1043c19 Compare March 3, 2016 15:59

ruflin added a commit that referenced this pull request Mar 3, 2016

Merge pull request #1092 from andrewkroh/feature/fields-and-tags

974c514

Add the ability to set custom fields

ruflin merged commit 974c514 into elastic:master Mar 3, 2016

andrewkroh added the v5.0.0-alpha1 label Mar 3, 2016

This was referenced Mar 3, 2016

Ability to add custom fields to all beats #726

Closed

Winlogbeat add "fields" and "fields_under_root" configuration options #989

Closed

andrewkroh deleted the feature/fields-and-tags branch March 4, 2016 05:54

ruflin mentioned this pull request Mar 15, 2016

Filebeat: File specific document type, fields, tags #719

Closed

andrewkroh mentioned this pull request May 2, 2016

Filebeat: retain original type of fields added in configuration #1534

Closed

mariussturm mentioned this pull request Oct 4, 2016

winlogbeat missing gl2_source_collector value Graylog2/collector-sidecar#73

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the ability to set custom fields #1092

Add the ability to set custom fields #1092

andrewkroh commented Mar 2, 2016

andrewkroh commented Mar 2, 2016

ruflin Mar 2, 2016

ruflin commented Mar 2, 2016

monicasarbu commented Mar 2, 2016

urso commented Mar 2, 2016

urso Mar 2, 2016

andrewkroh Mar 2, 2016

ruflin commented Mar 2, 2016

ruflin commented Mar 2, 2016

andrewkroh commented Mar 2, 2016

andrewkroh commented Mar 2, 2016

ruflin commented Mar 3, 2016

ruflin commented Mar 3, 2016

ruflin Mar 3, 2016

andrewkroh commented Mar 3, 2016

danielmotaleite commented Mar 28, 2016

andrewkroh commented Mar 28, 2016

andrewkroh commented Mar 28, 2016

PauliniMrtns commented Mar 21, 2018

andrewkroh commented Mar 21, 2018

PauliniMrtns commented Mar 21, 2018

andrewkroh commented Mar 21, 2018

mehak1bharagava commented Nov 22, 2019

andrewkroh commented Nov 22, 2019

mehak1bharagava commented Nov 22, 2019

Add the ability to set custom fields #1092

Add the ability to set custom fields #1092

Conversation

andrewkroh commented Mar 2, 2016

Changes Affecting All Beats

Changes Affecting Filebeat

Changes Affecting Winlogbeat

Examples

Open Questions

andrewkroh commented Mar 2, 2016

ruflin Mar 2, 2016

Choose a reason for hiding this comment

ruflin commented Mar 2, 2016

monicasarbu commented Mar 2, 2016

urso commented Mar 2, 2016

urso Mar 2, 2016

Choose a reason for hiding this comment

andrewkroh Mar 2, 2016

Choose a reason for hiding this comment

ruflin commented Mar 2, 2016

ruflin commented Mar 2, 2016

andrewkroh commented Mar 2, 2016

andrewkroh commented Mar 2, 2016

ruflin commented Mar 3, 2016

ruflin commented Mar 3, 2016

ruflin Mar 3, 2016

Choose a reason for hiding this comment

andrewkroh commented Mar 3, 2016

danielmotaleite commented Mar 28, 2016

andrewkroh commented Mar 28, 2016

andrewkroh commented Mar 28, 2016

PauliniMrtns commented Mar 21, 2018

andrewkroh commented Mar 21, 2018

PauliniMrtns commented Mar 21, 2018

andrewkroh commented Mar 21, 2018

mehak1bharagava commented Nov 22, 2019

andrewkroh commented Nov 22, 2019

mehak1bharagava commented Nov 22, 2019