Fields merging in 6.0 differs from 5.x

[tehmoon]

Following the bug report I filed on discuss: https://discuss.elastic.co/t/fields-merging-in-6-0-differs-from-5-x/109838

I was in the process of upgrade my filebeats from 5.6 to 6.0 on linux. But it broke my logstash pipeline. The reason is -- I think -- a bug in the merging fields process of the fields option in the yaml config file.

My configuration is as follow:

filebeat.prospectors:

- input_type: log
  paths:
    - /xxx/ansible_events/events.log
  fields:
    type: ansible_events

- input_type: log
  paths:
    - /xxx/xxx/log/main.log
  include_lines: ['^\[info\]'] # include info log
  exclude_lines: ['\)$', 'GET /users ', 'GET /ping '] # exclude lines that doesn't have the Sent information and also /users and /ping
  fields:
    type: cs_main

- input_type: log
  paths:
    - /var/log/syslog
    - /var/log/auth.log
    - /var/log/kern.log
  fields:
    type: syslog

name: filebeat

fields:
  env: production
  datacenter: yyyy
  group: zzzz

output.logstash:
  hosts: ["x.x.x.x:5044"]

path.data: /xxx/elastic_beats/filebeat/data
path.logs: /xxxx/elastic_beats/filebeat/log

I do all the sorting on the fields.type field. Everything worked fine in 5.6.4, the files syslog,auth.log and kern.log would have fields.type set to syslog. events.log would have it set to ansible_events and main.log to cs_main.

After upgrading to 6.0.0 all my events were set to syslog. A quick debugging showed that filebeat was taking the last input_type declared. To prove it, I just moved some section on top to the bottom and all of the events got that fields.type.

I'm not too sure how to debug further, but I'll happy to answer quickly if need help.

[ruflin]

I did a quick test here to reproduce it. The following still works as expected:

- type: log
  paths:
    - .../test1.log
  fields:
    type: test1

- type: log
  paths:
    - .../test2.log
  fields:
    type: test2

But if I add global fields like below, I have fields.type: test2 also in the events from test1.log.

fields:
  group: zzzz

@urso The problem must happen when merging global with local fields. Perhaps some state?

@tehmoon Thanks for reporting. A workaround for now would be to specify all fields local for each prospector.

[tehmoon]

@ruflin Thanks for checking! Interestingly enough, I didn't think of completely copying the whole fields field under each prospectors. I'll give it a try tomorrow. In any case this solution will work for me.

Regards,

[urso]

Thanks for testing. I will have a look.

[tehmoon]

@ruflin I do confirm that duplicating the whole field fields did fix it for me. I'm going to leave it like that since I would rather want to upgrade and have 3 lines in the config file duplicated every time than leaving a component lagging behind.

Thank you for responding so quick!!