null value fields are dropped from JSON decoding

[PhaedrusTheGreek]

• Filebeat 5.6.1 under OSX

Given a file like this:

{"test-null": null, "@timestamp":"2016-11-01T14:39:36.322", "type": 42,  "level":"INFO", "message":"test"}

With the following FB config:

filebeat.registry_file: /tmp/reg
filebeat.prospectors:

- paths:
    - /path/to/json-data.json
  json.keys_under_root: false
  close_eof: true

output.elasticsearch:
  hosts: ["http://localhost:9200"]

logging.level: debug

The test-null field is dropped before being sent to ES:

 {
        "_index": "filebeat-2017.11.06",
        "_type": "doc",
        "_id": "AV-SYVk_9oC3GfO5heZP",
        "_score": 1,
        "_source": {
          "@timestamp": "2017-11-06T17:29:29.150Z",
          ...
          "input_type": "log",
          "json": {
            "@timestamp": "2016-11-01T14:39:36.322",
            "level": "INFO",
            "message": "test",
            "type": 42
          },
          "offset": 1075,
          "source": "/path/to/json-data.json",
          "type": "log"
        }
      }

While hoping to make use of Elasticsearch's null_value mapping parameter.

[andrewkroh]

It's probably not the JSON decoder. Beats drop null values as part of the output process.

[ph]

@andrewkroh The null value mapping is an handy settings we could allow user to configure that behavior? @PhaedrusTheGreek an ingest pipeline could be a workaround for now, the downside is this might be more costly on space than the null value mapping? I don't know how this is implemented.

[PhaedrusTheGreek]

Yes, we could workaround in Logstash, but there is a lot of re-determination of information already present then lost from the original JSON document that would lead to inefficiencies in processing, space savings, maintenance/management, etc. We might prefer to wait for a fix if it's foreseeable.

[ph]

From @urso
https://github.com/elastic/beats/blob/master/libbeat/common/event.go#L53-L59

[ph]

Just to update this issue, we have decided to keep the current behavior as the default and add an opt-in flag to retain null values.

[gmoskovicz]

@ph

add an opt-in flag to retain null values.

What PR was this added in ?

[StephenOTT]

Was this added? @ph

[Zoya-bhavya]

i am sending my json to logsatsh why do event is dropped at filebeat ? how to handle it ?

[ph]

Not yet, I will resurrect that.

[Screamik]

Hello, guys!
Any news about the workaround for the issue? @ph