[Filebeat] Data loss when JSON decoding fails

[andrewkroh]

When JSON decoding fails Filebeat should include the raw data in the message field.

filebeat.prospectors:
- paths:
  - input.json
  json.keys_under_root: true
  json.add_error_key: true

output.console.enabled: true

input.json

Hello world!
{bad: json}
{"hello": "world"}

ata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.2"
  },
  "source": "/Users/akroh/Downloads/filebeat-6.2.2-darwin-x86_64/input.json",
  "offset": 13,
  "error": {
    "message": "Error decoding JSON: invalid character 'H' looking for beginning of value",
    "type": "json"
  },
  "beat": {
    "name": "macbook",
    "hostname": "macbook",
    "version": "6.2.2"
  }
}
{
  "@timestamp": "2018-03-08T21:52:43.891Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.2"
  },
  "source": "/Users/akroh/Downloads/filebeat-6.2.2-darwin-x86_64/input.json",
  "offset": 25,
  "error": {
    "type": "json",
    "message": "Error decoding JSON: invalid character 'b' looking for beginning of object key string"
  },
  "beat": {
    "name": "macbook",
    "hostname": "macbook",
    "version": "6.2.2"
  }
}
{
  "@timestamp": "2018-03-08T21:52:43.891Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.2"
  },
  "offset": 44,
  "hello": "world",
  "source": "/Users/akroh/Downloads/filebeat-6.2.2-darwin-x86_64/input.json",
  "beat": {
    "name": "macbook",
    "hostname": "macbook",
    "version": "6.2.2"
  }
}

[crazyvikas23]

#6591 This should solve it. But i think if json.add_error_key is set to false, data would be lost any way. So we should add another key for raw data.

[andrewkroh]

Fixed by #6591.