[Filebeat] - Add raw json field when unmarshaling fails

[crazyvikas23]

Fixes #6516.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ruflin]

jenkins, test it

[andrewkroh]

I would like to see the raw JSON contained in the message field on error. Would that be possible to implement?

[crazyvikas23]

@andrewkroh I think you are right, message key inside error should only contain error message. I have added raw json data in message field outside.

[andrewkroh]

I think this is going to produce json.message in the final event and I'd like to have just message. It looks to me like the message.Content field is being populated with the text (raw json) when an error occurs.

So I think something upstream in the harvester needs to change to allow the fields["message"] = message.Content to be set. I think the issue lies within this code:

beats/filebeat/input/log/harvester.go

Lines 272 to 307 in a93acbd

	text := string(message.Content)
	// Check if data should be added to event. Only export non empty events.
	if !message.IsEmpty() && h.shouldExportLine(text) {
	fields := common.MapStr{
	"source": state.Source,
	"offset": state.Offset, // Offset here is the offset before the starting char.
	}
	fields.DeepUpdate(message.Fields)
	// Check if json fields exist
	var jsonFields common.MapStr
	if f, ok := fields["json"]; ok {
	jsonFields = f.(common.MapStr)
	}
	data.Event = beat.Event{
	Timestamp: message.Ts,
	}
	if h.config.JSON != nil && len(jsonFields) > 0 {
	ts := reader.MergeJSONFields(fields, jsonFields, &text, *h.config.JSON)
	if !ts.IsZero() {
	// there was a `@timestamp` key in the event, so overwrite
	// the resulting timestamp
	data.Event.Timestamp = ts
	}
	} else if &text != nil {
	if fields == nil {
	fields = common.MapStr{}
	}
	fields["message"] = text
	}
	data.Event.Fields = fields
	}

@ruflin I think we need your Filebeat expertise on this modification.

[crazyvikas23]

I think what you want is to append a message key in the final event. That should be pretty straightforward to implement. I have done that change but it seems too hackish.

[andrewkroh]

LGTM. It would be good to have a test case added to json_test.go that exercises the new code you added.

[andrewkroh]

Maybe we shouldn't put this raw message into the error logs. I'm worried about sensitive information being leaked inside of the Filebeat logs.

[andrewkroh]

That looks like it accomplishes it. 👍

[crazyvikas23]

@andrewkroh I don't think we can write a test case for same in json_test.go because file it contains unit test cases for json decoding, which already works fine. We should take this as integration test elsewhere.

[andrewkroh]

jenkins, test it

[andrewkroh]

jenkins, test it