Add target for decode_json_fields #3169

martinscholz83 · 2016-12-10T21:52:28Z

This change add the output from decode_json_field to json field instead of message.

elasticsearch-release · 2016-12-10T21:57:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2016-12-10T21:57:04Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

martinscholz83 · 2016-12-12T08:00:42Z

This implementation added the output into the json field and doesn't remove the original field. Question is if we should remove the original field?

andrewkroh · 2016-12-12T16:18:57Z

This PR would break some of the use cases below. I recommend keeping the current behavior but adding an optional target configuration setting to control the destination. With that, I think all the use cases below would be covered.

There is no need to remove the original field in this processor because that could be done using using the drop_fields processor.

Use cases:

Decoding JSON in JSON
- Recipe: Use decode_json_fields with max_depth > 1. The target field is the source field in order to replace the strings with proper JSON objects.
- Status: Working now.
Decoding structure logs from an app running in Docker
- Recipe: Use Filebeat JSON decoder + decode_json_fields. The JSON decoder works on the wrapper added by Docker then decode_json_fields works on the structured log event from the app.
- Status: Working now.
- Note: If we had target this could be done without the Filebeat JSON decoder using the recipe above.
Decoding JSON events containing multiline data
- Recipe: Use Filebeat JSON decoder + Filebeat multiline. Docker puts each log line into a JSON object so if the application writes multline line events (like a stacktrace) then Filebeat must decode multiple JSON events and aggregate their messages.
- Status: Working now.
Decoding multiline JSON (pretty)
- Recipe: Use Filebeat multiline aggregator + decode_json_fields.
- Status: Not working for Filebeat because the target is the source (e.g. message is replaced with an object causing mapping conflict, Add target and remove_field options to decode_json_field #3134).
Decoding arbitrary JSON fields
- Recipe: Use decode_json_fields.
- Status: Working but with limitations. You can only replace the string field with an object. You cannot write the decoded data to a different target field. To do that you must be able to set the target field.

tsg · 2016-12-12T16:54:43Z

I'm +1 on adding a "target" option.

martinscholz83 · 2016-12-13T08:50:45Z

Are there names which the target field can not be named to. For example target: "source", target; offset. What i mean is should we compare the target field against a greylist.

andrewkroh · 2016-12-13T15:25:41Z

libbeat/processors/actions/decode_json_fields.go

@@ -17,18 +17,21 @@ type decodeJSONFields struct {
 	fields       []string
 	maxDepth     int
 	processArray bool
+	target       string


What if I want to target the root of the event? How would I configure this?

I think if you make it a *string then you can determine the difference between target being set but empty and target not present at all.

Thanks for that tip 👍 Still have to learn a lot when coming from OO. There is one more question. Like you say if we want to add to root it now looks like

{ "": { "test": { "test": "test" } }, "@timestamp": "2016-12-14T08:20:16.157Z", "beat": { "hostname": "4201halwsd00001", "name": "4201halwsd00001", "version": "6.0.0-alpha1" }, "input_type": "log", "message": "{\n \"test\": \"test\",\n \"test\": {\n \"test\": \"test\"\n }\n}", "offset": 73, "source": "input.json", "type": "log" }

It looks right? Or should we filter for that empty string and print depth one

{ "test": { "test": "test" }, "@timestamp": "2016-12-14T08:20:16.157Z", "beat": { "hostname": "4201halwsd00001", "name": "4201halwsd00001",

andrewkroh · 2016-12-13T15:27:00Z

libbeat/processors/actions/decode_json_fields_test.go

@@ -111,6 +111,36 @@ func TestValidJSONDepthTwo(t *testing.T) {

 }

+func TestTargetOption(t *testing.T) {


Thanks for adding a test case. 👍

karmi · 2016-12-14T08:22:38Z

Hi @maddin2016, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

andrewkroh · 2016-12-14T15:01:19Z

libbeat/processors/actions/decode_json_fields.go

@@ -75,7 +77,12 @@ func (f decodeJSONFields) Run(event common.MapStr) (common.MapStr, error) {
 				continue
 			}

-			_, err = event.Put(field, output)
+			if f.target != nil {


There will be three cases to handle here. When *f.target == "" you can't use event.Put, instead you need to add the fields from output to event. Looking at the json decoding in Filebeat, the default behavior is to not overwrite keys unless overwrite_key is set to true. So I would follow this model to be consistent.

martinscholz83 · 2016-12-15T12:19:45Z

@andrewkroh, how should we check if overwrite_keys is true or false. Should we add a separate config attribute for decode_json_fields?

…arget-decode-json-field

andrewkroh · 2016-12-22T15:04:44Z

Yeah, decode_json_fields would need its own boolean overwrite_keys config option.

andrewkroh · 2016-12-23T19:38:50Z

libbeat/processors/actions/decode_json_fields.go

+				if len(*f.target) > 0 {
+					_, err = event.Put(*f.target, output)
+				} else {
+					switch t := output.(type) {


The logic looks good. But now the complexity of the Run method is a bit high with all of the nested if/for/switch statements. WDYT about breaking this into smaller functions?

What do you think of adding a global helper function for that switch part. Because now we duplicate code. For example, helpers.jsonhelper.OverwriteJsonKeys?

Yeah, that makes sense if there is duplication. 👍

martinscholz83 · 2017-01-03T11:12:41Z

Hi @andrewkroh, do you have the chance to read my last commit. My idea was to create helper libraries under libbeat. For this case e.g

function OverwriteJsonKeys(event common.Mapstr, keys map[string]interface{})
{
...
}

andrewkroh · 2017-01-03T14:29:32Z

@maddin2016 Thanks a lot! I'm merging this now so that it can be included in the next release. Please open a new PR for the refactoring work.

ruflin added Filebeat Filebeat review labels Dec 12, 2016

andrewkroh reviewed Dec 13, 2016

View reviewed changes

martinscholz83 force-pushed the add-target-decode-json-field branch from 54cc2ce to 7568437 Compare December 14, 2016 08:24

andrewkroh reviewed Dec 14, 2016

View reviewed changes

martinscholz83 force-pushed the add-target-decode-json-field branch from 01748de to 45ecddb Compare December 15, 2016 07:16

martinscholz83 added 6 commits December 22, 2016 12:58

Add decoded output to json field

e245126

fix test

1fd75b0

Add target option

6cf262d

Make target default nil

00ae54f

Add filter to check if target belongs to root

93ad30f

Add test for target root

e5b4813

martinscholz83 force-pushed the add-target-decode-json-field branch from be86754 to e5b4813 Compare December 22, 2016 11:59

Merge remote-tracking branch 'refs/remotes/elastic/master' into add-t…

869e814

…arget-decode-json-field

martinscholz83 added 5 commits December 22, 2016 18:35

Add option overwrite_keys for decode_json

962e18e

Hardcode json_error key

d7f9ef0

Set config

de156a2

Fix test

877868a

Add changelog entry

2967c00

martinscholz83 mentioned this pull request Dec 23, 2016

Introduce namespace for metricset specific config options #3182

Closed

andrewkroh reviewed Dec 23, 2016

View reviewed changes

andrewkroh merged commit 3180831 into elastic:master Jan 3, 2017

martinscholz83 mentioned this pull request Jan 4, 2017

Add jsonhelper #3280

Merged

monicasarbu changed the title ~~Add target for decoded_json_field~~ Add target for decode_json_field Jan 9, 2017

monicasarbu changed the title ~~Add target for decode_json_field~~ Add target for decode_json_fields Jan 9, 2017

f-ld mentioned this pull request Feb 3, 2017

Add decode_json_fields processor to Beats config files #3513

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add target for decode_json_fields #3169

Add target for decode_json_fields #3169

martinscholz83 commented Dec 10, 2016

elasticsearch-release commented Dec 10, 2016

elasticmachine commented Dec 10, 2016

martinscholz83 commented Dec 12, 2016

andrewkroh commented Dec 12, 2016 •

edited

Loading

tsg commented Dec 12, 2016

martinscholz83 commented Dec 13, 2016

andrewkroh Dec 13, 2016

martinscholz83 Dec 14, 2016 •

edited

Loading

andrewkroh Dec 13, 2016

karmi commented Dec 14, 2016

andrewkroh Dec 14, 2016

martinscholz83 commented Dec 15, 2016

andrewkroh commented Dec 22, 2016

andrewkroh Dec 23, 2016 •

edited

Loading

martinscholz83 Dec 29, 2016 •

edited

Loading

andrewkroh Jan 3, 2017

martinscholz83 commented Jan 3, 2017

andrewkroh commented Jan 3, 2017

		@@ -111,6 +111,36 @@ func TestValidJSONDepthTwo(t *testing.T) {

		}

		func TestTargetOption(t *testing.T) {

Add target for decode_json_fields #3169

Add target for decode_json_fields #3169

Conversation

martinscholz83 commented Dec 10, 2016

elasticsearch-release commented Dec 10, 2016

elasticmachine commented Dec 10, 2016

martinscholz83 commented Dec 12, 2016

andrewkroh commented Dec 12, 2016 • edited Loading

tsg commented Dec 12, 2016

martinscholz83 commented Dec 13, 2016

andrewkroh Dec 13, 2016

Choose a reason for hiding this comment

martinscholz83 Dec 14, 2016 • edited Loading

Choose a reason for hiding this comment

andrewkroh Dec 13, 2016

Choose a reason for hiding this comment

karmi commented Dec 14, 2016

andrewkroh Dec 14, 2016

Choose a reason for hiding this comment

martinscholz83 commented Dec 15, 2016

andrewkroh commented Dec 22, 2016

andrewkroh Dec 23, 2016 • edited Loading

Choose a reason for hiding this comment

martinscholz83 Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

andrewkroh Jan 3, 2017

Choose a reason for hiding this comment

martinscholz83 commented Jan 3, 2017

andrewkroh commented Jan 3, 2017

andrewkroh commented Dec 12, 2016 •

edited

Loading

martinscholz83 Dec 14, 2016 •

edited

Loading

andrewkroh Dec 23, 2016 •

edited

Loading

martinscholz83 Dec 29, 2016 •

edited

Loading