Add doc for json_decode_fields processor

[dedemorton]

I'm eager to get this processor documented, so I created a first draft of content for the Beats docs. I think this needs more detail. It's meant to be a starting point. I don't read Go very well, so I might have gotten some of the details wrong.

@suraj-soni I thought it might be easier for you to review this (rather than create the docs yourself) because our doc structure a little eccentric due to single sourcing some of the config content.

In addition to adding more detail to the descriptions (if you think it's needed), it would be nice to have a real example that shows the message string, the config that you would use to decode it, and the JSON object that would result. Can someone help me with this?

This PR fixes issue #2923

[andrewkroh]

The name of the processor is decode_json_fields instead of json_decode_fields.

[dedemorton]

whoops...good catch.

[andrewkroh]

@dedemorton Can you please mark this processor as experimental for 5.1 so that we can fix these config parameters in the next release. They should be named differently in the code to be consistent (like process_array instead of processArray).

[andrewkroh]

LGTM. @dedemorton Is this still in progress?

[dedemorton]

@andrewkroh I thought it might be nice to have a real example that shows the message string, the config that you would use to decode it, and the JSON object that would result. Do you think it's necessary, or am I over-thinking this? Can you help me come up with an example? Here's a starting point (based on the test case, though I think you didn't like the formatting of the string with all the escaping, so I've tried to fix that...haven't tested it tho).

Here's the text I would add (again, let me know if you think this belabors the obvious).

---

In the following example, a field called msg contains the following string:

'{"log":"{\\"level\\":\\"info\\"}\","stream":"stderr","count":3}'

To decode this string as a JSON object, you would use the following config:

+[source,yaml]
+-----------------------------------------------------
+processors:
+ - decode_json_fields:
+     fields: ["msg"]
+     process_array: false 
+     max_depth: 1
+-----------------------------------------------------

The resulting object would look like this:

[source,json]
-----------------------------------------------------
"log":    "{\"level\":\"info\"}",
"stream": "stderr",
"count":  3
-----------------------------------------------------

[andrewkroh]

@dedemorton Yes, I think that would be very helpful. Here's an example with Filebeat where the JSON decoded by Filebeat contained a field whose value is a JSON object encoded as a string. We apply the processor to decode that inner JSON object.

input.json

{ "outer": "value", "inner": "{\"data\": \"value\"}" }

filebeat.yml

filebeat.prospectors:
- paths:
    - input.json
  json.keys_under_root: true

processors:
  - decode_json_fields:
      fields: [inner]

output.console.pretty: true

output

{
  "@timestamp": "2016-12-06T17:38:11.541Z",
  "beat": {
    "hostname": "macbook13.local",
    "name": "macbook13.local",
    "version": "6.0.0-alpha1"
  },
  "inner": {
    "data": "value"
  },
  "input_type": "log",
  "offset": 55,
  "outer": "value",
  "source": "input.json",
  "type": "log"
}

[suraj-soni]

Apologies for the delayed review on this request. Had little to no access to technology for the better part of last week. Looking into this right now.

[suraj-soni]

Barring one typo in the param name, it LGTM

[suraj-soni]

Typo : "max_depth"

[dedemorton]

thanks. good catch!

[suraj-soni]

In @andrewkroh's example above, the fields configuration needs to be an array of strings

processors: decode_json_fields : fields : ["inner"]

[dedemorton]

@andrewkroh I've added your example (and changed the single quotes to double quotes as suggested, though I would think that single quotes is a valid string, no?). Anyhow, please review. Thanks!

[dedemorton]

@andrewkroh Still working on this one to get the wording right...so don't merge yet.

[dedemorton]

@andrewkroh Ok, the PR is ready for real now. I've added your example. I also reworked the topic where we introduce options for enhancing the data because I felt that the existing description was kind of confusing. Sorry to clutter up this PR...perhaps I should have made those changes in a different PR, but I had to strike while the iron was hot. :-)

@monicasarbu might want to take a look at the changes that I made to filebeat-filtering.asciidoc and processors.asciidoc to verify that I've captured what needs to be said there? If this is better, I'f apply the changes to the other beats, too.

[andrewkroh]

Please change the hostname and name fields to host.example.com. We use that value in other examples (like in Metricbeat).

[dedemorton]

done

[andrewkroh]

I like the ~80 char line wrap. 👍