[Metricbeat http module enhancement] Request for ability to parse more complex JSON objects in response

[radoondas]

Hi,
with current implementation, when you attempt to save response using HTTP module in Metricbeat and that response consists of complex JSON, it fails. That said, JSON must be a simple 'dictionary' like object.

For example, response as an array of JSON objects will produce an error message.
Response from a call to an API

[
{
  "fieldA" : "ValueA",
  "fieldB" : "Value B"
},
{
  "fieldA" : "ValueC",
  "fieldB" : "Value D"
},
{ ... }
]

Error

"error": {
    "message": "json: cannot unmarshal array into Go value of type map[string]interface {}"
  }

It would be best it this kind of a response could be parseable and would produce one document per object in array.

Thank you.

[christiangalsterer]

I can see here the following to options:
a) to add an artificial/additional "root" if the response is an array.
bI create a new event for each array element

[exekias]

I would go for b and create an event for each element in the array if that's what you get in the response

[ruflin]

+1 on option b as this is kind of the default what we do with arrays in metricbeat, especially as arrays (nested documents) don't work well in Kibana.

Implementation wise it should be pretty simple. Any suggestion for the name of the config option?

[christiangalsterer]

OK, I will open a PR in the next days. Should we try to detect this situation automatically and return an event for every array element as a list of arrays as anyway not working, i.e. we don't need a config option?

[exekias]

I would say we can do that without config options (autodetect). As of today you get an error, so it's not really changing a working behavior, but enhancing existing support.

Thank you for taking this!

[ruflin]

TBH I'm undecided on this one as I would like to get the user as much control as possible over what happens to the data. My assumption is the user is either having data without and array or with an array per endpoint he fetches data from. If the endpoint with an array does not return one, also an error should be logged. Also the array endpoint can lead to quite a bit of documents potentially, so I think opt in would be a good idea.

One thing I think would be interesting here is that in case we can't decode, we let the user know that there is such a config that "could" fix it.

[ruflin]

I tried to find this issue but couldn't so I opened #6472 I'm closing this one in favor of #6472 as it has more details about suggested implementation.

@christiangalsterer Did you ever find time to work on this?

[christiangalsterer]

Hi @ruflin,
sorry I didn't had as much time as I hope for for various reasons, but I can do in the next days.