[pkg/ottl] Support json flatten operations

[danelson]

Component(s)

• pkg/ottl
• processor/transform

Is your feature request related to a problem? Please describe.

Original discussion in https://cloud-native.slack.com/archives/C01N6P7KR6W/p1699999733202679

Some backends do not support json slices/maps. It would be nice if there were functions to support flattening the data. Consider the following

{
  "name": "test",
  "address": {
    "street": "first",
    "house": 1234
  },
  "occupants": ["user 1", "user 2"]
}

Possible outputs

1. Flatten to stringified json

   {
       "name": "test",
       "address": "{\"street\":\"first\", \"house\": 1234}",
       "occupants": "[\"user 1\", \"user 2\"]"
   }

2. Flatten to modified attribute names

   {
       "name": "test",
       "address.street": "first",
       "address.house": 1234,
       "occupants.0": "user 1",
       "occupants.1": "user 2"
   }

Initial questions:

• Can/should style 2 above support multiple styles (dot, underscore, path, etc.)?
• What if the backend supports slices but not maps? Can we only apply this to 1 or the other?
• Should there be a limit on depth?

Describe the solution you'd like

A function such as FlattenJSON, FlattenMap, FlattenSlice

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

I definitely prefer format 2. I hadn't thought about flattening slices like that. Is the normal? I expected slices to be left alone but your suggestion makes sense.

[bryan-aguilar]

Is there any standard here when it comes to flattening? Follow up question, would we want to support both types of flattening?

[ImDevinC]

I think the second solution is probably better, that's how Cloudwatch would parse these logs. The first one only exists because I wrote a simple processor for this on my own and that was the easier solution for me at the time.

[danelson]

I also prefer option 2.

I hadn't thought about flattening slices like that. Is the normal?

I am not sure if it is desirable. A long array would cause an explosion of attributes which may cause other issues.

That being said, some backends only support arrays of primitive types so

Works

{
  "items": ["value1", "value2"]
}

Does not work

{
  "items":
  [
    {
      "key": "value1"
    },
    {
      "key": "value2"
    }
  ]
}

It would be nice to have some options to handle this.

[bryan-aguilar]

A long array would cause an explosion of attributes which may cause other issues.

I think this is a good point but I think this is an inherent risk when flattening data like this.

With option 2 wouldn't this

{
  "items":
  [
    {
      "key": "value1"
    },
    {
      "key": "value2"
    }
  ]
}

become

{
  items.0.key: value1
  items.1.key: value2
}

?

[TylerHelmuth]

@bryan-aguilar I feel like option 1 is more of a stringify solution that flattening. I'd rather see that accomplished via set[attributes["address"], String(attributes["address"])).

I agree with your interpretation of how the slice would be flattened.

[danelson]

I think this is a good point but I think this is an inherent risk when flattening data like this.

If the desire is to prevent backends from rejecting data then this might trading 1 problem for another. For instance my backend doesn't supported json maps or more than 255 attributes. If I had (psuedo json)

{
  "key": "value",
  "slice": [ <300 items in here> ]
}

then my backend would drop "slice" but allow the rest of the payload. But if we turned this into

{
  "key": "value",
  "slice.0": "<item0>",
  "slice.1": "<item1>",
  ...
  "slice.299": "<item299"
}

Now my backend would drop the entire payload.

Could we support some type of depth parameter for nested levels and some kind of limit parameter for slice length to help guard against this?

I think you can use limit but I don't know if you could be targeted in your approach

[bryan-aguilar]

@danelson this is the same issue with objects though right? You could have an object with 350 nested fields that would flatten to 350 separate fields.

{
  "name": "test",
  "bigobj": {
    "field1": 1,
    "field2": 2,
    ....,
    "field350":  350
  }
}

[ImDevinC]

@danelson wouldn't the existing truncate_all be another solution for removing those excess attributes and not dropping everything? So you could flattenJSON and then truncate_all to remove excess that would be dropped?

[danelson]

this is the same issue with objects though right?

Yes, you are correct. I guess I wasn't thinking about that since it hasn't been an issue for us.

wouldn't the existing truncate_all be another solution for removing those excess attributes and not dropping everything? So you could flattenJSON and then truncate_all to remove excess that would be dropped?

I think you mean limit? I would like to give precedence to attributes that do not originate in maps/slices. I don't think that is possible. Since I don't know the names of the incoming attributes I have to apply flatten and then limit which will not work. My admittedly somewhat janky approach (because it isn't deterministic in ensuring the attribute limit is not breached) would be to support a limit option in the flatten function so I choose up to n items.

[danelson]

I realize my use case may not be standard. Just thought it would help the discussion.

I would be happy if there was support for the following which I think solves a lot of generic use cases

• Flattening maps only
• Flattening maps and slices
• A depth option for maps so that deeply nested structures can be ignored

[bryan-aguilar]

@danelson I wonder if your use case should be considered a complete separate concern. Should flatten function only worry about flattening. You should have the ability to size down your slices/maps before or after the flatten operation takes place.

Does the functionality to trim down the json object before or after flattening already?

[TylerHelmuth]

A depth option for maps so that deeply nested structures can be ignored

It is best to have these types of checks in the Where clause instead of the function itself. To enable this use case we could have a Depth converter that takes a map and returns its depth.

[danelson]

It is best to have these types of checks in the Where clause instead of the function itself.

This makes sense to me.

This may veering off track now, but is there a way to write transforms that apply this to subsets of data?

If I want to flatten a specific map I think we are saying that this proposal would allow for something like

flatten(attributes["maybe_deep_map"]) where Depth(attributes["maybe_deep_map"]) < 10

But would it be able flatten all maps that less than some depth when I don't know the name?

flatten(attributes) where Depth(attributes) < 10

[TylerHelmuth]

But would it be able flatten all maps that less than some depth when I don't know the name?

At the moment you would need to know the name.

[ImDevinC]

@TylerHelmuth just to clarify, we're saying that flatten(attributes) wouldn't work? That was the original problem I was trying to solve for, as dynamic attributes make it very difficult to know the name.

[danelson]

I think we are saying that flatten(attributes) would flatten everything. I was looking for more control which we think would be best provided via other checks and would be outside the scope of this issue.

[evan-bradley]

I think if we have a high degree of confidence that most users will want to limit map depth or list length when flattening a map of arbitrary keys, adding optional parameters would be an okay move here since it would simplify the overall UX for this functionality. If we can't do that, I think we could use #29289 to solve this case. Some combination of the below statements/conditions could be used to limit depth or remove keys that won't pass validation.

# NOTE: needs #29289 to work
for key, _ in attributes:
  flatten(attributes[key]) where IsMap(attributes[key]) and Depth(attributes[key]) < 10
  flatten(attributes[key]) where IsList(attributes[key]) and Len(attributes[key]) < 256
  delete_key(attributes, key) where IsMap(attributes[key]) and Depth(attributes[key]) >= 10
  delete_key(attributes, key) where IsList(attributes[key]) and Len(attributes[key]) >= 256

[puckpuck]

There should be an ability to prefix all the keys in the resulting flattened map.

{
  "name": "test",
  "address": {
    "street": "first",
    "house": 1234
  },
  "occupants": ["user 1", "user 2"]
}

results in something like this:

{
    "app.name": "test",
    "app.address.street": "first",
    "app.address.house": 1234,
    "app.occupants.0": "user 1",
    "app.occupants.1": "user 2"
}

This keeps my keys neatly packaged in a namespace so as to not be confused with other attributes.

[TylerHelmuth]

I can see that being an optional parameter. It gets more complicated if you want to namespace the nested maps. I think in that case the best solution is to call flatten multiple times with the different prefixes. I don't want to support namespacing the sub maps.

[DaveWK]

I'd just like to chime in that I am looking for this as well.

My particular use case is that I have JSON-encoded logs that are being scraped by OTEL using the journald receiver and want to collapse all the log elements that currently are shipped under Body.XXX to root-level elements, and then also re-parse the Body.MESSAGE element into sub-elements (since it's a json-formatted log) -- since there are arbitrary json elements within the MESSAGE log the current static mapping solution is not maintainable

[TylerHelmuth]

I've created #30455 to add this function. All interested parties please review.