[pkg/ottl]: Add functions for parsing CSV

[BinaryFissionGames]

Component(s)

pkg/ottl, processor/transform

Is your feature request related to a problem? Please describe.

Currently, there is no equivalent to stanza's CSV parsing operator in OTTL.

I would like to be able to parse CSV with OTTL, instead of using the logtransformprocessor which is planned to be eventually removed.

Describe the solution you'd like

Add 3 converter functions:

ParseCSV(target, header, delimiter, header_delimiter)
ParseCSVLazyQuotes(target, header, delimiter, header_delimiter)
ParseCSVIgnoreQuotes(target, header, delimiter, header_delimiter)

where:

• target is a Getter that returns a string
• header is a Getter that returns an array of string values (should accept either a string literal or a path expression)
• delimiter is a string
• and header_delimeter is a string

These functions all return a pcommon.Map struct that is the result of parsing the target string as CSV.

In terms of implementation, both ParseCSV and ParseCSVLazyQuotes will use encoding/csv to parse CSV. ParseCSVIgnoreQuotes will use strings.Split to parse. This is the same way that that the stanza package accomplishes CSV parsing.

The difference between the three functions is that:
ParseCSV will do typical CSV parsing
ParseCSVLazyQuotes will parse with the LazyQuotes option (see: https://pkg.go.dev/encoding/csv#Reader)
ParseCSVIgnoreQuotes will parse by completely ignoring quotes, just using strings.Split to split on the delimiter

Describe alternatives you've considered

ParseCSVIgnoreQuotes might be similar enough to Split that we could consider removing it.
What's nice about keeping this function is that it creates a map from the split, e.g.:

header: "Field1,Field2"
value: "Value 1,Value 2"

Turns into

Parsed: {"Field1": "Value 1", "Field2": "Value 2"}

And I don't think there's a currently a way to accomplish that column headers + row -< map operation currently, so I think it's still a valuable function.

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[BinaryFissionGames]

I'd like to take on adding these functions once we agree on design.

[TylerHelmuth]

In this situation the data was not collected by the filelogreceiver correct?

I believe we could use Optional parameters to reduce the functionality to 1 function. I also think both delimiters could be Optional with a default value of ,

[BinaryFissionGames]

Yes, this would be in a situation where the log is received and is not able to be parsed with operators in one of the stanza-based operators (e.g. receiving direct OTLP data with the otlp receiver).

It's been a while since I've touched a lot of the OTTL stuff, I completely missed that there are optional parameters!

Would it make sense for the signature to look something like this?

ParseCSV(target, header, Optional[delimiter], Optional[header_delimiter], Optional[lazyQuotes], Optional[ignoreQuotes])

We could condense the lazyQuotes and ignoreQuotes options into a single mode enum (e.g. one of quotes, lazyQuotes, ignoreQuotes), since really they should be considered mutually exclusive. I think I'd probably prefer that:

ParseCSV(target, header, Optional[delimiter], Optional[header_delimiter], Optional[mode])

[TylerHelmuth]

ParseCSV(target, header, Optional[delimiter], Optional[header_delimiter], Optional[mode])

This would be my preference. Annoyingly functions can't specify their own enums yet, a it would be a string, similar to strategy in merge_maps.

[TylerHelmuth]

@evan-bradley please take a look at this proposal

[evan-bradley]

This all sounds good to me.

[BinaryFissionGames]

@TylerHelmuth @evan-bradley I've opened a PR for adding the ParseCSV converter, when either of you get a chance to look!