Clarify the input/output expectations of parsers

[tigrannajaryan]

In discussion here we discovered the need to review/rethink the decision about what LogRecord fields are the input and output of the parsers.

Today most parsers use Body string as the input and put the results into Body as nested fields. The question is whether this is the right and desirable approach or parsers should output the results into Attributes field. Should all parsers behave the same way? Should the behaviour be configurable?

[tigrannajaryan]

@djaglowski assigning this to you as agreed.

[djaglowski]

@tigrannajaryan, @pmm-sumo, @gillg: I've compiled some thoughts on this and related discussions in this doc. I attempt to rationalize several aspects of the current design, and propose a number changes. Looking forward to hearing your thoughts on this.

[gregoryfranklin]

Has there been any more discussion on this?

I'm looking at the journald input, which currently dumps everything into the body. The only thing it pulls out is the timestamp.

Ideally (IMHO), the body would be a string containing MESSAGE and all the other fields stored as attributes. Additionally, the priority would be converted to an OpenTelemetry severity. Given journald has a well defined list of fields [1], many of them could be converted to semantic conventions such as process.pid.

So, the question is whether things like the journald input should convert messages to a more native opentelemetry format? If not, should there be an option, operator or processor that could do it all in one go?

(I can create a PR to change the journald input if there is agreement)

1. https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html

[djaglowski]

Has there been any more discussion on this?

Yes, very recently actually. There's been a lot of progress towards stabilizing the data model, so this is being revisited as a next step. This doc includes some proposed changes. The doc is preliminary, so anything there will be written up as an issue against this repo before implementation.

In regards to the journald_input operator, I think there is merit to the general idea, but we should have more detailed discussion on that proposal on a separate issue.

[djaglowski]

Closed by log-collection#431