feat(EKS Fargate): Add multiline support to EKS Fargate

[rnishtala-sumo]

Add multiline support to EKS Fargate

https://stagdata.long.sumologic.net/ui/#/search/create?id=HS2q7MasCkcxQkWvkMdVFUEfYJ3Ku7DYbByPc4XL

• Changelog updated or skip changelog label added
• Documentation updated
• Template tests added for new features

[sumo-drosiek]

Why do we need to reset it? We can use resource attribute in logstransform's recombine operator. Now I see we set it to use in logstransform and then we remove it, which seems like unnecessary overhead

[aboguszewski-sumo]

Yes, I agree. Initially we didn't think about that, but that makes a lot of sense.

[rnishtala-sumo]

I think we need to set the cloudwatch.log.stream as a record attribute so that we could use it to group all the log records under a resource. If we don't set this as a record attribute, then what would I group by?

this is from the doc for groupby attr

If the log record and metric data point has at least one of the specified attributes key, it will be moved to a Resource with the same value for these attributes

[aboguszewski-sumo]

I guess he's referring to setting this after grouping.

[rnishtala-sumo]

yes that makes sense, made the change.

[rnishtala-sumo]

@sumo-drosiek @aboguszewski-sumo actually I get the following warning when I try to use the log stream resource attribute as the source identifier for the recombine operator

2023-05-19T17:00:15.352Z    warn    recombine/recombine.go:235    entry does not contain the source_identifier, so it may be pooled with other sources    {"kind": "processor", "name": "logstransform/cloudwatch", "pipeline │
│ ": "logs/collector/otelcloudwatch", "operator_id": "merge-cri-lines", "operator_type": "recombine"}

I'll revert to copying the log stream attribute to record level to prevent unexpected issues.

[rnishtala-sumo]

confirmed that using a log stream record attr as the source id clears the above warning. In the interest of avoiding a scenario where we combine logs from multiple sources (as the above warning indicates), I'd like to keep the original changes

[sumo-drosiek]

The following is working for me:

configuration:

receivers:
  filelog:
    include:
      - ./tmp/logs/multiline.json  
    start_at: beginning
exporters:
  logging:
    verbosity: detailed
processors:
  logstransform/containers_parse_json:
    operators:
      - if: body matches "^{[\\s\\S]+"
        parse_from: body
        parse_to: body
        type: json_parser
      - type: add
        field: resource["cloudwatch.log.stream"]
        value: resource_attribute
  logstransform:
    operators:
      - id: merge-multiline-logs
        combine_field: body.log
        combine_with: "\n"
        is_first_entry: body.log matches "^a"
        source_identifier: resource["cloudwatch.log.stream"]
        # source_identifier: attributes["log.file.name"]
        type: recombine
service:
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - logstransform/containers_parse_json
        - logstransform
      exporters:
        - logging

file (./tmp/logs/multiline.json ):

{"log": "abc"}
{"log": "def"}
{"log": "ghi"}
{"log": "asdc"}

[rnishtala-sumo]

changed the source id to resource["cloudwatch.log.stream"] and that seems to work, no warnings. thanks!
https://stagdata.long.sumologic.net/ui/#/search/create?id=RDmXCQ1yhB5eBla9BPzgBCvMFYprpbG7EY0DAT7M

[sumo-drosiek]

IMHO, no need to move cloudwatch.log.stream to record level atributes. We whoulc be able to use resource level attribute

[sumo-drosiek]

Looks hacky 😅

[rnishtala-sumo]

yeah we're grouping all the logs by the cloudwatch log stream. we'll also plan to raise an upstream issue for the cloudwatch receiver so that the logs are placed (correctly) under a log stream resource.

[sumo-drosiek]

Thanks!

Need to investigate why resoure attributes are not working with recombine operator