multiline parser for systemd Input plugin

[igorkchyts]

Bug Report

Describe the bug

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

2019-12-16 10:50:39,091 INFO [Timer-Driven Process Thread-7] o.a.n.processors.standard.LogAttribute LogAttribute[id=b0b835b7-016e-1000-1f89-20de2ee9d4dd] logging for flow file StandardFlowFileRecord[uuid=5e9da5e2-6541-4da7-9c60-ad19ecea3e2a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576152124890-2, container=default, section=2], offset=287653, length=2],offset=0,name=5e9da5e2-6541-4da7-9c60-ad19ecea3e2a,size=2]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
        Value: 'Mon Dec 16 10:50:29 IST 2019'
Key: 'lineageStartDate'
        Value: 'Mon Dec 16 10:50:29 IST 2019'
Key: 'fileSize'
        Value: '2'
FlowFile Attribute Map Content
Key: 'filename'
        Value: '5e9da5e2-6541-4da7-9c60-ad19ecea3e2a'
Key: 'path'
        Value: './'
Key: 'uuid'
        Value: '5e9da5e2-6541-4da7-9c60-ad19ecea3e2a'
--------------------------------------------------

Example from another container:

2019-12-17 15:08:30,851 [INFO ] [execute_flow] [getEnvsFromTopology:c9a8597f-16d2-4a76-a45e-634eb9db4210]: execute_flow_callback={
    "cloudEventsVersion": "0.1",
    "eventType": "",
        "eventID":"c9a8597f-16d2-4a76-a45e-634eb9db4210",
    "eventTime": "2019-12-17T15:08:29+0000",
    "source": "",
    "contentType": "application/json",
    "extensions": {
        "state": "Completed",
        "eventContext": ""
        },
    "data": {

        "flowName": "getEnvsFromTopology",
        "company": "",
        "project": "",
        "requestExternalEventID": "712288864f617f00449be3218110c726",
        "operationStatus": "success",
        "responseDetails": [{"replyItemName":"flow_output_message","replyItemStatus":"success","replyItemValue":{"Environment":[{"name":"PROD","type":"PROD"},{"name":"PET1","type":"PET"},{"name":"PET2","type":"PET"},{"name":"PET3","type":"PET"},{"name":"UAT1","type":"UAT"},{"name":"UAT2","type":"UAT"},{"name":"UAT3","type":"UAT"},{"name":"UAT4","type":"UAT"},{"name":"UAT5","type":"UAT"},{"name":"UAT6","type":"UAT"},{"name":"UAT7","type":"UAT"},{"name":"UAT8","type":"UAT"},{"name":"UAT9","type":"UAT"},{"name":"UAT10","type":"UAT"},{"name":"ST1","type":"ST"},{"name":"ST2","type":"ST"},{"name":"ST3","type":"ST"},{"name":"ST4","type":"ST"},{"name":"ST5","type":"ST"},{"name":"ST6","type":"ST"},{"name":"ST7","type":"ST"},{"name":"ST8","type":"ST"},{"name":"ST9","type":"ST"},{"name":"ST10","type":"ST"}]},"replyItemType":"Object"}]
    }
}

• Steps to reproduce the problem:
  Use Fluentbit systemd input plugin.
  Run any application that writes multiline logs.

Expected behavior
Ship multiline output logs as a single-line-log to elasticsearch, using systemd input plugin of Fluentbit.

Your Environment

• Version used:
  1.0.4

• Configuration:
  [INPUT]
  Name systemd
  Tag logging
  Path /var/log/journal
  Read_From_Tail On

  [PARSER]
  Name json
  Format json
  Time_Key time
  Time_Format %d/%b/%Y:%H:%M:%S %z

  [PARSER]
  Name docker
  Format json
  Time_Key time
  Time_Format %Y-%m-%dT%H:%M:%S.%L
  Time_Keep On
  # Command | Decoder | Field | Optional Action
  # =============|==================|=================
  Decode_Field_As escaped log

  [PARSER]
  Name syslog
  Format regex
  Regex ^<(?[0-9]+)>(?[^ ]* {1,2}[^ ]* [^ ]) (?[^ ]) (?[a-zA-Z0-9_/.-])(?:[(?[0-9]+)])?(?:[^\:]:)? (?.)$
  Time_Key time
  Time_Format %b %d %H:%M:%S

• Environment name and version (e.g. Kubernetes? What version?):
  1.15.1

• Server type and version:

• Operating System and version:
  Rhel 7.5

• Filters and plugins:

Additional context
We want to maintain the systemd input plugin and parse these log records by date, so then we send them to elasticsearch as a single-line-log.
Is this possible to do?

Thank you.

[edsiper]

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!