Skip to content

Commit

Permalink
Support multiline logs in logstash/log fileset of Filebeat (elastic#8562
Browse files Browse the repository at this point in the history
)

Multiline JDBC plugin logs were not parsed correctly. From now on the module is capable of aggregating log lines into a single multiline event and its pipeline can parse it correctly.
  • Loading branch information
kvch authored Oct 5, 2018
1 parent c67b88a commit 7271c55
Show file tree
Hide file tree
Showing 5 changed files with 26 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
- Update CRI format to support partial/full tags. {pull}8265[8265]
- Fix some errors happening when stopping syslog input. {pull}8347[8347]
- Fix RFC3339 timezone and nanoseconds parsing with the syslog input. {pull}8346[8346]
- Support multiline logs in logstash/log fileset of Filebeat. {pull}8562[8562]

*Heartbeat*

Expand Down
4 changes: 4 additions & 0 deletions filebeat/module/logstash/log/config/log.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,7 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
pattern: ^\[[0-9]{4}-[0-9]{2}-[0-9]{2}
negate: true
match: after
5 changes: 3 additions & 2 deletions filebeat/module/logstash/log/ingest/pipeline-plain.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@
"field": "message",
"pattern_definitions": {
"LOGSTASH_CLASS_MODULE": "[\\w\\.]+\\s*",
"LOGSTASH_LOGLEVEL": "INFO|ERROR|DEBUG|FATAL|WARN|TRACE"
"LOGSTASH_LOGLEVEL": "INFO|ERROR|DEBUG|FATAL|WARN|TRACE",
"GREEDYMULTILINE" : "(.|\n)*"
},
"patterns": [
"\\[%{TIMESTAMP_ISO8601:logstash.log.timestamp}\\]\\[%{LOGSTASH_LOGLEVEL:logstash.log.level}\\s?\\]\\[%{LOGSTASH_CLASS_MODULE:logstash.log.module}\\] %{GREEDYDATA:logstash.log.message}"
"\\[%{TIMESTAMP_ISO8601:logstash.log.timestamp}\\]\\[%{LOGSTASH_LOGLEVEL:logstash.log.level}\\s?\\]\\[%{LOGSTASH_CLASS_MODULE:logstash.log.module}\\] %{GREEDYMULTILINE:logstash.log.message}"
]
}
},
Expand Down
4 changes: 4 additions & 0 deletions filebeat/module/logstash/log/test/logstash-plain.log
Original file line number Diff line number Diff line change
@@ -1 +1,5 @@
[2017-10-23T14:20:12,046][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-20T03:55:00,318][INFO ][logstash.inputs.jdbc ] (0.058950s) Select Name as [person.name]
, Address as [person.address]
from people

14 changes: 14 additions & 0 deletions filebeat/module/logstash/log/test/logstash-plain.log-expected.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,19 @@
"logstash.log.module": "logstash.modules.scaffold",
"offset": 0,
"prospector.type": "log"
},
{
"@timestamp": "2017-11-20T03:55:00,318",
"fileset.module": "logstash",
"fileset.name": "log",
"input.type": "log",
"log.flags": [
"multiline"
],
"logstash.log.level": "INFO",
"logstash.log.message": "(0.058950s) Select Name as [person.name]\n, Address as [person.address]\nfrom people\n",
"logstash.log.module": "logstash.inputs.jdbc ",
"offset": 175,
"prospector.type": "log"
}
]

0 comments on commit 7271c55

Please sign in to comment.