Filebeat 5.3.0 regex parsing regression

[karlgrose]

For confirmed bugs, please report:

• Version: Filebeat 5.3.0
• Operating System: RHEL 7.3
• Steps to Reproduce:

The following regex pattern:

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} '

worked for 5.2.2 but causes a panic with 5.3.0 [1]. Using this instead:

multiline.pattern: '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] '

allows filebeat to run again.

[1]
Mar 31 01:08:21 auth-d1 filebeat: panic: runtime error: index out of range
Mar 31 01:08:21 auth-d1 filebeat: goroutine 1 [running]:
Mar 31 01:08:21 auth-d1 filebeat: panic(0x9344c0, 0xc42000e060)
Mar 31 01:08:21 auth-d1 filebeat: /usr/local/go/src/runtime/panic.go:500 +0x1a1
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/libbeat/common/match.isPrefixNumDate(0xc4202fa700, 0x7f49d5d4b000)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/libbeat/common/match/cmp.go:107 +0x108
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/libbeat/common/match.compile(0xc4202fa700, 0xc4202fa700, 0xc4202700d4, xc4202fa4d0, 0x0)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/libbeat/common/match/compile.go:43 +0x8ec
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/libbeat/common/match.Compile(0xc4201940a0, 0x1c, 0x93caa0, 0x96d2e0, 0x6d201, 0x7f49d5cf7458)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/libbeat/common/match/matcher.go:60 +0x79
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/libbeat/common/match.(*Matcher).Unpack(0xc42019f060, 0xc4201940a0, 0x1c 0x1c, 0x0)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/libbeat/common/match/matcher.go:96 +0x39
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.unpackWith(0xc42006ff00, 0x96d2e0, 0x42019f060, 0x16, 0xc85340, 0xc420011ec0, 0xc42018de01, 0x8c98eb)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/unpack.go:183 +0x6da
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyMergeValue(0xc42006ff00, 0x0, 0x, 0x0, 0x0, 0x971ae0, 0xc42019f060, 0x199, 0xc85340, 0xc420011ec0, ...)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:377 +0x21a
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyGetField(0xc42018de00, 0xc42006f00, 0x0, 0x0, 0x0, 0x0, 0x8c98eb, 0x7, 0x971ae0, 0xc42019f060, ...)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:279 +0x288
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyStruct(0xc42006ff00, 0x991980, 0c42019f000, 0x199, 0xc42018de00, 0x0, 0xc42027d000)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:242 +0xa00
Mar 31 01:08:21 auth-d1 filebeat: github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyInto(0xc42006ff00, 0x991980, 0xc2019f000, 0x199, 0xc42018de00, 0xc42027d208, 0x40f05e)
Mar 31 01:08:21 auth-d1 filebeat: /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:139 +0x49c
Mar 31 01:08:21 auth-d1 systemd: filebeat2.service: main process exited, code=exited, status=2/INVALIDARGUMENT

[Detzler]

same problem after upgrade to 5.3..

restored previous version:

apt-get install filebeat=5.2.2

[urso]

the regression is due to the literal suffix in the regex. Fix has been merged to master and backported to 5.3 branch.

Workaround for 5.3 (if you can not drop the suffix): Change the leading \d{4} to \d\d\d\d to force the matcher to use old-style regex instead of compiling an optimized matcher (which generates the panic). The fix adds support for a literal suffix to the optimized matcher + ensures \d{4} and \d\d\d\d will be compiled into the exact same regex.