Ingest pipeline for filebeat postgresql module fails on multi-core thread_id

[dbuijs]

The ingest pipeline for the filebeat postgesql module defined here:

beats/filebeat/module/postgresql/log/ingest/pipeline.json

fails to parse postgresql log messages from multi-core machines because of the grok pattern for THREAD_ID.

On multi-core postgresql, the thread_id looks like this:

 [832-1]

Which will cause the whole grok pattern to fail because it expects to see a thread_id that looks like this:

[832]

[ruflin]

Could you share some example logs, the version of postgresql you use and your filebeat version?

[dbuijs]

I'm running postgresql 9.5, Filebeat version is 6.4.3 (amd64), libbeat 6.4.3

Here are some example messages that aren't parsed properly:

2018-11-17 03:14:01 EST [835-1] LOG:  autovacuum launcher started
2018-11-17 05:01:38 EST [832-1] LOG:  checkpoints are occurring too frequently (25 seconds apart)
2018-11-17 05:01:38 EST [832-2] HINT:  Consider increasing the configuration parameter "max_wal_size".
2018-11-17 03:13:56 EST [768-1] postgres@postgres FATAL:  the database system is starting up
2018-11-17 03:13:57 EST [771-1] postgres@postgres FATAL:  the database system is starting up

They do parse properly if I change the thread_id from [771-1] to [771]. My understanding is that the extra number appended to the pid is a session identifier (https://www.postgresql.org/docs/9.5/runtime-config-logging.html#GUC-LOG-LINE-PREFIX).