Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multi-core thread_id in filebeat postgresql module #9482

Merged
merged 6 commits into from
Dec 13, 2018
Merged

Add support for multi-core thread_id in filebeat postgresql module #9482

merged 6 commits into from
Dec 13, 2018

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Dec 11, 2018

Currently we only support postgresql log looks like:

2018-11-17 03:14:01 EST [835] LOG: autovacuum launcher started

But this will fail when parsing postgresql log messages from multi-core machines which looks like:

2018-11-17 03:14:01 EST [835-1] LOG: autovacuum launcher started

This PR is to add optional core id after the original thread id to parse multi-core postgresql logs. Please see more details in #9156

closes #9156

@@ -83,6 +83,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...master[Check the HEAD d
- Added `detect_null_bytes` selector to detect null bytes from a io.reader. {pull}9210[9210]
- Added `syslog_host` variable to HAProxy module to allow syslog listener to bind to configured host. {pull}9366[9366]
- Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format {issue}8015[8015] {issue}6111[6111] {pull}8768[8768].
- Add support for multi-core thread_id in postgresql module {issue}9156[9156]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add also a reference to this PR please

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep! will do!

@@ -0,0 +1,19 @@
2018-11-17 13:14:01 CEST [12975-1] [unknown]@[unknown] LOG: incomplete startup packet
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the expected file for this one please, I mentioned to remove the other one that was not related to this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano yes! I added it in the previous commit but the @timestamp failed on check job. That's why I tried to remove it to see if the check job will pass :-) I will definitely put it back in.

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, wait to jenkins, and please add a comment with a reference to the issue to fix the timestamps thing, thanks!

@kaiyan-sheng
Copy link
Contributor Author

@jsoriano Thanks for your help. I just created an issue for the timestamp problem: #9494

@kaiyan-sheng kaiyan-sheng added review and removed in progress Pull request is currently in progress. labels Dec 11, 2018
Copy link
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but needs a rebase.

Could you also update the PR description with what was changed in the PR? Please keep the links to the Github issues ;-)

@kaiyan-sheng
Copy link
Contributor Author

@ruflin Sounds good! I just updated the PR description. Thanks!

@kaiyan-sheng kaiyan-sheng merged commit 384b11b into elastic:master Dec 13, 2018
@jsoriano jsoriano added the needs_backport PR is waiting to be backported to other branches. label Dec 13, 2018
kaiyan-sheng added a commit that referenced this pull request Dec 17, 2018
…gresql module (#9482) (#9528)

* Add support for multi-core thread_id in filebeat postgresql module (#9482)

* Add support for multi-core thread_id in filebeat postgresql module

* Change timestamp for test log file

* update changelog and add multi-core.log-expected.json

* Change timestamp in example log file

* Update fields.go

(cherry picked from commit 384b11b)

* Run make update

* Run test with 6.x Elasticsearch
@kaiyan-sheng kaiyan-sheng deleted the multi-core-postgresql branch January 31, 2019 15:23
@kaiyan-sheng kaiyan-sheng removed the needs_backport PR is waiting to be backported to other branches. label Jan 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ingest pipeline for filebeat postgresql module fails on multi-core thread_id
3 participants