Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse the timezone in PostgreSQL logs as part of the date #12338

Merged
merged 5 commits into from
Jun 3, 2019
Merged

Parse the timezone in PostgreSQL logs as part of the date #12338

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ba6de6a
Properly parse the timezone in PostgreSQL logs
jsoriano May 29, 2019
fe31a49
Mark `postgresql.log.timestamp` field as deprecated
jsoriano May 29, 2019
b0a68d2
Merge remote-tracking branch 'origin/master' into postgresql-date-tim…
jsoriano May 29, 2019
de14e8e
Deprecate field on 7.3.0
jsoriano May 29, 2019
e467ae5
Add deprecation notice for timestamp field
jsoriano Jun 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
- Skipping unparsable log entries from docker json reader {pull}12268[12268]
- Parse timezone in PostgreSQL logs as part of the timestamp {pull}12338[12338]
- Require client_auth by default when ssl is enabled for tcp input {pull}12333[12333]

*Heartbeat*
Expand Down
3 changes: 3 additions & 0 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12149,6 +12149,9 @@ Fields from the PostgreSQL log files.
*`postgresql.log.timestamp`*::
+
--

deprecated[7.3.0]

The timestamp from the log line.


Expand Down
2 changes: 1 addition & 1 deletion filebeat/module/postgresql/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions filebeat/module/postgresql/log/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
Fields from the PostgreSQL log files.
fields:
- name: timestamp
deprecated: 7.3.0
jsoriano marked this conversation as resolved.
Show resolved Hide resolved
description: >
The timestamp from the log line.
- name: core_id
Expand Down
9 changes: 4 additions & 5 deletions filebeat/module/postgresql/log/ingest/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
"field": "message",
"ignore_missing": true,
"patterns": [
"^%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:event.timezone} \\[%{NUMBER:process.pid:long}(-%{BASE16FLOAT:postgresql.log.core_id:long})?\\] ((\\[%{USERNAME:user.name}\\]@\\[%{POSTGRESQL_DB_NAME:postgresql.log.database}\\]|%{USERNAME:user.name}@%{POSTGRESQL_DB_NAME:postgresql.log.database}) )?%{WORD:log.level}: (duration: %{NUMBER:temp.duration:float} ms statement: %{GREEDYDATA:postgresql.log.query}|%{GREEDYDATA:message})"
"^%{DATETIME:postgresql.log.timestamp} \\[%{NUMBER:process.pid:long}(-%{BASE16FLOAT:postgresql.log.core_id:long})?\\] ((\\[%{USERNAME:user.name}\\]@\\[%{POSTGRESQL_DB_NAME:postgresql.log.database}\\]|%{USERNAME:user.name}@%{POSTGRESQL_DB_NAME:postgresql.log.database}) )?%{WORD:log.level}: (duration: %{NUMBER:temp.duration:float} ms statement: %{GREEDYDATA:postgresql.log.query}|%{GREEDYDATA:message})"
],
"pattern_definitions": {
"LOCALDATETIME": "[-0-9]+ %{TIME}",
"DATETIME": "[-0-9]+ %{TIME} %{WORD:event.timezone}",
"GREEDYDATA": "(.|\n|\t)*",
"POSTGRESQL_DB_NAME": "[a-zA-Z0-9_]+[a-zA-Z0-9_\\$]*"
}
Expand All @@ -20,9 +20,8 @@
"field": "postgresql.log.timestamp",
"target_field": "@timestamp",
"formats": [
"yyyy-MM-dd HH:mm:ss.SSS", "yyyy-MM-dd HH:mm:ss"
],
"ignore_failure": true
"yyyy-MM-dd HH:mm:ss.SSS zz", "yyyy-MM-dd HH:mm:ss zz"
]
}
}, {
"script": {
Expand Down
72 changes: 36 additions & 36 deletions filebeat/module/postgresql/log/test/postgresql-9.6-debian-with-slowlog.log-expected.json
View file
Open in desktop

Large diffs are not rendered by default.

32 changes: 16 additions & 16 deletions filebeat/module/postgresql/log/test/postgresql-9.6-multi-core.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"@timestamp": "2017-04-03T22:32:14.322Z",
"@timestamp": "2017-04-03T20:32:14.322Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -12,13 +12,13 @@
"message": "incomplete startup packet",
"postgresql.log.core_id": 1,
"postgresql.log.database": "unknown",
"postgresql.log.timestamp": "2017-04-03 22:32:14.322",
"postgresql.log.timestamp": "2017-04-03 22:32:14.322 CEST",
"process.pid": 12975,
"service.type": "postgresql",
"user.name": "unknown"
},
{
"@timestamp": "2017-04-03T22:32:14.322Z",
"@timestamp": "2017-04-03T20:32:14.322Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -30,13 +30,13 @@
"message": "database \"user\" does not exist",
"postgresql.log.core_id": 1,
"postgresql.log.database": "user",
"postgresql.log.timestamp": "2017-04-03 22:32:14.322",
"postgresql.log.timestamp": "2017-04-03 22:32:14.322 CEST",
"process.pid": 5404,
"service.type": "postgresql",
"user.name": "postgres"
},
{
"@timestamp": "2017-04-03T22:35:22.389Z",
"@timestamp": "2017-04-03T20:35:22.389Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.duration": 37598000,
Expand All @@ -53,13 +53,13 @@
"postgresql.log.core_id": 2,
"postgresql.log.database": "postgres",
"postgresql.log.query": "SELECT n.nspname as \"Schema\",\n\t c.relname as \"Name\",\n\t CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' END as \"Type\",\n\t pg_catalog.pg_get_userbyid(c.relowner) as \"Owner\"\n\tFROM pg_catalog.pg_class c\n\t LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\n\tWHERE c.relkind IN ('r','')\n\t AND n.nspname <> 'pg_catalog'\n\t AND n.nspname <> 'information_schema'\n\t AND n.nspname !~ '^pg_toast'\n\t AND pg_catalog.pg_table_is_visible(c.oid)\n\tORDER BY 1,2;",
"postgresql.log.timestamp": "2017-04-03 22:35:22.389",
"postgresql.log.timestamp": "2017-04-03 22:35:22.389 CEST",
"process.pid": 5404,
"service.type": "postgresql",
"user.name": "postgres"
},
{
"@timestamp": "2017-07-31T13:36:43.557Z",
"@timestamp": "2017-07-31T17:36:43.557Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -70,12 +70,12 @@
"log.offset": 897,
"message": "autovacuum launcher started",
"postgresql.log.core_id": 1,
"postgresql.log.timestamp": "2017-07-31 13:36:43.557",
"postgresql.log.timestamp": "2017-07-31 13:36:43.557 EST",
"process.pid": 835,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:36:44.227Z",
"@timestamp": "2017-07-31T17:36:44.227Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -86,12 +86,12 @@
"log.offset": 967,
"message": "checkpoints are occurring too frequently (25 seconds apart)",
"postgresql.log.core_id": 1,
"postgresql.log.timestamp": "2017-07-31 13:36:44.227",
"postgresql.log.timestamp": "2017-07-31 13:36:44.227 EST",
"process.pid": 832,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:46:02.670Z",
"@timestamp": "2017-07-31T17:46:02.670Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -102,12 +102,12 @@
"log.offset": 1069,
"message": "Consider increasing the configuration parameter \"max_wal_size\".",
"postgresql.log.core_id": 2,
"postgresql.log.timestamp": "2017-07-31 13:46:02.670",
"postgresql.log.timestamp": "2017-07-31 13:46:02.670 EST",
"process.pid": 832,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:46:23.016Z",
"@timestamp": "2017-07-31T17:46:23.016Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -119,13 +119,13 @@
"message": "the database system is starting up",
"postgresql.log.core_id": 1,
"postgresql.log.database": "postgres",
"postgresql.log.timestamp": "2017-07-31 13:46:23.016",
"postgresql.log.timestamp": "2017-07-31 13:46:23.016 EST",
"process.pid": 768,
"service.type": "postgresql",
"user.name": "postgres"
},
{
"@timestamp": "2017-07-31T13:46:55.637Z",
"@timestamp": "2017-07-31T17:46:55.637Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -137,7 +137,7 @@
"message": "the database system is starting up",
"postgresql.log.core_id": 1,
"postgresql.log.database": "postgres",
"postgresql.log.timestamp": "2017-07-31 13:46:55.637",
"postgresql.log.timestamp": "2017-07-31 13:46:55.637 EST",
"process.pid": 771,
"service.type": "postgresql",
"user.name": "postgres"
Expand Down
20 changes: 10 additions & 10 deletions filebeat/module/postgresql/log/test/postgresql-9.6-new-timestamp.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
{
"@timestamp": "2017-07-31T13:36:43.000Z",
"@timestamp": "2017-07-31T17:36:43.000Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -11,12 +11,12 @@
"log.offset": 0,
"message": "autovacuum launcher started",
"postgresql.log.core_id": 1,
"postgresql.log.timestamp": "2017-07-31 13:36:43",
"postgresql.log.timestamp": "2017-07-31 13:36:43 EST",
"process.pid": 835,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:36:44.000Z",
"@timestamp": "2017-07-31T17:36:44.000Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -27,12 +27,12 @@
"log.offset": 66,
"message": "checkpoints are occurring too frequently (25 seconds apart)",
"postgresql.log.core_id": 1,
"postgresql.log.timestamp": "2017-07-31 13:36:44",
"postgresql.log.timestamp": "2017-07-31 13:36:44 EST",
"process.pid": 832,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:46:02.000Z",
"@timestamp": "2017-07-31T17:46:02.000Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -43,12 +43,12 @@
"log.offset": 164,
"message": "Consider increasing the configuration parameter \"max_wal_size\".",
"postgresql.log.core_id": 2,
"postgresql.log.timestamp": "2017-07-31 13:46:02",
"postgresql.log.timestamp": "2017-07-31 13:46:02 EST",
"process.pid": 832,
"service.type": "postgresql"
},
{
"@timestamp": "2017-07-31T13:46:23.000Z",
"@timestamp": "2017-07-31T17:46:23.000Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -60,13 +60,13 @@
"message": "the database system is starting up",
"postgresql.log.core_id": 1,
"postgresql.log.database": "postgres",
"postgresql.log.timestamp": "2017-07-31 13:46:23",
"postgresql.log.timestamp": "2017-07-31 13:46:23 EST",
"process.pid": 768,
"service.type": "postgresql",
"user.name": "postgres"
},
{
"@timestamp": "2017-07-31T13:46:55.000Z",
"@timestamp": "2017-07-31T17:46:55.000Z",
"ecs.version": "1.0.0",
"event.dataset": "postgresql.log",
"event.module": "postgresql",
Expand All @@ -78,7 +78,7 @@
"message": "the database system is starting up",
"postgresql.log.core_id": 1,
"postgresql.log.database": "postgres",
"postgresql.log.timestamp": "2017-07-31 13:46:55",
"postgresql.log.timestamp": "2017-07-31 13:46:55 EST",
"process.pid": 771,
"service.type": "postgresql",
"user.name": "postgres"
Expand Down
Loading