Add support for pg_stat_statements to Metricbeat Postgresql module (#…

…7060) Add a new metricset, `statements` to the Metricbeat Postgres module. This metricset will poll [`pg_stat_statements`](https://www.postgresql.org/docs/9.6/static/pgstatstatements.html) and provide per-query statistic information which is not obtainable with just the current `activity` and `database` metricsets.
elastic · Jun 4, 2018 · 7a80be8 · 7a80be8
1 parent b9d2150
commit 7a80be8
Show file tree

Hide file tree

Showing 16 changed files with 613 additions and 3 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -262,6 +262,7 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
 - Add apiserver metricset to Kubernetes module. {pull}7059[7059]
 - Add maxmemory to redis info metricset. {pull}7127[7127]
 - Set guest as default user in RabbitMQ module. {pull}7107[7107]
+- Add postgresql statement metricset. {issue}7048[7048] {pull}7060[7060]
 - Update `state_container` metricset to support latest `kube-state-metrics` version. {pull}7216[7216]
 
 *Packetbeat*

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -11245,6 +11245,221 @@ type: date
 Time at which these statistics were last reset.
 
 
+--
+
+[float]
+== statement fields
+
+One document per query per user per database, showing information related invocation of that query, such as cpu usage and total time. Collected by querying pg_stat_statements.
+
+
+
+*`postgresql.statement.user.id`*::
++
+--
+type: long
+
+OID of the user logged into the backend that ran the query.
+
+
+--
+
+*`postgresql.statement.database.oid`*::
++
+--
+type: long
+
+OID of the database the query was run on.
+
+
+--
+
+*`postgresql.statement.query.id`*::
++
+--
+type: long
+
+ID of the statement.
+
+
+--
+
+*`postgresql.statement.query.text`*::
++
+--
+Query text
+
+
+--
+
+*`postgresql.statement.query.calls`*::
++
+--
+type: long
+
+Number of times the query has been run.
+
+
+--
+
+*`postgresql.statement.query.rows`*::
++
+--
+type: long
+
+Total number of rows returned by query.
+
+
+--
+
+*`postgresql.statement.time.total`*::
++
+--
+type: long
+
+Total number of milliseconds spent running query.
+
+
+--
+
+*`postgresql.statement.time.min`*::
++
+--
+type: long
+
+Minimum number of milliseconds spent running query.
+
+
+--
+
+*`postgresql.statement.time.max`*::
++
+--
+type: long
+
+Maximum number of milliseconds spent running query.
+
+
+--
+
+*`postgresql.statement.time.mean`*::
++
+--
+type: long
+
+Mean number of milliseconds spent running query.
+
+
+--
+
+*`postgresql.statement.time.stddev`*::
++
+--
+type: long
+
+Population standard deviation of time spent running query, in milliseconds.
+
+
+--
+
+*`postgresql.statement.memory.shared.hit`*::
++
+--
+type: long
+
+Total number of shared block cache hits by the query.
+
+
+--
+
+*`postgresql.statement.memory.shared.read`*::
++
+--
+type: long
+
+Total number of shared block cache read by the query.
+
+
+--
+
+*`postgresql.statement.memory.shared.dirtied`*::
++
+--
+type: long
+
+Total number of shared block cache dirtied by the query.
+
+
+--
+
+*`postgresql.statement.memory.shared.written`*::
++
+--
+type: long
+
+Total number of shared block cache written by the query.
+
+
+--
+
+*`postgresql.statement.memory.local.hit`*::
++
+--
+type: long
+
+Total number of local block cache hits by the query.
+
+
+--
+
+*`postgresql.statement.memory.local.read`*::
++
+--
+type: long
+
+Total number of local block cache read by the query.
+
+
+--
+
+*`postgresql.statement.memory.local.dirtied`*::
++
+--
+type: long
+
+Total number of local block cache dirtied by the query.
+
+
+--
+
+*`postgresql.statement.memory.local.written`*::
++
+--
+type: long
+
+Total number of local block cache written by the query.
+
+
+--
+
+*`postgresql.statement.memory.temp.read`*::
++
+--
+type: long
+
+Total number of temp block cache read by the query.
+
+
+--
+
+*`postgresql.statement.memory.temp.written`*::
++
+--
+type: long
+
+Total number of temp block cache written by the query.
+
+
 --
 
 [[exported-fields-prometheus]]

diff --git a/metricbeat/docs/modules/postgresql.asciidoc b/metricbeat/docs/modules/postgresql.asciidoc
@@ -103,9 +103,13 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-postgresql-database,database>>
 
+* <<metricbeat-metricset-postgresql-statement,statement>>
+
 include::postgresql/activity.asciidoc[]
 
 include::postgresql/bgwriter.asciidoc[]
 
 include::postgresql/database.asciidoc[]
 
+include::postgresql/statement.asciidoc[]
+
diff --git a/metricbeat/docs/modules/postgresql/statement.asciidoc b/metricbeat/docs/modules/postgresql/statement.asciidoc
@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-postgresql-statement]]
+=== PostgreSQL statement metricset
+
+beta[]
+
+include::../../../module/postgresql/statement/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-postgresql,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../module/postgresql/statement/_meta/data.json[]
+----
diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc
@@ -95,9 +95,10 @@ This file is generated! See scripts/docs_collector.py
 |<<metricbeat-module-php_fpm,PHP_FPM>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-php_fpm-pool,pool>> beta[]  
 |<<metricbeat-module-postgresql,PostgreSQL>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
-.3+| .3+|  |<<metricbeat-metricset-postgresql-activity,activity>>   
+.4+| .4+|  |<<metricbeat-metricset-postgresql-activity,activity>>   
 |<<metricbeat-metricset-postgresql-bgwriter,bgwriter>>   
 |<<metricbeat-metricset-postgresql-database,database>>   
+|<<metricbeat-metricset-postgresql-statement,statement>> beta[]  
 |<<metricbeat-module-prometheus,Prometheus>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .2+| .2+|  |<<metricbeat-metricset-prometheus-collector,collector>> beta[]  
 |<<metricbeat-metricset-prometheus-stats,stats>> beta[]  

diff --git a/metricbeat/include/fields.go b/metricbeat/include/fields.go
diff --git a/metricbeat/include/list.go b/metricbeat/include/list.go
@@ -103,6 +103,7 @@ import (
 	_ "github.com/elastic/beats/metricbeat/module/postgresql/activity"
 	_ "github.com/elastic/beats/metricbeat/module/postgresql/bgwriter"
 	_ "github.com/elastic/beats/metricbeat/module/postgresql/database"
+	_ "github.com/elastic/beats/metricbeat/module/postgresql/statement"
 	_ "github.com/elastic/beats/metricbeat/module/prometheus"
 	_ "github.com/elastic/beats/metricbeat/module/prometheus/collector"
 	_ "github.com/elastic/beats/metricbeat/module/prometheus/stats"

diff --git a/metricbeat/module/postgresql/_meta/Dockerfile b/metricbeat/module/postgresql/_meta/Dockerfile
@@ -1,2 +1,3 @@
 FROM postgres:9.5.3
-HEALTHCHECK --interval=10s --retries=6 CMD psql -h localhost -U postgres -l
+COPY docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
+HEALTHCHECK --interval=10s --retries=6 CMD psql -h localhost -U postgres -l
diff --git a/metricbeat/module/postgresql/_meta/docker-entrypoint-initdb.d/0-enable-pg_stat_statements.sh b/metricbeat/module/postgresql/_meta/docker-entrypoint-initdb.d/0-enable-pg_stat_statements.sh
@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+echo -e "shared_preload_libraries = 'pg_stat_statements'\npg_stat_statements.max = 10000\npg_stat_statements.track = all" >> $PGDATA/postgresql.conf
diff --git a/...ule/postgresql/_meta/docker-entrypoint-initdb.d/1-create-extension-pg_stat_statements.sql b/...ule/postgresql/_meta/docker-entrypoint-initdb.d/1-create-extension-pg_stat_statements.sql
@@ -0,0 +1 @@
+create extension pg_stat_statements;
diff --git a/metricbeat/module/postgresql/statement/_meta/data.json b/metricbeat/module/postgresql/statement/_meta/data.json
@@ -0,0 +1,54 @@
+{
+    "@timestamp": "2017-10-12T08:05:34.853Z",
+    "beat": {
+        "hostname": "host.example.com",
+        "name": "host.example.com"
+    },
+    "metricset": {
+        "host": "postgresql:5432",
+        "module": "postgresql",
+        "name": "statement",
+        "rtt": 115
+    },
+    "postgresql": {
+        "statement": {
+            "user": {
+                "id": 10
+            },
+            "database": {
+                "oid": "12407"
+            },
+            "query": {
+                "id":  "3240664890",
+                "text": "SELECT pg_sleep(?);",
+                "calls": "2",
+                "rows": "2",
+                "time": {
+                    "total": 120066.497,
+                    "min": 60029.533,
+                    "max": 60036.964,
+                    "mean": 60033.2485,
+                    "stddev": 3.71549999999843
+                },
+                "memory": {
+                    "shared": {
+                        "hit": 0,
+                        "read": 0,
+                        "dirtied": 0,
+                        "written": 0,
+                    },
+                    "local": {
+                        "hit": 0,
+                        "read": 0,
+                        "dirtied": 0,
+                        "written": 0,
+                    },
+                    "temp": {
+                        "read": 0,
+                        "written": 0,
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/metricbeat/module/postgresql/statement/_meta/docs.asciidoc b/metricbeat/module/postgresql/statement/_meta/docs.asciidoc
@@ -0,0 +1,3 @@
+=== postgresql statement MetricSet
+
+This is the statement metricset of the module postgresql.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		#!/usr/bin/env bash
		echo -e "shared_preload_libraries = 'pg_stat_statements'\npg_stat_statements.max = 10000\npg_stat_statements.track = all" >> $PGDATA/postgresql.conf
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		=== postgresql statement MetricSet

		This is the statement metricset of the module postgresql.