Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add mariadb module for filebeat #7422

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added filebeat/docs/images/kibana-mariadb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 24 additions & 0 deletions filebeat/module/mariadb/_meta/config.reference.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#- module: mariadb
# Error logs
#error:
#enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:

# Slow logs
#slowlog:
#enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
16 changes: 16 additions & 0 deletions filebeat/module/mariadb/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
- module: mariadb
# Error logs
error:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Slow logs
slowlog:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
62 changes: 62 additions & 0 deletions filebeat/module/mariadb/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
:modulename: mariadb

== MariaDB module

The +{modulename}+ module collects and parses the slow logs and error logs
created by Rodrigo Floriano de Souza - @rdglinux <https://churrops.io/>

include::../include/what-happens.asciidoc[]

[float]
=== Compatibility

The +{modulename}+ module was tested with logs from versions 6.3.0.

[float]
=== Example dashboard

This module comes with a sample dashboard. For example:

[role="screenshot"]
image::./images/kibana-mariadb.png[]

include::../include/configuring-intro.asciidoc[]

The following example shows how to set paths in the +modules.d/{modulename}.yml+
file to override the default paths for slow logs and error logs:

["source","yaml",subs="attributes"]
-----
- module: mariadb
error:
enabled: true
var.paths: ["/path/to/log/mariadb/error.log*"]
slowlog:
enabled: true
var.paths: ["/path/to/log/mariadb/mariadb-slow.log*"]
-----


To specify the same settings at the command line, you use:

["source","sh",subs="attributes"]
-----
./{beatname_lc} --modules {modulename} -M "mariadb.error.var.paths=[/path/to/log/mariadb/error.log*]" -M "mariadb.slowlog.var.paths=[/path/to/log/mariadb/mariadb-slow.log*]"
-----



//set the fileset name used in the included example
:fileset_ex: error

include::../include/config-option-intro.asciidoc[]

[float]
==== `error` log fileset settings

include::../include/var-paths.asciidoc[]

[float]
==== `slowlog` fileset settings

include::../include/var-paths.asciidoc[]
11 changes: 11 additions & 0 deletions filebeat/module/mariadb/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
- key: mariadb
title: "MariaDB"
description: >
Module for parsing the MariaDB log files.
short_config: true
fields:
- name: mariadb
type: group
description: >
Fields from the MariaDB log files.
fields:
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"hits": 0,
"timeRestore": false,
"description": "",
"title": "Filebeat MariaDB Dashboard",
"uiStateJSON": "{\"P-1\":{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}}",
"panelsJSON": "[{\"col\":1,\"id\":\"MariaDB-slowest-queries\",\"panelIndex\":1,\"row\":8,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"MariaDB-Slow-queries-over-time\",\"panelIndex\":2,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":7,\"id\":\"MariaDB-error-logs\",\"panelIndex\":3,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":7,\"columns\":[\"mariadb.error.level\",\"mariadb.error.message\"],\"id\":\"Filebeat-MariaDB-error-log\",\"panelIndex\":4,\"row\":8,\"size_x\":6,\"size_y\":5,\"sort\":[\"@timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":7,\"id\":\"MariaDB-Error-logs-levels\",\"panelIndex\":5,\"row\":4,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"MariaDB-Slow-logs-by-count\",\"panelIndex\":6,\"row\":4,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"}]",
"optionsJSON": "{\"darkTheme\":false}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"sort": [
"@timestamp",
"desc"
],
"hits": 0,
"description": "",
"title": "Filebeat MariaDB Slow log",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"_exists_:mariadb.slowlog\"}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}"
},
"columns": [
"_source"
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"sort": [
"@timestamp",
"desc"
],
"hits": 0,
"description": "",
"title": "Filebeat MariaDB error log",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"filebeat-*\",\"query\":{\"query_string\":{\"query\":\"_exists_:mariadb.error\",\"analyze_wildcard\":true}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}"
},
"columns": [
"mariadb.error.level",
"mariadb.error.message"
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"visState": "{\"title\":\"MariaDB Slow logs by count\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"bottom\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"mariadb.slowlog.query\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
"description": "",
"title": "MariaDB Slow logs by count",
"uiStateJSON": "{}",
"version": 1,
"savedSearchId": "Filebeat-MariaDB-Slow-log",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"visState": "{\"title\":\"MariaDB Slow queries over time\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Slow queries\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}}],\"listeners\":{}}",
"description": "",
"title": "MariaDB Slow queries over time",
"uiStateJSON": "{\"vis\":{\"colors\":{\"Slow queries\":\"#EF843C\"}}}",
"version": 1,
"savedSearchId": "Filebeat-MariaDB-Slow-log",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"visState": "{\"title\":\"MariaDB error logs\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Error logs\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}}],\"listeners\":{}}",
"description": "",
"title": "MariaDB error logs",
"uiStateJSON": "{\"vis\":{\"colors\":{\"Count\":\"#447EBC\",\"Error logs\":\"#1F78C1\"}}}",
"version": 1,
"savedSearchId": "Filebeat-MariaDB-error-log",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"visState": "{\"title\":\"MariaDB slowest queries\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"mariadb.slowlog.query_time.sec\",\"customLabel\":\"Query time\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"mariadb.slowlog.query\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Query\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"mariadb.slowlog.user\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"User\"}}],\"listeners\":{}}",
"description": "",
"title": "MariaDB slowest queries",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"version": 1,
"savedSearchId": "Filebeat-MariaDB-Slow-log",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"visState": "{\"title\":\"MariaDB Error logs levels\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"bottom\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"mariadb.error.level\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
"description": "",
"title": "MariaDB Error logs levels",
"uiStateJSON": "{\"vis\":{\"colors\":{\"Note\":\"#9AC48A\",\"Warning\":\"#F9934E\",\"ERROR\":\"#E24D42\"}}}",
"version": 1,
"savedSearchId": "Filebeat-MariaDB-error-log",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
Loading