-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Convert Filebeat mysql.* to ECS (#10008)
- Convert many fields under `mysql.*` to ECS. Previous field names are field aliases towards the new corresponding ECS field: - mysql.error.message => message - mysql.error.level => log.level - mysql.error.thread_id => process.thread.id - mysql.slowlog.ip => source.ip - mysql.slowlog.host => source.domain - mysql.slowlog.user => user.name - Add expected test file to test changes to error logs - Coerce int fields: process.thread.id, mysql.slowlog.id, mysql.slowlog.rows_sent, mysql.slowlog.rows_examined and mysql.slowlog.timestamp - Coerce float fields: mysql.slowlog.query_time.sec, mysql.slowlog.lock_time.sec - Set event.created with the `@timestamp` from Filebeat - Change username in test file from 'apphost' to 'appuser', to make it easier to distinguish them in the expected file - Populate event.duration based on mysql.slowlog.query_time.sec - Get rid of a workaround for Ingest Node that was fixed in 5.0
- Loading branch information
Showing
12 changed files
with
299 additions
and
109 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| 161209 13:08:33 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql | ||
| 2016-12-09T12:08:33.335060Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). | ||
| 2016-12-09T12:08:33.335892Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path. | ||
| 2016-12-09T12:08:33.336610Z 0 [Note] /usr/local/Cellar/mysql/5.7.10/bin/mysqld (mysqld 5.7.10) starting as process 61571 ... | ||
| 2016-12-09T12:08:33.345527Z 0 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive | ||
| 2016-12-09T12:08:33.351596Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins | ||
| 2016-12-09T12:08:33.784722Z 0 [Note] /usr/local/Cellar/mysql/5.7.10/bin/mysqld: ready for connections. | ||
| Version: '5.7.10' socket: '/tmp/mysql.sock' port: 3306 Homebrew | ||
| 2016-12-09T22:21:02.443689Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 772568ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.) | ||
| 161209 14:18:50 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. | ||
| 161209 14:18:50 [Note] Plugin 'FEDERATED' is disabled. | ||
| 161209 14:18:50 InnoDB: The InnoDB memory heap is disabled |
148 changes: 148 additions & 0 deletions
148
filebeat/module/mysql/error/test/error.log-expected.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| [ | ||
| { | ||
| "@timestamp": "2016-12-09T13:08:33.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.offset": 0, | ||
| "message": "mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql", | ||
| "mysql.error.timestamp": "161209 13:08:33" | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.335Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Warning", | ||
| "log.offset": 92, | ||
| "message": "TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.335060Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.335Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Warning", | ||
| "log.offset": 282, | ||
| "message": "Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.335892Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.336Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Note", | ||
| "log.offset": 478, | ||
| "message": "/usr/local/Cellar/mysql/5.7.10/bin/mysqld (mysqld 5.7.10) starting as process 61571 ...", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.336610Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.345Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Warning", | ||
| "log.offset": 603, | ||
| "message": "Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.345527Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.351Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Note", | ||
| "log.offset": 742, | ||
| "message": "InnoDB: Mutexes and rw_locks use GCC atomic builtins", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.351596Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T12:08:33.784Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Note", | ||
| "log.offset": 832, | ||
| "message": "/usr/local/Cellar/mysql/5.7.10/bin/mysqld: ready for connections.", | ||
| "mysql.error.timestamp": "2016-12-09T12:08:33.784722Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.offset": 935, | ||
| "message": "Version: '5.7.10' socket: '/tmp/mysql.sock' port: 3306 Homebrew" | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T22:21:02.443Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Note", | ||
| "log.offset": 1002, | ||
| "message": "InnoDB: page_cleaner: 1000ms intended loop took 772568ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)", | ||
| "mysql.error.timestamp": "2016-12-09T22:21:02.443689Z", | ||
| "process.thread.id": 0 | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T14:18:50.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Warning", | ||
| "log.offset": 1176, | ||
| "message": "Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.", | ||
| "mysql.error.timestamp": "161209 14:18:50" | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T14:18:50.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.level": "Note", | ||
| "log.offset": 1367, | ||
| "message": "Plugin 'FEDERATED' is disabled.", | ||
| "mysql.error.timestamp": "161209 14:18:50" | ||
| }, | ||
| { | ||
| "@timestamp": "2016-12-09T14:18:50.000Z", | ||
| "ecs.version": "1.0.0-beta2", | ||
| "event.dataset": "mysql.error", | ||
| "event.module": "mysql", | ||
| "fileset.name": "error", | ||
| "input.type": "log", | ||
| "log.offset": 1422, | ||
| "message": "InnoDB: The InnoDB memory heap is disabled", | ||
| "mysql.error.timestamp": "161209 14:18:50" | ||
| } | ||
| ] |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.