Filebeat Kibana module log fileset #7052
Conversation
ruflin
added
in progress
|
@tsg The Kibana log is pretty interesting as it's all in json which is pretty nice. The downside of it is there are lots of fields and I assume in my test logs not all fields that can show up in there. At the moment I'm not sure how we should deal with all these fields as I don't think it will scale to map all of them. I was thinking perhpas we can pick the ones we think are most relevant and put all the others under a namespace and just say it's object type and mapping will be done dynamically? |
ruflin
force-pushed
the
kibana-filebeat-module
branch
2 times, most recently
from
7031c13 to
1f4889f
Compare
ruflin
changed the title
ruflin
added
review
and removed
in progress
|
PR updated with proposed solution for json handling (see PR message). Ready for review. |
ruflin
force-pushed
the
kibana-filebeat-module
branch
from
bb090b7 to
6835c40
Compare
ruflin
force-pushed
the
kibana-filebeat-module
branch
from
6835c40 to
273305f
Compare
filebeat/filebeat.reference.yml
Outdated
| @@ -161,6 +161,16 @@ filebeat.modules: | |||
| #var.paths: | |||
|
|
|||
|
|
|||
| #------------------------------- kibana Module ------------------------------- | |||
| - module: {{ module }} | |||
There was a problem hiding this comment.
Hmmm... This seems to be a problem of the generator. Do you mind changing it manually now. I am opening a PR with the fix in the meantime.
| @@ -0,0 +1,8 @@ | |||
| - module: {{ module }} | |||
There was a problem hiding this comment.
Also, this file is problematic.
| @@ -0,0 +1,8 @@ | |||
| - module: {{ module }} | |||
|
@kvch Fixed. Glad you spotted this. |
This adds the Kibana module to Filebeat reading the Kibana log. The Kibana log is in JSON format which simplifies reading it but at the same time it provides many fields which are not necessarily know in advance. To solve this issue The data is put under `kibana.log.meta` and the most common known fields are picked into `kibana.log.*` or directly put into the global field like `message`. The fields under `meta` are stored as keyword. This make sure all the meta information around the event is still in the index but prevents potential type conflicts like long vs double. Additional change: * Module tests always overwrite the pipeline. This should make development easier.
ruflin
force-pushed
the
kibana-filebeat-module
branch
from
3a1a4c7 to
9ba1c16
Compare
|
@kvch I made 2 additional small changes: Added a beta label to the docs and adjusted the path to read logs from. |
| @@ -0,0 +1,63 @@ | |||
| { | |||
| "description": "Pipeline for parsing Kibana log logs", | |||
There was a problem hiding this comment.
Sorry for spotting it quite late, but could you please remove "log" from the description.
|
I tested it locally, works like a charm. After that minor problem is fixed, it's ready to be merged. |
|
Would it be possible to add at least a simple Kibana dashboard? So far all FB modules come with at least a dashboard, and I'd like to keep it that way :) |
|
Added Dashboards to the meta issue here: #7036 |
* Filebeat Kibana module with log fileset This adds the Kibana module to Filebeat reading the Kibana log. The Kibana log is in JSON format which simplifies reading it but at the same time it provides many fields which are not necessarily know in advance. To solve this issue The data is put under `kibana.log.meta` and the most common known fields are picked into `kibana.log.*` or directly put into the global field like `message`. The fields under `meta` are stored as keyword. This make sure all the meta information around the event is still in the index but prevents potential type conflicts like long vs double. Additional change: * Module tests always overwrite the pipeline. This should make development easier. * add beta label * Fix review comment for pipeline
* Filebeat Kibana module with log fileset This adds the Kibana module to Filebeat reading the Kibana log. The Kibana log is in JSON format which simplifies reading it but at the same time it provides many fields which are not necessarily know in advance. To solve this issue The data is put under `kibana.log.meta` and the most common known fields are picked into `kibana.log.*` or directly put into the global field like `message`. The fields under `meta` are stored as keyword. This make sure all the meta information around the event is still in the index but prevents potential type conflicts like long vs double. Additional change: * Module tests always overwrite the pipeline. This should make development easier. * add beta label * Fix review comment for pipeline
This adds the Kibana module to Filebeat reading the Kibana log. The Kibana log is in JSON format which simplifies reading it but at the same time it provides many fields which are not necessarily know in advance. To solve this issue The data is put under
kibana.log.metaand the most common known fields are picked intokibana.log.*or directly put into the global field likemessage. The fields undermetaare stored as keyword. This make sure all the meta information around the event is still in the index but prevents potential type conflicts like long vs double.Additional change: