New Filebeat module: Kafka logs #4885
Conversation
kvch
force-pushed
the
feature/filebeat-kafka-module-log
branch
from
450129a to
b17ed4b
Compare
| @@ -0,0 +1,14 @@ | |||
| module_version: 1.0 | |||
|
|
|||
| kafka.home: /opt/kafka[_-\.0-9]* | |||
There was a problem hiding this comment.
I think this could be made also a var, which can be referred from the paths array. This way, the user can only set that variable. Unless I missremember, the variables can refer to variables defined before them using the syntax {{.home}}.
|
Nice work so far, could you also add a screenshot of the Kibana dashboard? |
kvch
force-pushed
the
feature/filebeat-kafka-module-log
branch
from
19cfa17 to
a45d829
Compare
|
I added the Kibana dashboard along with some missing documentation. I also made And for the record the GC logs will come in a separate PR. So it can be merged, after it passes the review and the tests. |
| - /opt/kafka[_-\.0-9]*a | ||
| - name: paths | ||
| default: | ||
| - ${kafka_home}/logs/controller.log* |
There was a problem hiding this comment.
Does this work? I would have expected this to use go template syntax, so something like {{.kafka_home}}.
kvch
force-pushed
the
feature/filebeat-kafka-module-log
branch
from
a45d829 to
774248f
Compare
|
jenkins, retest it pls |
|
|
||
| # Set custom paths for Kafka. If left empty, | ||
| # Filebeat will look under /opt. | ||
| #var.kafka_home: |
There was a problem hiding this comment.
I think we can put the default in the commented line here, so it's clear for the user how to edit. I.e.:
#var.kafka_home: /opt/kafka*
Btw, it's /opt/kafka*, not just /opt, right?
| #var.kafka_home: | ||
|
|
||
| # Set custom paths for the log files. If left empty, | ||
| # Filebeat will choose the paths depending on your OS. |
There was a problem hiding this comment.
This comment is out-of-date now, I think. It should say it chooses the paths based on the kibana_path value.
|
There seems to be some issue with the Kibana dashboard:
I'm not sure if something changed on the Kibana side that broke our saving/loading of dashboards.. |
|
I see now that this also affects the Postgres dashboards, but not other dashboards. |
|
jenkins, test it |
kvch
force-pushed
the
feature/filebeat-kafka-module-log
branch
from
774248f to
2c57cb9
Compare
|
I updated the dashboards and now it is working. I will open a separate PR for the PostgreSQL dashboards. |
This module is based on https://www.elastic.co/blog/monitoring-kafka-with-elastic-stack-1-filebeat written by @urso.
This PR only includes the log fileset.