Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Add Kafka JMX metricsets #14330

Merged
merged 33 commits into from
Nov 20, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
12b3c58
Add kafka borker light module
ChrsMark Oct 30, 2019
cc25ce9
Add docker files
ChrsMark Oct 30, 2019
84ceb2a
fix config files
ChrsMark Oct 30, 2019
e9a26b0
Add producer tasks in docker env
ChrsMark Oct 31, 2019
10de02c
Add producer metricset
ChrsMark Oct 31, 2019
0f17f51
Add cosumer task in docker env
ChrsMark Oct 31, 2019
6f24a77
Add consumer metricset
ChrsMark Oct 31, 2019
e6bc8db
Fix back run script
ChrsMark Oct 31, 2019
5ea1c60
Add fields
ChrsMark Nov 1, 2019
c2da3a3
Add docs
ChrsMark Nov 1, 2019
e98ee2f
Move kafka jmx to oss
ChrsMark Nov 12, 2019
9426d3f
Add go files
ChrsMark Nov 12, 2019
6febb14
Merge remote-tracking branch 'upstream/master' into add_kafka_broker_jmx
ChrsMark Nov 15, 2019
7006b14
Remove go files and update docs and list.go
ChrsMark Nov 15, 2019
3543af0
Add extra docs
ChrsMark Nov 15, 2019
100b68c
Address review comments
ChrsMark Nov 15, 2019
923bb3d
Add changelog entry
ChrsMark Nov 15, 2019
223cc43
Merge remote-tracking branch 'upstream/master' into add_kafka_broker_jmx
ChrsMark Nov 15, 2019
e749669
Merge remote-tracking branch 'upstream/master' into add_kafka_broker_jmx
ChrsMark Nov 18, 2019
77d65fb
Remove congig.reference file
ChrsMark Nov 18, 2019
a427180
Define port in Kafka integration tests
ChrsMark Nov 18, 2019
045bac1
fix remaining ports
ChrsMark Nov 18, 2019
1b5b1a5
Fix healthcheck script in Kafka container
ChrsMark Nov 18, 2019
e251525
Comment out extra jolokia exposed ports
ChrsMark Nov 19, 2019
6d4971b
Add less aggressive kafka producer
ChrsMark Nov 19, 2019
e3a02e5
Merge remote-tracking branch 'upstream/master' into add_kafka_broker_jmx
ChrsMark Nov 20, 2019
ffcbaa6
Add back jolokia ports in docker
ChrsMark Nov 20, 2019
fbba3dd
Fix config ports
ChrsMark Nov 20, 2019
2dfa5b8
Increase number of retries in healthcheck
ChrsMark Nov 20, 2019
1209c62
Update docker image version
ChrsMark Nov 20, 2019
10f982d
Update x-pack configs
ChrsMark Nov 20, 2019
2e5437b
Increase tests' timeout
ChrsMark Nov 20, 2019
54b58c2
Fix fields' type
ChrsMark Nov 20, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add consumer metricset
Signed-off-by: chrismark <chrismarkou92@gmail.com>
  • Loading branch information
ChrsMark committed Nov 12, 2019
commit 6f24a772e5d3f1bf8db5027be9a6b7fb809b45eb
2 changes: 1 addition & 1 deletion x-pack/metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,7 @@ metricbeat.modules:

#-------------------------------- Kafka Module --------------------------------
- module: kafka
metricsets: ["broker", "consumergroup", "partition", "producer"]
metricsets: ["broker", "consumer", "consumergroup", "partition", "producer"]
period: 10s
hosts: ["localhost:8779"]
enabled: true
Expand Down
2 changes: 1 addition & 1 deletion x-pack/metricbeat/module/kafka/_meta/config.reference.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- module: kafka
metricsets: ["broker", "consumergroup", "partition", "producer"]
metricsets: ["broker", "consumer", "consumergroup", "partition", "producer"]
period: 10s
hosts: ["localhost:8779"]
enabled: true
Expand Down
2 changes: 2 additions & 0 deletions x-pack/metricbeat/module/kafka/_meta/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
#metricsets:
# - broker
# - partition
# - consumer
# - consumergroup
# - producer

period: 10s
hosts: ["localhost:8779"]
#client_id: metricbeat
Expand Down
53 changes: 53 additions & 0 deletions x-pack/metricbeat/module/kafka/consumer/data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"@timestamp": "2019-10-31T13:22:06.700Z",
"@metadata": {
"beat": "metricbeat",
"type": "_doc",
"version": "8.0.0"
},
"kafka": {
"consumer": {
"bytes_consumed": 2.9521300228e+10,
"fetch_rate": 0,
"records_consumed": 1.23075656e+08,
"mbean": "kafka.consumer:client-id=consumer-1,type=consumer-fetch-manager-metrics"
}
},
"event": {
"dataset": "kafka.consumer",
"module": "kafka",
"duration": 7042831
},
"ecs": {
"version": "1.2.0"
},
"host": {
"name": "pr.local",
"hostname": "pr.local",
"architecture": "x86_64",
"os": {
"kernel": "18.7.0",
"build": "18G95",
"platform": "darwin",
"version": "10.14.6",
"family": "darwin",
"name": "Mac OS X"
},
"id": "883134FF-0EC4-5E1B-9F9E-FD06FB681D84"
},
"agent": {
"hostname": "pr.local",
"id": "79dd1677-1bea-4efd-9131-e8ca464eddf0",
"version": "8.0.0",
"type": "metricbeat",
"ephemeral_id": "e40f5843-d3aa-4bdc-a100-64022b70851b"
},
"metricset": {
"name": "consumer",
"period": 10000
},
"service": {
"address": "localhost:8774",
"type": "kafka"
}
}
21 changes: 21 additions & 0 deletions x-pack/metricbeat/module/kafka/consumer/manifest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
default: true
input:
module: jolokia
metricset: jmx
defaults:
namespace: "consumer"
hosts: ["localhost:8774"]
path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
jmx.mappings:
- mbean: 'kafka.consumer:client-id=*,type=consumer-fetch-manager-metrics'
attributes:
- attr: fetch-rate
field: fetch_rate
- attr: bytes-consumed-total
field: bytes_consumed
- attr: records-consumed-total
field: records_consumed
- mbean: 'kafka.consumer:client-id=*,type=consumer-metrics'
attributes:
- attr: incoming-byte-total
field: bytes_in
1 change: 1 addition & 0 deletions x-pack/metricbeat/module/kafka/module.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ name: kafka
metricsets:
- broker
- producer
- consumer
2 changes: 2 additions & 0 deletions x-pack/metricbeat/modules.d/kafka.yml.disabled
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@
#metricsets:
# - broker
# - partition
# - consumer
# - consumergroup
# - producer

period: 10s
hosts: ["localhost:8779"]
#client_id: metricbeat
Expand Down