Adds Kafka Plugin #35
Conversation
|
Would it be possible to update this to use the line protocol instead? The InfluxDB client supports it and has methods for parsing things out. More on the line protocol here: |
If I'm calling into the client, would it still make sense to have this as a plugin in telegraf? |
|
Telegraf should still be responsible for posting the metrics to InfluxDB. I was just mentioning the client because it can be used to parse the line protocol. But actually I was wrong, it's in the |
|
👍 Any recommendations on the optimal batch size? |
|
@emils best to have it a configuration option in the plugin. Maybe default to a size of 2k points? |
es
force-pushed
the
plugins/kafka
branch
3 times, most recently
from
8adc828 to
9fbcdde
Compare
|
Updated to use the line protocol instead of JSON. @pauldix could you please take a second 👀? I'll make sure to squash the commits once you're happy with the changes. |
|
@emils Looks good. The only last thing I'd do is to rename the plugin from |
es
force-pushed
the
plugins/kafka
branch
4 times, most recently
from
bf49c02 to
009dd53
Compare
|
Squashed the commits & renamed the plugin to |
| zookeeperPeers = ["localhost:2181"] | ||
|
|
||
| # Batch size of points sent to InfluxDB | ||
| batchSize = 10` |
There was a problem hiding this comment.
Sorry, one last comment. I think the default batch size should be 1000. InfluxDB performance will be much better that way.
The Kafka consumer plugin polls a specified Kafka topic and adds messages to InfluxDB. The plugin assumes messages follow the line protocol. Consumer Group is used to talk to the Kafka cluster so multiple instances of telegraf can read from the same topic in parallel.
|
Looks great, thanks @emils! |
Kafka plugin polls a specified Kafka topic and adds messages to InfluxDB. The plugin assumes messages in the topic are JSON objects with 'measurement', 'tags', 'time', & 'values' keys. Consumer Group is used to talk to the Kafka cluster so multiple instances of telegraf can read from the same topic in parallel.
Running the test requires running Zookeeper & Kafka. The spotify/kafka docker container works pretty well for this.
@evanphx
\cc @marc-barry