kafka consumer plugin issue

[panda87]

Hi,
As part of my tests Im using Telegraf to send metrics to Kafka.
From Kafka I plan to consume the metrics to InfluxDB.

As part of my tests I tried to use the Kafka (consumer) plugin, but the problem is that the plugin doesn't consume any logs from the relevant topic, unfortunately I can't see any problem. (in the zk I can see it consumed 20K logs but then it stopped without send them to Influx)

My Influxdb ver is 0.9.4.2 and telegraf 0.2.
All the metrics are line since they came from Telegraf.

Maybe someone else encountered this problem?

[sparrc]

I can't recreate, can you provide your configuration on both sides of kafka?

Note that your Kafka topic of the output and input plugins must match. Currently the default topics do not match, which we should probably fix.

[panda87]

What do you mean configuration on both sides of kafka?
The kafka configurations or telegraf output + kafka consumer plugins?

I manually created the topic and also watched the offset changed in zk, but its stopped after 20K logs of 10M. but anyway I didn't see anything sent to influx.

I also tried to send the logs to another kafka topic but again, it didn't create the new topic.

[sparrc]

yes, please show your configuration, for output and consumer plugins.

I am now also seeing problems with the consumer plugin, but I haven't yet found out the problem

[sparrc]

@panda87 forget about the configuration. As far as I can tell, the kafka consumer plugin can only anything on the 1st run. I don't think it's ever been a functional plugin 😭

I'm going to need to significantly refactor it to get it working, probably making it a ServicePlugin, I'll try to do that today

[panda87]

Thanks @sparrc!
If you are planning to refactor this plugin, can I ask you to add another parameters in order to fit this plugin for more use cases.

1. Offset management - manage what is the starting offset for consuming (last offset or manual offset number).
2. Partitions - let us choose the number of partitions in order to fit the amount of partitions of the topic. Since if the topic has more than more than 1 partition, this will take much more time to consume the logs from.

Last thing, the refactor will include the ability to add more than one topic to consume from (multiple plugins like the new multiple outputs)?

Thanks a lot.
D.

[sparrc]

@panda87 I don't know if I have the kafka expertise to implement these features at the moment. I see references to these in the documentation of the consumer library we're using (https://godoc.org/github.com/wvanbergen/kafka/consumergroup), but I don't quite see how we should be utilizing them in telegraf.

I can, however, have it support multiple topics