use group assignment to fetch consumer offset instead of fetch all partitions

[iamgd67]

use group assignment to fetch consumer offset instead of fetch all partitions
to reduce request size & server cordinator log (by default, server will log all partition offset fetch request details)

[danielqsj]

thanks ! @iamgd67

[scholzj]

@iamgd67 @danielqsj It looks to me like this change caused that the Kafka Exporter now shows only lag for connected consumer groups. But when for example your client (all instances) dies and isn't connected for some time, it drops it and doesn't show the lag anymore. I assume it indeed reduces the request size as described. But it also reduces the functionality and impacts how useful the lag metrics are now. Was this change of behaviour intentional?

[iamgd67]

@scholzj
maybe it's better to make fetch all/assigned partitions an option.
on the other way, normally these metrics will be grabbed and saved by time serial data base like Prometheus, and history data may access there.

[scholzj]

on the other way, normally these metrics will be grabbed and saved by time serial data base like Prometheus, and history data may access there.

That is true, the last update will be stored normally. But when your app stops working, it becomes quickly outdated

[Kaali09]

Agree with @scholzj. When the consumer stuck with one partition and lag is getting increased then we will not come to know the same.
Also, when the consumer dead because of some error/issue, the lag keep on increase which will impact the pipeline.

Hence It is important to monitor the partition wise lag as well.

Request you enable the flag to get the partition level metrics and to get the consumer group lag metrics when the consumer is dead/not active state.

[iamgd67]

@scholzj @Kaali09 thanks for reporting the issue of not showing dead group's offset, I saw other people submit issues about this too, so create a pr(#255) to better handle dead/self-managed group's offset.