Consumer Kafka

The "kafka" consumer module connects to a single Kafka cluster and consumes the __consumer_offsets topic in the cluster. This is the special internal topic where all consumer offsets and group metadata are committed. This information is then sent to the Storage subsystem for use in group status evaluations. This retrieves information on all "new consumer" clients, as well as older consumers that have switched over to Kafka-committed offsets.

Configuration

The subheading must be a unique name, such as [consumer.myconsumers]. The "myconsumers" part is the name that will be used when logging.

[consumer.myconsumers]
class-name="kafka"
cluster="myclustername"
servers=[ "kafka01.example.com:10251", "kafka02.example.com:10251", "kafka03.example.com:10251" ]
client-profile="profile-name"
offsets-topic="__consumer_offsets"
start-latest=true
group-allowlist=".*"
group-denylist="^not-this-group$"

Key	Value Type	Required	Default Value	Description
class-name	string	yes	(empty)	This is the name of the consumer module type. For this module, it must be "kafka"
cluster	string	yes	(empty)	This is the name of the cluster to which the consumers in this module belong.
servers	list of hostname:port	yes	(none)	A list of strings in the form "hostname:port" that point to the servers in the Kafka cluster. At least one is required.
client-profile	string	no	(empty)	The name of a configured client-profile section. If no profile is specified, a default configuration for the client will be used.
offsets-topic	string	no	__consumer_offsets	The name of the internal topic used for consumer offsets.
start-latest	boolean	no	false	If true, start consuming the offsets topic from the last (latest) offset. Otherwise, start from the earliest offset.
group-allowlist	string (regex)	no	(empty)	If specified, only send offsets for groups that match this regular expression.
group-denylist	string (regex)	no	(empty)	If specified, only send offsets for groups that DO NOT match this regular expression. This is processed after the allowlist (if specified).