Reading last produced committed offset

[mxk1235]

I currently read low and high watermark offsets for each topic partition, and then consume exact ranges. However, sometimes the messages are not yet committed by the producer, so I'd like to add logic to stop short of reading that message.

@edenhill Is there a way to do that? essentially i would like to read the offset of earliest uncommitted and un-aborted transaction by the producer. i was reading rd_kafka_committed(), but I came to the realization that maybe it has to do with consumer group offsets, rather than producer.

[edenhill]

What you describe is the default behaviour of librdkafka (isolation.level=read_committed), a consumer with this set (which is the default) will only see messages up to the LSO - last stable offset.
You will need librdkafka v1.2.0 or later.

[mxk1235]

Right, it's just that we would like to know the offsets before calling consume(), so that we don't schedule batches that will either 1) wait for messages to commit, or 2) surface EOF when we run ahead of the number.