Stop at end of topic/partition

[arnaud-lb]

It doesn't seem to be possible to stop consuming when the end of a partition is reached.

I'm using compacted topics, and one common use-case I have is to start by reading them entirely from the beginning to the end. However, there is no easy way to stop consuming when the end is reached.

I tried a few things, but the library is not making that easy.

Basically, here is what I have to do currently:

1. Read the newest offset: client.GetOffset(topic, partition, sarama.OffsetNewest)
2. Consume every message
3. If a message with offset >= newest offset is seen, stop consuming
4. If topic is empty, 3 may never happen, so I have to also produce a dummy message before step 1, for instance

It would be awesome if Sarama provided a way to stop consuming when the end of a partition is reached and/or to be notified about it.

[eapache]

Similar to #666 and #628, this is not something we provide because kafka does not really have the concept; the entire design is based on the fact that streams are never-ending.

[arnaud-lb]

I believe that the Kafka protocol notifies the client when the end of the partition is reached ?

librdkafka and librdkafka-based clients also notify the user when the end of the partition is reached.

[eapache]

I believe that the Kafka protocol notifies the client when the end of the partition is reached ?

Not to my knowledge.

[wvanbergen]

It tells you the latest available offset while consuming (HighWaterMark): #339, which you could use for this purpose.

[TheRoSS]

Here is an answer #2116