chore: tweaks to handling of pending commits #1368
Open
+23
−40
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a combination of two related changes to the handling of pending commits during rebalance operations in
KafkaConsumerActor
.Single definition of pending commit effect
Whether immediately committing offsets, or queueing the request in
State.pendingCommits
, the necessary effect is built at once.This avoids the need to repeat similar logic in disparate call sites, and makes visible in a single place how the request will be processed and logged.
Ensure pending commits order
Commits would be added to the chain of pending commits based only on the rebalancing state. Given that the rebalancing state is updated (via
ConsumerRebalanceListener.onPartitionsAssigned
) separate from when pending commits are processed (afterpoll
), it could happen that commits emitted later would be processed before earlier pending ones.This updates the condition for queueing commits to take into account the prior existence of pending commits.
In addition, the condition for processing pending commits in
poll
is also updated to disregard whether a rebalance operation was ongoing at the start of the poll. Instead, the existence of pending commits along with a non-rebalancing
state are a sufficient trigger.This ensures that rebalance operations that might conclude within a single consumer poll do not leave behind any pending commits.
At the moment, these possibilities are theoretical as commit operations are serialized via
KafkaConsumerActor
's request queue, and don't happen concurrently to polls. That said, the cost of the fixes is trivial and being explicit about the conditions may prevent future bugs, if the surrounding context changes.