Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: tweaks to handling of pending commits #1368

Open
wants to merge 2 commits into
base: series/3.x
Choose a base branch
from

Conversation

biochimia
Copy link
Contributor

This is a combination of two related changes to the handling of pending commits during rebalance operations in KafkaConsumerActor.

Single definition of pending commit effect

Whether immediately committing offsets, or queueing the request in State.pendingCommits, the necessary effect is built at once.

This avoids the need to repeat similar logic in disparate call sites, and makes visible in a single place how the request will be processed and logged.

Ensure pending commits order

Commits would be added to the chain of pending commits based only on the rebalancing state. Given that the rebalancing state is updated (via ConsumerRebalanceListener.onPartitionsAssigned) separate from when pending commits are processed (after poll), it could happen that commits emitted later would be processed before earlier pending ones.

This updates the condition for queueing commits to take into account the prior existence of pending commits.

In addition, the condition for processing pending commits in poll is also updated to disregard whether a rebalance operation was ongoing at the start of the poll. Instead, the existence of pending commits along with a non-rebalancing state are a sufficient trigger.

This ensures that rebalance operations that might conclude within a single consumer poll do not leave behind any pending commits.

At the moment, these possibilities are theoretical as commit operations are serialized via KafkaConsumerActor's request queue, and don't happen concurrently to polls. That said, the cost of the fixes is trivial and being explicit about the conditions may prevent future bugs, if the surrounding context changes.

Whether immediately committing offsets, or queueing the request in
`State.pendingCommits`, the necessary effect is built at once.

This avoids the need to repeat similar logic in disparate call sites,
and makes visible in a single place how the request will be processed
and logged.
Commits would be added to the chain of pending commits based only on the
rebalancing state. Given that the rebalancing state is updated (via
`ConsumerRebalanceListener.onPartitionsAssigned`) separate from when
pending commits are processed (after `poll`), it could happen that
commits emitted later would be processed before earlier pending ones.

This updates the condition for queueing commits to take into account the
prior existence of pending commits.

In addition, the condition for processing pending commits in `poll` is
also updated to disregard whether a rebalance operation was ongoing at
the start of the poll. Instead, the existence of pending commits along
with a non-`rebalancing` state are a sufficient trigger.

This ensures that rebalance operations that might conclude within a
single consumer poll do not leave behind any pending commits.

At the moment, these possibilities are theoretical as commit operations
are serialized via `KafkaConsumerActor`'s request queue, and don't
happen concurrently to polls. That said, the cost of the fixes is
trivial and being explicit about the conditions may prevent future bugs,
if the surrounding context changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant