Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support per message unacknowledged redelivery on RedeliverUnAcknowled… #70

Merged
merged 1 commit into from
Oct 17, 2016

Conversation

merlimat
Copy link
Contributor

Rebased from @sschepens #51

Motivation

Pulsar should be more robust in terms of which messages it redelivers when there are unacked messages, right now Pulsar redelivers ALL currently unacked messages, this increases A LOT the chance to get duplicate messages.

Modifications

Protobuf:

  • RedeliverUnacknowledgedMessages now carries an optional list of MessageIdData specifying the messages to be redelivered.

Broker:

  • Now supports receiving a MessageIds on redelivery and adds them to the replay list, only if the could be atomically removed from the unacked message list (maybe we're lucky and someone acked or redelivered the message before we try).

Client:

  • Extracted UnAckedMessageTracker to an interface and renamed current implementation to SimpleUnAckedMessageTracker. We probably want to also make it configurable so that a user could tweak the behavior and have full control (such as more precise time tracking).
  • ConsumerConfiguration now allows for enabling per message redelivery and sepcifying how big a batch can be before asking the Broker to redeliver all unacked messages.
  • SimpleUnAckedMessageTracker now always sends MessageIds to redelivery and Consumer determines whether it should use per message redelivery or full redelivery.
  • PartitionedConsumer calls per message redelivery on its internal clients by splitting MessageIds by partition.

Result

Change should theoretically be backwards-compatible because Pulsar uses Protobuf.
Users can chose to prevent message duplicates on Redelivery be incrementing network traffic to Broker (sending message ids).

This is a first try at this implementation, please provide feedback!
I would also need a little guidance with tests, if I run mvn test now it hangs on some strange BookKeeper tests on project managed-ledger. I would also like some opinion on where to write tests.

Cheers!

@yahoocla
Copy link

CLA is valid!

@merlimat merlimat added this to the 1.15 milestone Oct 17, 2016
@merlimat merlimat added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Oct 17, 2016
@merlimat merlimat merged commit e274a24 into apache:master Oct 17, 2016
@merlimat merlimat deleted the per_message_redelivery branch October 18, 2016 17:38
sijie pushed a commit to sijie/pulsar that referenced this pull request Mar 4, 2018
* add client side class validation

* adding and fixing

* cleaning up
hrsakai pushed a commit to hrsakai/pulsar that referenced this pull request Dec 10, 2020
Change-Id: I480f5b8bb580887543cde66824b2f5704f8fa6e8

- use goimports import lib
- use switch-case instrad of if-else for newClient in pulsar/impl_client.go
hangc0276 pushed a commit to hangc0276/pulsar that referenced this pull request May 26, 2021
* [CI] Build and publish integration test images

*Motivation*

Build and publish integration test images to avoid building images in integration tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants