Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes around service discovery, engine and consumer. #327

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

FlorentinDUBOIS
Copy link
Collaborator

  • The first fix is around service discovery to not create lookup topic request amplification when we are able to determine if the topic is already a partition.
  • The second fix is all about lookup request as well as there are performed systematically even though we aren't running a consumer with topic discovery based on a regex.
  • The third fix is a tricky one that hit us in production at Clever Cloud, where our consumers have an issue with batched messages that create an underflow on the remaining_messages variable. The symptom was that consumer stop receiving messages, because they do not send any flow control message due to the invalid remaining message number. We also fix a bug that create a reconnect loop, because the driver consider that the newly created connection was already disconnected due to same origin.

Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
@FlorentinDUBOIS
Copy link
Collaborator Author

The dashboard that I use to check if the fix works :D We are able to treat batch messages!

image

Copy link
Collaborator

@CleverAkanoa CleverAkanoa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions regarding hazardous casting

src/consumer/engine.rs Show resolved Hide resolved
src/consumer/engine.rs Outdated Show resolved Hide resolved
src/consumer/engine.rs Show resolved Hide resolved
src/service_discovery.rs Show resolved Hide resolved
FlorentinDUBOIS and others added 7 commits December 19, 2024 13:26
…d topic

Co-authored-by: @Wonshtrum <eloi.demolis@clever-cloud.com>
Co-authored-by: @KannarFr <alexandre.duval@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
…egex for topic discovery

Co-authored-by: @Wonshtrum <eloi.demolis@clever-cloud.com>
Co-authored-by: @KannarFr <alexandre.duval@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
…ssage

Co-authored-by: @Wonshtrum <eloi.demolis@clever-cloud.com>
Co-authored-by: @KannarFr <alexandre.duval@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Signed-off-by: Florentin Dubois <florentin.dubois@clever-cloud.com>
Copy link
Collaborator

@CleverAkanoa CleverAkanoa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants