feat: expose rackID option in ingester samara client #3395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Support for KIP-392 in Sarama consumer client This will allow ingester consumers to fetch from the closest replica. On branch add-kafka-consumer-client-rack Changes to be committed: modified: cmd/ingester/app/flags.go modified: cmd/ingester/app/flags_test.go modified: pkg/kafka/consumer/config.go Signed-off-by: Shai Moria <shai.moria@zooz.com> Signed-off-by: shaimoria <shai.moria@zooz.com>
shyimo
force-pushed
the
add-kafka-consumer-client-rack
branch
from
c4ad44d to
3bc0a3e
Compare
Codecov Report
@@ Coverage Diff @@
## master #3395 +/- ##
==========================================
- Coverage 96.48% 96.47% -0.02%
==========================================
Files 260 260
Lines 15215 15220 +5
==========================================
+ Hits 14680 14683 +3
- Misses 452 453 +1
- Partials 83 84 +1
Continue to review full report at Codecov.
|
yurishkuro
reviewed
Nov 16, 2021
cmd/ingester/app/flags.go
Outdated
| flagSet.String( | ||
| KafkaConsumerConfigPrefix+SuffixRackID, | ||
| "", | ||
| "Configure the 'rack' in which the consumer resides to enable - must be supported by kafka server") |
There was a problem hiding this comment.
Hi @yurishkuro.
i update the description
|
@yurishkuro add explanation for the use case |
set better description for 'rack-id' cli flag Changes to be committed: modified: cmd/ingester/app/flags.go Signed-off-by: shaimoria <shai.moria@zooz.com>
shyimo
force-pushed
the
add-kafka-consumer-client-rack
branch
from
2fc220d to
824ca28
Compare
yurishkuro
approved these changes
Nov 18, 2021
Merge branch 'master' into add-kafka-consumer-client-rack
…hyimo/jaeger into add-kafka-consumer-client-rack
rbroggi
pushed a commit
to rbroggi/jaeger
that referenced
this pull request
Nov 22, 2021
4 tasks
4 tasks
yurishkuro
pushed a commit
that referenced
this pull request
Apr 20, 2024
## Which problem is this PR solving? RackID option was exposed in [this](#3395) PR but it was not properly parsed from passed flags and set on Sarama client ## Description of the changes - configures Parsed RackID on ConsumerConfig - Fix failing lint issues ## How was this change tested? - Tested by printing the config passed to sarama client ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` Signed-off-by: Saketh kappala <43525626+sappusaketh@users.noreply.github.com>
varshith257
pushed a commit
to varshith257/jaeger
that referenced
this pull request
May 3, 2024
## Which problem is this PR solving? RackID option was exposed in [this](jaegertracing#3395) PR but it was not properly parsed from passed flags and set on Sarama client ## Description of the changes - configures Parsed RackID on ConsumerConfig - Fix failing lint issues ## How was this change tested? - Tested by printing the config passed to sarama client ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` Signed-off-by: Saketh kappala <43525626+sappusaketh@users.noreply.github.com> Signed-off-by: Vamshi Maskuri <gwcchintu@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: shaimoria shai.moria@zooz.com
Which problem is this PR solving?
It is common to have a Kafka cluster spanning multiple datacenters. For example, a common deployment is within an AWS region in which each availability zone is treated as a datacenter. In order to reduce cross region/rack network cost, we can tell the consumer which rack he's located on. this way the consumer will fetch only from partitions that are located on he's region/rack
Short description of the changes
kafka.consumer.rack-idcli argument in order to ingester consumers to fetch from closest replica