regression: enabling rackawareness causes severe throughput drops

[lizthegrey]

Versions

Sarama	Kafka	Go
v1.29.0	v3.0.0 (Confluent 7)	v1.17.1

Regression has been bisected to 1aac8e5

See #1927 (comment) for another report than just mine.

Configuration

Pertinent config variables:

	c.conf = conf
	if (c.LD != nil && c.LD.BoolVariationCtx(
		ctx,
		launchdarkly.FlagKafkaRackAwareFollowerFetch,
		types.UserAndTeam{User: nil, Team: nil})) {
		c.rackID = os.Getenv("AZ")
	}

broker.rack=<%= node["ec2"]["availability_zone"] %>
replica.selector.class=<%= node["kafka"]["replica"]["selector"]["class"] %>

default['kafka']['replica']['selector']['class'] = "org.apache.kafka.common.replica.RackAwareReplicaSelector"

Problem Description

When AZ is not populated (due to a bug on our side, hah), or when FlagKafkaRackAwareFollowerFetch is false, things behave normally. But throughput drops by 75% if the flag is set and the Sarama library is at or past 1aac... https://share.getcloudapp.com/nOu54vNq
There is a corresponding lag in timestamps between producer and consumer: https://share.getcloudapp.com/DOu6LBAQ

[lizthegrey]

@dnwe opened as per your request

[dnwe]

Thank you!

[lizthegrey]

anything I can do to assist here? rolling back the commit might help, but also would break that fix so...

[dnwe]

@lizthegrey apologies, I've actually been off sick this week so hadn't had a chance to code up a fix for this yet, but I do aim to look at it soon.

Essentially I believe the issue is that Kafka only ever computes a "preferred read replica" when your FetchRequest has gone to the leader of the partition and you've provided a client RackID. In that case the FetchResponse contains the preferred replica (if it differs from the leader) and omits any data, then a well behaved client should disconnect and start fetching from that preferred replica instead. However, when you then send FetchRequests to the follower, the "preferred read replica" field is omitted from its responses and hence #1936 kicks in because it thinks no preference exists and you're consuming from a non-leader, so it forces the consumer back to the real leader. Hence the massive drop in throughput you see as the consumer is flip flopping FetchResponses

[dnwe]

@lizthegrey so I've pushed up #2076 as a tentative fix, but (full disclaimer) I haven't yet actually tried this out against a real cluster so I'll need to run through with some of that and then code up some test cases too before I can merge, but if you'd like to try it out in the meantime that would be really helpful :)

[lizthegrey]

We have test clusters for this exact reason. I will flip on and let you know.

[lizthegrey]

Fixed by #2076 and confirmed working in Honeycomb production.

[mtj075]

@lizthegrey apologies, I've actually been off sick this week so hadn't had a chance to code up a fix for this yet, but I do aim to look at it soon.

Essentially I believe the issue is that Kafka only ever computes a "preferred read replica" when your FetchRequest has gone to the leader of the partition and you've provided a client RackID. In that case the FetchResponse contains the preferred replica (if it differs from the leader) and omits any data, then a well behaved client should disconnect and start fetching from that preferred replica instead. However, when you then send FetchRequests to the follower, the "preferred read replica" field is omitted from its responses and hence #1936 kicks in because it thinks no preference exists and you're consuming from a non-leader, so it forces the consumer back to the real leader. Hence the massive drop in throughput you see as the consumer is flip flopping FetchResponses

sorry, i think this sarama's code change made the consumer client ignore kafka's leader replica(preferred replica ) .Once the isr change ,the consumer client can not fetch from leader replica. kafka had fix preferred replica from isr. https://github.com/apache/kafka/pull/12877/files#diff-78812e247ffeae6f8c49b1b22506434701b1e1bafe7f92ef8f8708059e292bf0