Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MetadataRequest V5 has wrong encoding for "all topics" #1352

Closed
dnwe opened this issue Apr 10, 2019 · 1 comment · Fixed by #1353
Closed

MetadataRequest V5 has wrong encoding for "all topics" #1352

dnwe opened this issue Apr 10, 2019 · 1 comment · Fixed by #1353

Comments

@dnwe
Copy link
Collaborator

dnwe commented Apr 10, 2019

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: 2b82435
Kafka Version: 1.1.0
Go Version: 1.11.8

Configuration

What configuration values are you using for Sarama and Kafka?

	cfg := sarama.NewConfig()
	cfg.Net.TLS.Enable = true
	cfg.Version = sarama.V1_1_0_0
Logs
[Sarama] 2019/04/10 22:08:13 Initializing new client
[Sarama] 2019/04/10 22:08:13 client/metadata fetching metadata for all topics from broker kafka-2:9092
[Sarama] 2019/04/10 22:08:19 Connected to broker at kafka-2:9092 (unregistered)
[Sarama] 2019/04/10 22:08:19 client/brokers registered new broker #2 at kafka-2:9092
[Sarama] 2019/04/10 22:08:19 client/brokers registered new broker #1 at kafka-1:9092
len(topics) received 0 topics in all metadata request
Problem Description

TL;DR if the client config version is at least 1.0.0.0 then the "full" metadata refresh always returns an empty topic list

Underlying cause

Since e3ecf7b the tryRefreshMetadata for a "full" metadata request now appears to use the V5 MetadataRequest if config version is at least 1.0.0.0

https://github.com/Shopify/sarama/blob/2b824357033b9904e3a32bf7fae2efe76f1591fd/client.go#L761-L765

However, this commit appeared to also add an additional if statement to the encoding of the request to send an empty array rather than a null if the provided topics list was an empty slice

https://github.com/Shopify/sarama/blob/2b824357033b9904e3a32bf7fae2efe76f1591fd/metadata_request.go#L13

I'm not sure why this was done as consulting the protocol guide the V5 request uses the same nullable array for topics as the V1-V4 requests.

Metadata Request (Version: 5) => [topics] allow_auto_topic_creation
topics => STRING
allow_auto_topic_creation => BOOLEAN

Field Description
topics An array of topics to fetch metadata for. If the topics array is null fetch metadata for all topics.
allow_auto_topic_creation If this and the broker config auto.create.topics.enable are true, topics that don't exist will be created by the broker. Otherwise, no topics will be created by the broker.
@dnwe
Copy link
Collaborator Author

dnwe commented Apr 10, 2019

Unfortunately this bug is included in the just-released v1.22.0 milestone

@bai bai closed this as completed in #1353 Apr 12, 2019
squ1d123 added a commit to squ1d123/kafka_exporter that referenced this issue Apr 14, 2019
imduffy15 added a commit to imduffy15/kafka_exporter that referenced this issue Aug 27, 2020
* upstream/master:
  Add Helm Chart
  updated SCRAM to use generator function in the case of creating multiple connections
  using latest commit on master for samara as it contains <IBM/sarama#1352> fix
  changed mechanism type to have scram prefix
  updaed to use go modules
  Support for scram sasl mechanisms
imriz pushed a commit to myheritage/kafka_exporter that referenced this issue Mar 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant