Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache the topics metadata for some time to reduce load on the cluster #74

Open
jorgelbg opened this issue Nov 13, 2018 · 0 comments
Open

Comments

@jorgelbg
Copy link
Contributor

When the exporter is connected to a big cluster each call to the RefreshMetadata() method is expensive because Kafka needs to pull data from the entire cluster. Since the metadata info doesn't change very often it would be helpful to be able to control this with a refresh interval.

This means that we could keep a low scrape interval on the Prometheus side for the lag/offsets and just query the metadata every once in a while. The specific configuration would be dependant on how often a new topic/partition is added/changed in the cluster.

We could use a custom flag like --refresh.metadata with a default value of 30s. Still, the metrics would be accurate because we don't cache the lag/offsets, only the metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant