metrics, visualization: StatsD Exporter, Prometheus Rules & Grafana Dashboard #939
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- we use prometheus as source - we also make use of simple [grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
nosahama
changed the title
nosahama
changed the title
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||
|
Waiting a little bit if someone else want to take a look, then gonna merge it |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
This PR adds a
statsd-exporterto the service, which will translate existing statsd metrics in influxdb style to prometheus metrics. We also add a simple grafana dashboard to Karapace, thus we can visualize the various metrics.We also now take the statsd host and port values from the config/env, rather than being hard coded within the client.
Sample prometheus
rulesfile was added and a calculated metric is used within the dashboard. Spin up the container, send some requests and see the metrics on grafana and see the firing alerts on the prometheus alert manager.Alerts firing from the prometheus
rulesfile, added with a low threshold so we see it fire:This is not final, the recorded metrics could use better promql techniques, this is just a base for any future work and at least we visualize what already exists in the service, i.e. the Schema Reader panel uses the exported statsd metrics and the HTTP requests use the added prometheus metrics, thus we have 2 metric handlers within the service and should migrate to one in the long run.
A good question would be how and where will karapace send it's metrics while running within the Aiven environment.