Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus metrics are merged when they differ by trimmable suffixes #6931

Closed
dashpole opened this issue Dec 21, 2021 · 1 comment · Fixed by #6932
Closed

Prometheus metrics are merged when they differ by trimmable suffixes #6931

dashpole opened this issue Dec 21, 2021 · 1 comment · Fixed by #6932
Labels
bug Something isn't working comp:prometheus Prometheus related issues

Comments

@dashpole
Copy link
Contributor

Describe the bug
The prometheus receiver is renaming metrics if they differ from another metric by a trimmable suffix, and associating them with the wrong metric metadata.

Steps to reproduce

With the metrics:

# HELP my_metric_total A counter metric
# TYPE my_metric_total counter
my_metric_total 0.5
# HELP my_metric A gauge metric
# TYPE my_metric gauge
my_metric 2

The my_metric_total metric is renamed to my_metric, and the gauge metadata is used. This is because with the current logic, we checked for a trimmed metric name before we check for the untrimmed metric name. So if a metric exists with the metadata for my_metric, the my_metric_total metric will use that if it exists.

This results in a single metric named my_metric, which is a gauge, and has values 0.5 and 2.

What did you expect to see?

I expect to see both the my_metric and my_metric_total with correct metadata, and correct values.

What version did you use?
Version: HEAD

Additional context

In #4865, we handled the case in which the metric python_gc_objects_collected_total had metadata under python_gc_objects_collected, which was supposed to be how the python client wrote counter metrics.

However, after experimenting with the python client, I found that to be incorrect:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 167.0
python_gc_objects_collected_total{generation="1"} 174.0
python_gc_objects_collected_total{generation="2"} 0.0

Python does attach _total to the metric, but it does not send metadata without the _total suffix. There is a test case that tests for that, and should be removed or reworked.

@dashpole dashpole added the bug Something isn't working label Dec 21, 2021
@dashpole
Copy link
Contributor Author

cc @gracewehner @Aneurysm9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working comp:prometheus Prometheus related issues
Projects
None yet
2 participants