Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics clashing name #3940

Closed
M0NsTeRRR opened this issue Dec 27, 2020 · 7 comments · Fixed by #3944
Closed

Metrics clashing name #3940

M0NsTeRRR opened this issue Dec 27, 2020 · 7 comments · Fixed by #3944
Assignees
Milestone

Comments

@M0NsTeRRR
Copy link

I'm running exporter-exporter to expose caddy metric with TLS and cert authentication and it appears go_memstats_alloc_bytes
is in collision with an other metric

déc. 27 19:24:31 reverse-proxy exporter_exporter[43467]: time="2020-12-27T19:24:31+01:00" level=error msg="Verification for module 'caddy' failed: Failed to decode metrics from proxied server: text format parsing error in line 343: second HELP line for metric name \"go_memstats_alloc_bytes\""

The two metrics

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.058576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.4330096e+07

If I try to parse them with the prometheus client parser

$ python3 -c 'import sys; from prometheus_client.openmetrics import parser; list(parser.text_string_to_metric_families(sys.stdin.buffer.read().decode("utf-8")))'
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.058576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.4330096e+07

This is the output I get

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 24, in text_string_to_metric_families
    for metric_family in text_fd_to_metric_families(StringIO.StringIO(text)):
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 615, in text_fd_to_metric_families
    yield build_metric(name, documentation, typ, unit, samples)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 479, in build_metric
    raise ValueError("Clashing name: " + name + suffix)
ValueError: Clashing name: go_memstats_alloc_bytes_total

Openmetrics suffixes
https://github.com/prometheus/client_python/blob/f81891f57bf3d20a9bb87ea62b6c6b4a6c06f803/prometheus_client/openmetrics/parser.py#L466

type_suffixes = {
        'counter': ['_total', '_created'],
        'summary': ['', '_count', '_sum', '_created'],
        'histogram': ['_count', '_sum', '_bucket', '_created'],
        'gaugehistogram': ['_gcount', '_gsum', '_bucket'],
        'info': ['_info'],
    }
@hairyhenderson
Copy link
Collaborator

Hi @M0NsTeRRR, I'm not sure what the issue is here - these are both metrics exposed by the Prometheus go client - https://github.com/prometheus/client_golang. They are named differently, and so by classical Prometheus rules they don't conflict, though I do recognize that OpenMetrics has some slightly different expectations.

To be clear, when you parse with the python client, are you parsing directly or through the exporter-exporter? If the latter, try taking that out of the equation and scraping metrics directly.

In the Caddy codebase we enable OpenMetrics negotiation automatically, which may be why the python client is running into that issue.

@M0NsTeRRR
Copy link
Author

Hello @hairyhenderson,

When I parse them with the python client I use the direct output of the caddy exporter.

I've read this issue : prometheus/common#253 (comment)
and it's the same case as here, if I'm not mistaken.

@hairyhenderson
Copy link
Collaborator

Thanks for finding that issue @M0NsTeRRR. It does look significantly related. I think what that means is the presence of go_memstats_alloc_bytes_total is a bug in prometheus/client_golang since it causes the client to be non-conformant with OpenMetrics.

I'm not sure what we can do other than disabling OpenMetrics negotiation support or - probably better - make it configurable.

It'd probably also be a good idea to file a bug in prometheus/client_golang, since it's producing this output.

Modulo holidays/etc, I can do both of these things at some point over the next few days 😉...

@M0NsTeRRR
Copy link
Author

Thanks for finding that issue @M0NsTeRRR. It does look significantly related. I think what that means is the presence of go_memstats_alloc_bytes_total is a bug in prometheus/client_golang since it causes the client to be non-conformant with OpenMetrics.

I'm not sure what we can do other than disabling OpenMetrics negotiation support or - probably better - make it configurable.

It'd probably also be a good idea to file a bug in prometheus/client_golang, since it's producing this output.

Modulo holidays/etc, I can do both of these things at some point over the next few days 😉...

Of course thanks you, i'm learning golang since few days so I can't help you a lot on that works ahah :), feel free to tag me here or in the new issue if you need some logs or config files

@hairyhenderson
Copy link
Collaborator

Ok, I've added an option to disable the negotiation in #3944, and logged prometheus/client_golang#829.

Once #3944 is merged this issue can be closed, since you'll be able to configure the metrics handler to avoid the OM negotiation, which should be fine.

It may be too late to get this into Caddy 2.3.0, but hopefully not 😅

@francislavoie francislavoie added this to the v2.3.0 milestone Dec 30, 2020
@hairyhenderson
Copy link
Collaborator

@M0NsTeRRR when 2.3.0 is released (probably sometime over the next week) you can upgrade to that and set disable_openmetrics.

If you're using the metrics handler that's on the admin endpoint you'll need to switch to the explicit metrics handler instead since the admin one can't be configured.

Based on conversation in the prometheus/client_golang issue there likely is a bug there where ambiguous metrics are being exposed.

@M0NsTeRRR
Copy link
Author

Okay, thanks you @hairyhenderson for your quick fix :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants