[RESOLVED] Metrics Parsing Error with Prometheus Input Plugin for Telegraf

[studersi]

We are attempting to integrate the metrics information introduced in mod_auth_openidc 2.4.15 into our monitoring setup.

We are using the Prometheus Input Plugin for Telegraf to collect the metrics.

Fetching the metrics appears to work as expected and the metrics are returned. However, there appears to be a parsing issue:
Telegraf log:

2024-01-17T09:19:19Z E! [inputs.prometheus] Error in plugin: error reading metrics for "https://123.123.123.123/oidc-metrics?format=prometheus&reset=true": reading text format failed: text format parsing error in line 9: second HELP line for metric name "example_com_authtype"

It looks like the response from mod_auth_openidc does not comply with the specification: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information.

Only one HELP line may exist for any given metric name.
Only one TYPE line may exist for a given metric name.

For example, the following output provides two HELP and TYPE lines for the metric example_com_authtype:

# HELP example_com_authtype The number of requests handled by mod_auth_openidc.
# TYPE example_com_authtype counter
example_com_authtype{handler="mod_auth_openidc"} 0

# HELP example_com_authtype The number of requests handled by AuthType openid-connect.
# TYPE example_com_authtype counter
example_com_authtype{handler="openid-connect"} 0

Other metrics do not appear to have this issue. For example, there is only one HELP and TYPE line for example_com_provider_metadata_bucket:

# HELP example_com_provider_metadata A histogram of provider discovery document requests.
# TYPE example_com_provider_metadata histogram
example_com_provider_metadata_bucket{le="0.1"} 0
example_com_provider_metadata_bucket{le="0.5"} 0
example_com_provider_metadata_bucket{le="1"} 0
example_com_provider_metadata_bucket{le="5"} 0
example_com_provider_metadata_bucket{le="10"} 0
example_com_provider_metadata_bucket{le="50"} 0
example_com_provider_metadata_bucket{le="100"} 0
example_com_provider_metadata_bucket{le="500"} 0
example_com_provider_metadata_bucket{le="1000"} 0
example_com_provider_metadata_bucket{le="5000"} 0
example_com_provider_metadata_bucket{le="+Inf"} 0
example_com_provider_metadata_sum 0
example_com_provider_metadata_count 0

Should these two example_com_authtype metrics be documented only once or are these separate metrics that should be named differently?

[zandbelt]

thanks for the feedback, interestingly enough Prometheus itself does not seem to have this issue; we'll indeed rename the metrics so they can co-exist,

alternatively it may be better to include the TYPE and HELP names only once in a generalized description, your suggestions are welcome

[studersi]

I have very little experience with the Prometheus format, so I cannot say what would be best.

From what I understand, the HELP and TYPE information describes the metric example_com_authtype itself independently of any labels that might be present. Therefore, it should probably not contain any information about the label at all.

Either way, from what I gather, there is always the possibility to add additional information using comments that are not HELP or TYPE. A plain comment would be ignored by Prometheus and any parser but could give additional information for debugging, when analyzing the response manually.
https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information

Lines with a # as the first non-whitespace character are comments. They are ignored unless the first token after # is either HELP or TYPE.

It looks like the examples always show the HELP and TYPE right before the actual measurements. Without a good reason to change this, I would probably stick to the way it is now. Would there be an advantage to grouping them together?

[studersi]

Looks like this is solved on master 👍

[zandbelt]

that is released now as part of https://github.com/OpenIDC/mod_auth_openidc/releases/tag/v2.4.15.1