Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow exemplar-only metrics? #1719

Open
LouisStAmour opened this issue May 21, 2021 · 0 comments
Open

Allow exemplar-only metrics? #1719

LouisStAmour opened this issue May 21, 2021 · 0 comments
Labels
area:data-model For issues related to data model spec:metrics Related to the specification/metrics directory

Comments

@LouisStAmour
Copy link

What are you trying to achieve?

I'd like to build a trace processor that can export real-time metrics but also follow up with sampled exemplars later. I can see how Prometheus could support this use case by publishing a list of exemplars with real-time metric data but if Otel Collector publishes metrics to another collector for export to Prometheus, it cannot do so using OTLP unless a convention is established to export exemplars without adjusting real-time metrics.

Additional context.

The equivalent feature would be an API call to AddExemplar directly without observing any values, which I've requested in the Prometheus Go Client community: prometheus/client_golang#868 That request also indicates what my use case is.

An alternative would be to not support publishing exemplars separately using OTLP and instead encourage clients receiving metrics to ignore the observed value contained in the metric if a flag is set or the date of the exemplar is before the date of the observation, perhaps.

It's also possible that rather than changing the spec, the Otel Collector could internally publish exemplars from trace processors without adjusting real-time metric data so that exemplars could be combined later into a proper Otel Metric with the real-time data sourced from a different metric. I could see this being the case with the Otel Collector Prometheus exporter that already knows how to combine multiple metrics to export them for Prometheus, it could use a hypothetical AddExemplar function to add the exemplar captured from a trace processor/sampler. There's an argument that publishing an exemplar could also include publishing the real-time metric used to compare and ensure the exemplar's value is an outlier. That's true, but it's extra data on the wire if you've separately published the real-time data and it's possible you're not looking at real-time data but actually the data at the timestamp associated with the exemplar - historical data - if kept in a circular buffer, say.

@LouisStAmour LouisStAmour added the spec:metrics Related to the specification/metrics directory label May 21, 2021
@jsuereth jsuereth added the area:data-model For issues related to data model label May 25, 2021
@reyang reyang assigned reyang and unassigned reyang May 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:data-model For issues related to data model spec:metrics Related to the specification/metrics directory
Projects
None yet
Development

No branches or pull requests

3 participants