Clarification of StartTimeUnixNano in metrics proto

[jrcamp]

The documentation around StartTimeUnixNano is a bit unclear because it says it MUST be set to the start of an interval:

// - StartTimeUnixNano MUST be set to the start of the interval when the data's
// type includes an AggregationTemporality. This field is not set otherwise.

Source

Then later it says 0 means it's unspecified.

// Value of 0 indicates that the timestamp is unspecified. In that case the
// timestamp may be decided by the backend.

Source

These seem a bit conflicting since I don't think an unspecified interval can be considered the start of an interval.

Also it's unclear what should be done when StartTimeUnixNano is unknown for a cumulative metric. Imagine that you are scraping an Apache status endpoint for "total_bytes_received" but you are unable to determine the start time of the server (when the metric was 0). What should StartTimeUnixNano be set to? (Perhaps depends on the outcome of the above issue.)

@bogdandrutu @jmacd

[jkwatson]

It seems like if you're scraping a cumulative metric, you should be recording that data via a SumObserver, which should use the LastValue aggregation, and end up as a Gauge-type metric in OTLP, which doesn't require a time range, IIRC.

[jrcamp]

It seems like if you're scraping a cumulative metric, you should be recording that data via a SumObserver, which should use the LastValue aggregation, and end up as a Gauge-type metric in OTLP, which doesn't require a time range, IIRC.

Looks like SumObserver ends up as Sum in OTLP. See this comment.

[jkwatson]

It seems like if you're scraping a cumulative metric, you should be recording that data via a SumObserver, which should use the LastValue aggregation, and end up as a Gauge-type metric in OTLP, which doesn't require a time range, IIRC.

Looks like SumObserver ends up as Sum in OTLP. See this comment.

Hm. Well, if you don't have a start time, then you probably need to make it a ValueObserver, which does report as a gauge. It's a shame we lose a little bit of information in this approach (namely, that it's monotonic).

[jmacd]

https://github.com/lightstep/opentelemetry-prometheus-sidecar takes an approach inherited from https://github.com/Stackdriver/stackdriver-prometheus-sidecar on this topic, which I've begun to think of as a cumulative-to-delta-to-cumulative transform. The idea is to leave a gap in the series at the point where a cumulative of unknown-reset-time appears, a point of transition where the actual rate of the series is unknown. The cumulative value first reported then becomes the reset value of the series, after which successive values are correctly timestamped cumulative / monotonic data points.

The gap in the series serves is reflected in one lost point, the reset value. In that gap, for all we know, any number of true resets may have taken place. Prometheus has similar ambiguity around resets, where instead of the gap proposed here, the value is presumed to have reset once any time the value falls from the previous known value in the series. https://github.com/lightstep/opentelemetry-prometheus-sidecar/blob/13e3b89cd20909a1dcc98d20e6fe1afdc437482a/retrieval/series_cache.go#L273

[jmacd]

This is a duplicate of #292