(0.14.1) https_request_count data does not match the data on GCP

[Waizzzzz]

Hello, today we've noticed that the data that is exported by this exporter doesn't match up with what GCP monitoring shows even if the queries are identical between both environments.

The metric in question is stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_request_count and in Grafana the total value shows to be about 60% less than that is shown in GCP.

The discrepancy is (logically) more noticeable in resources that have a lot more requests going through them.
The full query used in both environments is:

sum by(response_code_class) (stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_request_count{target_proxy_name="_REDACTED_"})

[kgeckhart]

So unfortunately, you're trying to compare apples and oranges by looking at sum of the exporter vs GCP due to this being a delta metric. The exporter is doing an in-memory accumulation of delta values it has accumulated since startup. Just like with a pure prometheus counter you should apply a rate or an increase to look at the change over time vs the raw value. This is where the data is intended to match GCP as close as possible. You can read more about some caveats in https://github.com/prometheus-community/stackdriver_exporter?tab=readme-ov-file#what-to-know-about-aggregating-delta-metrics