Phantom Metrics being emitted

[elebiodaslingshot]

Cloud: AWS
Runtime: Docker
Python Version: 3.11
Otel Version: 1.20.0
Collector: Otel-collector

Steps to reproduce
Describe exactly how to reproduce the error. Include a code sample if applicable.

init.py to configure provider:

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from interfaces.otel.resource import OtelResource
from opentelemetry.metrics._internal import _ProxyMeterProvider
from utilities import Config


if Config.IS_LOCAL_OTEL:
    exporter = ConsoleMetricExporter()
else:
    exporter = OTLPMetricExporter(endpoint=Config.OTEL_EXPORTER_OTLP_ENDPOINT)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=Config.OTEL_METRIC_EXPORT_INTERVAL_MS)
provider = metrics.get_meter_provider()
if isinstance(provider, _ProxyMeterProvider):
  provider = MeterProvider([reader], resource=OtelResource(
                service_name=Config.OTEL_SERVICE_NAME,
                service_version=Config.OTEL_SERVICE_VERSION,
                service_environment=Config.OTEL_SERVICE_ENVIRONMENT
              ).resource)
  metrics.set_meter_provider(provider)


def force_flush():
  """
  Helper function to otel metrics force flush call
  """
  provider.force_flush()

def shutdown():
  """
  Helper function to otel metrics shutdown call
  """
  provider.shutdown()

Code usage:

from interfaces.otel.metrics import metrics

_meter = metrics.get_meter(__name__)
_retrievals_counter = _meter.create_counter("retrievals")

I see these random massive peaks in metrics being reported. By massive im talking in the order of thousands to tens of thousands. When I go and check our APM to see how many times this metric should have been emitted it is not even close to the expected metric value. Sometimes the API wouldn't have been called at all and there are still metrics emitted as if it has been hit.

What is the expected behavior?

Metrics to not be reported that should not be being reported.

What is the actual behavior?

Phantom metrics being reported in datadog.

Additional context

None

[srikanthccv]

Metrics are reported periodically even if there is no change in value (no API request would have been made in your case).

[garry-cairns]

@elebiodaslingshot I suspect this is related to changing hosts, though it's hard to say without knowing more about your setup. I've seen a similar problem in my running applications and described my working solution elsewhere.

[elebiodaslingshot]

Metrics are reported periodically even if there is no change in value (no API request would have been made in your case).

This would make sense if the instrument was a gage, but its a counter; so I would expect a correlation to log of an api hit and a metric being exported.