exception in prometheus exporter AttributeError: 'frozenset' object has no attribute 'items'

[radder5]

Hi,
Using the code sample from https://github.com/open-telemetry/opentelemetry-python/blob/v1.12.0rc1/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py

from prometheus_client import start_http_server
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.metrics import get_meter_provider, set_meter_provider
from opentelemetry.sdk.metrics import MeterProvider
import random

start_http_server(port=8000, addr="localhost")

prefix = "MyAppPrefix"
reader = PrometheusMetricReader(prefix)

set_meter_provider(MeterProvider(metric_readers=[reader]))
meter = get_meter_provider().get_meter("myapp", "0.1.2")
counter = meter.create_counter(
    "requests",
    "requests",
    "number of requests",
)
labels = {"environment": "staging"}
counter.add(25, labels)

when accessing http://localhost:8000 and exception is thrown
File "/Users/m_652923/.pyenv/versions/3.8.9/lib/python3.8/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response) File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/prometheus_client/exposition.py", line 104, in _bake_output output = encoder(registry) File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/prometheus_client/exposition.py", line 197, in generate_latest for metric in registry.collect(): File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/prometheus_client/registry.py", line 97, in collect yield from collector.collect() File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/opentelemetry/exporter/prometheus/__init__.py", line 166, in collect self._translate_to_prometheus( File "/Users/m_652923/.pyenv/versions/otel/lib/python3.8/site-packages/opentelemetry/exporter/prometheus/__init__.py", line 204, in _translate_to_prometheus for key, value in number_data_point.attributes.items(): AttributeError: 'frozenset' object has no attribute 'items'

$pip freeze
backoff==1.11.1
certifi==2022.5.18.1
charset-normalizer==2.0.12
Deprecated==1.2.13
googleapis-common-protos==1.56.1
grpcio==1.46.3
idna==3.3
opentelemetry-api==1.12.0rc1
opentelemetry-exporter-otlp==1.12.0rc1
opentelemetry-exporter-otlp-proto-grpc==1.12.0rc1
opentelemetry-exporter-otlp-proto-http==1.12.0rc1
opentelemetry-exporter-prometheus==1.12.0rc1
opentelemetry-proto==1.12.0rc1
opentelemetry-sdk==1.12.0rc1
opentelemetry-semantic-conventions==0.31b0
prometheus-client==0.14.1
protobuf==3.20.1
requests==2.27.1
six==1.16.0
typing_extensions==4.2.0
urllib3==1.26.9
wrapt==1.14.1

This also happens when I try an up_down_counter

gauge = meter.create_up_down_counter(
        'requests_for_endpoints_ms',
        'millis',
        'Requests for endpoints in milliseconds'
    )

res = random.choice(results)
gauge.add(random.randint(10, 40), {'endpoint': res['endpoint']})

[agrapsas]

This happens for me, too, when I use the opentelemetry-exporter-otlp-proto-grpc exporter. If you remove the attributes you're passing it will work -- but, that is a pretty terrible experience. Passing no attributes or passing an empty dict will temporarily prevent the issue (but, again, terrible experience).

This is the line that is dying for me: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py#L273 as attributes is a frozenset and it is expecting a dict.

I have changed this line:

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/_view_instrument_match.py

Line 98 in 7413895

attributes = frozenset(attributes.items())

So that it is:

key = frozenset(attributes.items())

And then I only use key as a means of checking if self._attributes_aggregation has the attributes desired. This prevents mutation of attributes and subsequently resolves the issue.

I don't know if this is a good change as this is my first time cracking open the SDK. I'm happy to PR.

The block essentially becomes:

        key = frozenset(attributes.items())

        if key not in self._attributes_aggregation:
            with self._lock:
                if key not in self._attributes_aggregation:
                    if not isinstance(
                        self._view._aggregation, DefaultAggregation
                    ):
                        aggregation = (
                            self._view._aggregation._create_aggregation(
                                self._instrument,
                                attributes,
                                self._start_time_unix_nano,
                            )
                        )
                    else:
                        aggregation = self._instrument_class_aggregation[
                            self._instrument.__class__
                        ]._create_aggregation(
                            self._instrument,
                            attributes,
                            self._start_time_unix_nano,
                        )
                    self._attributes_aggregation[key] = aggregation

        self._attributes_aggregation[key].aggregate(measurement)

[radder5]

Yes, I had exactly the same experience with otel-exporter too.
I having to use the prometheus-exporter as I can't use the remoteprometheuswrite exporter from the otel collector - I only have the option to use a PULL design from prometheus. So until this is available I have to export directly from code to prometheus (although I'd prefer to decouple this with otel-collector)

[leptitchriss]

We are in the process of integrating OTLP and hitting this as well. Thank you for finding and taking the time to propose a solution to the issue @agrapsas !

[srikanthccv]

I don't know if this is a good change as this is my first time cracking open the SDK. I'm happy to PR.

@agrapsas The suggested change looks good to me. I believe you are correct and reason frozenset is used there because dict is not hashable for member checking.