Report dropped attributes/events/links for otlp/jaeger/zipkin exporters #1893

codeboten · 2021-06-04T19:12:50Z

Description

As mentioned in issue #1850, the spec added definition for how dropped attributes, links and events counts should be reported via non-otlp exporters. This PR addresses that.

This PR has been updated with the latest changes from the refactors.

Fixes #1850

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Unit tests added

Does This PR Require a Contrib Repo Change?

No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

...xporter-jaeger-proto-grpc/src/opentelemetry/exporter/jaeger/proto/grpc/translate/__init__.py

...emetry-exporter-jaeger-thrift/src/opentelemetry/exporter/jaeger/thrift/translate/__init__.py

exporter/opentelemetry-exporter-zipkin-json/tests/encoder/test_v1_json.py

exporter/opentelemetry-exporter-zipkin-json/tests/encoder/test_v2_json.py

exporter/opentelemetry-exporter-zipkin-json/tests/encoder/test_v1_json.py

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

lzchen · 2021-06-07T23:02:51Z

...porter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/trace_exporter/__init__.py

@@ -272,6 +272,18 @@ def _translate_data(
            self._translate_events(sdk_span)


I believe proto defines dropped_attributes_count nested inside Event and Link as well?

I think this still applies?

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

ocelotl · 2021-06-09T20:19:14Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+            if maxlen < 0:
+                raise ValueError
+        self.maxlen = maxlen
+        self._immutable = immutable


Hm... this does look a bit out of place, since isinstance(BoundedDict, MutableMapping) will be True.

ocelotl · 2021-06-09T20:25:46Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+        self.maxlen = maxlen
+        self._immutable = immutable
+        self.dropped = 0
+        self._dict = OrderedDict()  # type: OrderedDict


Just a heads up, we still support 3.6, since 3.7 dicts are insertion ordered.

This is code that's already existing in the SDK as BoundedDict, I moved it to the API as I needed it for Links

ocelotl · 2021-06-09T20:27:41Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+        if maxlen is not None:
+            if not isinstance(maxlen, int):
+                raise ValueError
+            if maxlen < 0:


Not sure if we should be doing type checking, but if we do, this check can be collapsed with an or.

ocelotl · 2021-06-09T20:34:09Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+    def from_map(cls, maxlen, immutable, mapping):
+        mapping = OrderedDict(mapping)
+        bounded_dict = cls(maxlen)
+        for key, value in mapping.items():


Hm, this seems to be arbitrary, I mean why should the first be kept instead of the last?

If it is impossible to not be arbitrary, this behavior should be documented.

Yup, I agree, although this is existing behaviour I do not wish to change w/ this PR.

ocelotl · 2021-06-09T20:35:33Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+        return self._dict.copy()
+
+    @classmethod
+    def from_map(cls, maxlen, immutable, mapping):


This is a bit out of place as a class method. How about passing the mapping to __init__ as an optional argument?

Yeah I'm not sure why the decision was made to have a class method on the BoundedDict in the first place.

ocelotl · 2021-06-09T21:51:27Z

opentelemetry-api/src/opentelemetry/attributes/__init__.py

+        for key, value in mapping.items():
+            bounded_dict[key] = value
+        bounded_dict._immutable = immutable  # pylint: disable=protected-access
+        return bounded_dict


I would suggest an implementation like this one:

from collections import OrderedDict from threading import Lock from unittest import TestCase from math import inf class BoundedOrderedDict(OrderedDict): def __init__(self, maxlen=inf): self._maxlen = maxlen self._dropped = 0 self._lock = Lock() @property def maxlen(self): return self._maxlen @property def dropped_attributes(self): return self._dropped def __setitem__(self, key, value): with self._lock: if self.maxlen == len(self): del self[next(iter(self.keys()))] self._dropped += 1 super().__setitem__(key, value) def __iter__(self): with self._lock: return super().__iter__() class BoundedOrderedDictTest(TestCase): def test_bounded_ordered_dict(self): test_dict = BoundedOrderedDict() test_dict[1] = 1 assert test_dict[1] == 1 test_dict = BoundedOrderedDict(maxlen=2) test_dict[0] = 1 assert len(test_dict) == 1 test_dict[0] = 0 assert len(test_dict) == 1 test_dict[1] = 1 assert len(test_dict) == 2 assert test_dict.dropped_attributes == 0 test_dict[2] = 2 assert len(test_dict) == 2 assert test_dict.dropped_attributes == 1 assert list(test_dict.items()) == [(1, 1), (2, 2)]

ocelotl · 2021-06-09T21:59:53Z

opentelemetry-api/src/opentelemetry/trace/__init__.py

-    @property
-    def attributes(self) -> types.Attributes:
-        return self._attributes
+        _LinkBase.__init__(self, context)


I think super would take care of this automatically.

I didn't think super would work with different arguments

I mean, we can achieve that by changing the parent classes, something like this:

class FirstParent: def __init__(self, first, *args): print(first) super().__init__(*args) class SecondParent: def __init__(self, second, *args): print(second) super().__init__(*args) class Child(FirstParent, SecondParent): pass Child("first", "second")

first second

opentelemetry-api/src/opentelemetry/attributes/__init__.py

ocelotl · 2021-06-09T22:11:56Z

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

-        return self._attributes
+        EventBase.__init__(self, name, timestamp)
+        Attributed.__init__(
+            self, attributes, limit=limit, immutable=True, filtered=True


This feels inconsistent. We are implementing immutability in Resource by returning a copy of attributes.

i agree, the handling of attributes is inconsistent currently. I'm hoping this PR will help with that. I don't know which way is the better way to handle it, but I'm open to suggestions

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

ocelotl · 2021-06-10T18:53:57Z

opentelemetry-api/src/opentelemetry/trace/__init__.py

-    @property
-    def attributes(self) -> types.Attributes:
-        return self._attributes
+        _LinkBase.__init__(self, context)


I mean, we can achieve that by changing the parent classes, something like this:

class FirstParent: def __init__(self, first, *args): print(first) super().__init__(*args) class SecondParent: def __init__(self, second, *args): print(second) super().__init__(*args) class Child(FirstParent, SecondParent): pass Child("first", "second")

first second

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

codeboten · 2021-06-10T20:49:09Z

Put this PR back to draft mode, as I'll be breaking the functionality that's not specific to the exporters in separate PRs. See #1909 for the first chunk of that work.

…ributes

codeboten · 2021-06-25T17:10:07Z

@lzchen @ocelotl updated the PR to account for the other changes made in #1915 and #1909

srikanthccv · 2021-06-26T03:00:47Z

opentelemetry-sdk/tests/trace/test_trace.py

+        self.assertEqual(3, span.dropped_events)
+        self.assertEqual(2, span.events[0].attributes.dropped)
+        self.assertEqual(2, span.links[0].attributes.dropped)
+        self.assertEqual(2, span.resource.attributes.dropped)


Hmm, are dropped attributes from events, links added to total dropped count or dropped_attributes_count is more like dropped_span_attributes_count?

Dropped attributes counts are independent for events/links/spans/resources.

Correct. My question is whether dropped attributes counts of events/links are reported for exporters?

Dropped attribute counts for events/links are reported for OTLP as defined in proto. I don't think they have been defined for the other exporters.

I believe we are not doing that in OTLP exporter https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/trace_exporter/__init__.py#L169.

Correct, I noted that in this comment. I believe we should add this as well.

I added them now, please take a look

Note that according to the spec, the Zipkin exporter does not provide a mechanism to export attributes for events, and support for Links is TBD.

The Jaeger exporter described in the spec suggests implementations MAY support attributes for links via logs, but doesn't require it.

Alex Boten added 6 commits June 3, 2021 15:03

adding support for dropped attributes/events/links to jaeger exporters

6ca05d6

cleanup

48193ca

lint

db8ef5c

adding support for dropped attributes/events/links to zipkin exporters

4ef469c

add tests for sdk

734471e

add tests for trace sdk

dbe3e32

codeboten requested review from a team, aabmass and lzchen and removed request for a team June 4, 2021 19:12

codeboten changed the title ~~Codeboten/dropped attributes~~ Report dropped attributes/events/links for jaeger/zipkin exporters Jun 4, 2021

fix lint

69704f7

codeboten added the Approve Public API check This label shows that the public symbols added or changed in a PR are strictly necessary label Jun 4, 2021

Alex Boten added 3 commits June 4, 2021 13:21

update changelog

8fe4266

Merge branch 'main' into codeboten/dropped_attributes

625e6ec

adding dropped counts for otlp

b7094ae

ocelotl suggested changes Jun 4, 2021

View reviewed changes

fix lint

967c1fd

codeboten changed the title ~~Report dropped attributes/events/links for jaeger/zipkin exporters~~ Report dropped attributes/events/links for otlp/jaeger/zipkin exporters Jun 4, 2021

fix lint

809ee60

ocelotl suggested changes Jun 4, 2021

View reviewed changes

exporter/opentelemetry-exporter-zipkin-json/tests/encoder/test_v2_json.py Outdated Show resolved Hide resolved

exporter/opentelemetry-exporter-zipkin-json/tests/encoder/test_v1_json.py Outdated Show resolved Hide resolved

Alex Boten added 4 commits June 7, 2021 11:51

remove single use method

a782291

cleanup test

ec3a28e

update tests

b8da999

Merge branch 'main' into codeboten/dropped_attributes

e2a1e45

codeboten requested a review from ocelotl June 7, 2021 19:06

fix lint

9785866

lzchen reviewed Jun 7, 2021

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py Outdated Show resolved Hide resolved

lzchen reviewed Jun 7, 2021

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py Show resolved Hide resolved

ocelotl approved these changes Jun 8, 2021

View reviewed changes

adding Attributed class

3e1ed3e

codeboten requested review from ocelotl and lzchen June 9, 2021 16:07

ocelotl suggested changes Jun 9, 2021

View reviewed changes

remove unused import

7ae76bd

ocelotl suggested changes Jun 10, 2021

View reviewed changes

codeboten marked this pull request as draft June 10, 2021 20:48

Alex Boten added 2 commits June 25, 2021 09:35

Merge remote-tracking branch 'origin/main' into codeboten/dropped_att…

c81fba7

…ributes

update code w/ latest changes in main

22fb2f6

codeboten force-pushed the codeboten/dropped_attributes branch from 43cd873 to 22fb2f6 Compare June 25, 2021 17:06

codeboten marked this pull request as ready for review June 25, 2021 17:07

codeboten requested a review from ocelotl June 25, 2021 17:07

srikanthccv reviewed Jun 26, 2021

View reviewed changes

Alex Boten added 4 commits June 28, 2021 15:49

add link/event attribute drop counters for otlp

4d3666a

adding dropped attributes for events to jaeger

5013d09

update test name for zipkin

d8259bb

fix lint

097cf5d

ocelotl approved these changes Jun 29, 2021

View reviewed changes

srikanthccv approved these changes Jun 30, 2021

View reviewed changes

lzchen approved these changes Jun 30, 2021

View reviewed changes

codeboten merged commit d724573 into open-telemetry:main Jun 30, 2021

codeboten deleted the codeboten/dropped_attributes branch June 30, 2021 22:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report dropped attributes/events/links for otlp/jaeger/zipkin exporters #1893

Report dropped attributes/events/links for otlp/jaeger/zipkin exporters #1893

codeboten commented Jun 4, 2021 •

edited

Loading

lzchen Jun 7, 2021

lzchen Jun 28, 2021

ocelotl Jun 9, 2021

ocelotl Jun 9, 2021

codeboten Jun 9, 2021

ocelotl Jun 9, 2021

ocelotl Jun 9, 2021

codeboten Jun 9, 2021

ocelotl Jun 9, 2021

codeboten Jun 9, 2021

ocelotl Jun 9, 2021

ocelotl Jun 9, 2021

codeboten Jun 9, 2021

ocelotl Jun 10, 2021

ocelotl Jun 9, 2021

codeboten Jun 9, 2021

ocelotl Jun 10, 2021

codeboten commented Jun 10, 2021

codeboten commented Jun 25, 2021

srikanthccv Jun 26, 2021

codeboten Jun 28, 2021

srikanthccv Jun 28, 2021

lzchen Jun 28, 2021

srikanthccv Jun 28, 2021

lzchen Jun 28, 2021

codeboten Jun 29, 2021

codeboten Jun 29, 2021

		@@ -272,6 +272,18 @@ def _translate_data(
		self._translate_events(sdk_span)

Report dropped attributes/events/links for otlp/jaeger/zipkin exporters #1893

Report dropped attributes/events/links for otlp/jaeger/zipkin exporters #1893

Conversation

codeboten commented Jun 4, 2021 • edited Loading

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten commented Jun 10, 2021

codeboten commented Jun 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten commented Jun 4, 2021 •

edited

Loading