Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify MeterProvider configurable cardinality limits #2960

Merged
merged 37 commits into from
May 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
61264c7
Specify MetricReader configurable limits
Nov 18, 2022
7ba02d1
Add PR number and spec-matrix entry
Nov 18, 2022
16f6170
edits
Nov 18, 2022
82fb36d
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Nov 28, 2022
0f78e17
changelog order
Nov 28, 2022
26b2ce9
corrections, update spec matrix
Nov 28, 2022
57d35a5
Update specification/metrics/sdk.md
jmacd Nov 30, 2022
e755ee6
Update specification/metrics/sdk.md
jmacd Nov 30, 2022
00c44f6
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Dec 12, 2022
8c87107
Merge branch 'jmacd/metrics_limits' of github.com:jmacd/opentelemetry…
Dec 12, 2022
20c7555
revision
Dec 13, 2022
dcbcaf3
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Dec 16, 2022
7cdc7ab
toc
Dec 16, 2022
f12c8b5
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Jan 4, 2023
df16924
Apply suggestions from code review
jmacd Jan 6, 2023
1d7a117
Merge branch 'main' into jmacd/metrics_limits
reyang Jan 17, 2023
15bce36
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Feb 9, 2023
d116a28
Update with ideas from Jan Spec SIG
Feb 9, 2023
a92c4c7
Merge branch 'jmacd/metrics_limits' of github.com:jmacd/opentelemetry…
Feb 9, 2023
c72fae4
refine text
Feb 10, 2023
bfbf348
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Mar 3, 2023
6cd74e0
Revise based on prototype in otel-launcher-go
Mar 3, 2023
0e59093
revise spelling
Mar 3, 2023
c4a5a61
revise spelling
Mar 3, 2023
7e2bf6d
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Mar 27, 2023
81ae9e7
clarify wording; give MetricReader control over the default
Mar 27, 2023
88c1557
reword
Mar 27, 2023
fbf66c7
use aggregation_cardinality_limit
Mar 28, 2023
e53c22d
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Apr 17, 2023
f1f47f4
clarify attribute SETs h/t @pirgeo
Apr 17, 2023
a0a6596
Update specification/metrics/sdk.md
jmacd Apr 17, 2023
7099f07
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
Apr 19, 2023
7505081
update changelog
Apr 19, 2023
9569ef5
lint
Apr 19, 2023
1a98c25
Update specification/metrics/sdk.md
jmacd May 4, 2023
4aaad07
Merge branch 'main' of github.com:open-telemetry/opentelemetry-specif…
May 5, 2023
cee0c74
Merge branch 'jmacd/metrics_limits' of github.com:jmacd/opentelemetry…
May 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ release.

- Add experimental histogram advice API.
([#3216](https://github.com/open-telemetry/opentelemetry-specification/pull/3216))
- Recommended cardinality limits to protect metrics pipelines against
excessive data production from a single instrument.
([#2960](https://github.com/open-telemetry/opentelemetry-specification/pull/2960))
- Specify second unit (`s`) and advice bucket boundaries of `[]`
for `process.runtime.jvm.gc.duration`.
([#3458](https://github.com/open-telemetry/opentelemetry-specification/pull/3458))
Expand Down
66 changes: 66 additions & 0 deletions specification/metrics/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ linkTitle: SDK
* [Use the maximum scale for single measurements](#use-the-maximum-scale-for-single-measurements)
* [Maintain the ideal scale](#maintain-the-ideal-scale)
* [Observations inside asynchronous callbacks](#observations-inside-asynchronous-callbacks)
* [Cardinality limits](#cardinality-limits)
+ [Synchronous instrument cardinality limits](#synchronous-instrument-cardinality-limits)
+ [Asynchronous instrument cardinality limits](#asynchronous-instrument-cardinality-limits)
- [Meter](#meter)
* [Duplicate instrument registration](#duplicate-instrument-registration)
* [Instrument name](#instrument-name)
Expand Down Expand Up @@ -235,6 +238,12 @@ are the inputs:
`exemplar_reservoir` (optional) to use for storing exemplars. This should be
a factory or callback similar to aggregation which allows different
reservoirs to be chosen by the aggregation.
* **Status**: [Experimental](../document-status.md) - the
`aggregation_cardinality_limit` (optional) associated with the view. This
should be a positive integer to be taken as a hard limit on the
number of data points that will be emitted during a single
collection by a single instrument. See [cardinality limits](#cardinality-limits),
below.
jmacd marked this conversation as resolved.
Show resolved Hide resolved

In order to avoid conflicts, views which specify a name SHOULD have an
instrument selector that selects at most one instrument. For the registration
Expand Down Expand Up @@ -582,6 +591,62 @@ execution.
The implementation MUST complete the execution of all callbacks for a
given instrument before starting a subsequent round of collection.

### Cardinality limits
jmacd marked this conversation as resolved.
Show resolved Hide resolved

**Status**: [Experimental](../document-status.md)

Views SHOULD support being configured with a cardinality limit to be
applied to all aggregators not configured by a specific view, specified
via `MetricReader` configuration.

View configuration SHOULD support applying per-aggregation cardinality limits.

The cardinality limit is taken as an exact, hard limit on the number
of data points that can be written per collection, per aggregation.
Each aggregation configured view MUST NOT output more than the
configured `aggregation_cardinality_limit` number of data points per
period.

The RECOMMENDED default aggregation cardinality limit is 2000.

An overflow attribute set is defined, containing a single attribute
`otel.metric.overflow` having (boolean) value `true`, which is used to
report a synthetic aggregation of the metric events that could not be
independently aggregated because of the limit.

The SDK MUST create an Aggregator with the overflow attribute set
prior to reaching the cardinality limit and use it to aggregate events
for which the correct Aggregator could not be created. The maximum
number of distinct, non-overflow attributes is one less than the
limit, as a result.

#### Synchronous instrument cardinality limits

Views of synchronous instruments with cumulative aggregation
temporality MUST continue to export the all attribute sets that were
observed prior to the beginning of overflow. Metric events
corresponding with attribute sets that were not observed prior to the
overflow will be reflected in a single data point described by (only)
the overflow attribute.
jmacd marked this conversation as resolved.
Show resolved Hide resolved

Views of synchronous instruments with delta aggregation temporality
MAY choose an arbitrary subset of attribute sets to output to maintain
the stated cardinality limit.

Regardless of aggregation temporality, the SDK MUST ensure that every
metric event is reflected in exactly one Aggregator, which is either
an Aggregator associated with the correct attribute set or an
aggregator associated with the overflow attribute set.

Events MUST NOT be double-counted or dropped during an
overflow.

#### Asynchronous instrument cardinality limits

Views of asynchronous instruments SHOULD prefer the first-observed
attributes in the callback when limiting cardinality, regardless of
aggregation temporality.

## Meter

Distinct meters MUST be treated as separate namespaces for the purposes of detecting
Expand Down Expand Up @@ -862,6 +927,7 @@ SHOULD provide at least the following:
* The `exporter` to use, which is a `MetricExporter` instance.
* The default output `aggregation` (optional), a function of instrument kind. If not configured, the [default aggregation](#default-aggregation) SHOULD be used.
* The default output `temporality` (optional), a function of instrument kind. If not configured, the Cumulative temporality SHOULD be used.
* The default aggregation cardinality limit to use, a function of instrument kind. If not configured, a default value of 2000 SHOULD be used.
jmacd marked this conversation as resolved.
Show resolved Hide resolved
jmacd marked this conversation as resolved.
Show resolved Hide resolved

The [MetricReader.Collect](#collect) method allows general-purpose
`MetricExporter` instances to explicitly initiate collection, commonly
Expand Down