[exporter/loadbalancing] Add the ability to load balance by Metric stream ID #32513
Comments
RichieSams
added
enhancement
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
RichieSams
changed the title
RichieSams
changed the title
|
@RichieSams Hi, Thanks for the proposal. It's yours |
|
I am also needing something similar to this, but likely need to dig a bit deeper and make this per-metric-attribute. My use case is, I have a custom processor that decorates telemetry with an attribute, placed either on the log, metric, or span. This key is what I want to partition on, and send to each back-end to be gathered and ultimately written to the same s3 output file. What is the recommended way to add this, as the PR associated with this change seems to be rejected? I know the PR is not sufficient as it only looks at resource attributes, and I would need to look deeper, and I assume split deeper as well. |
|
I believe @RichieSams will work on this, the PR was closed in favor of breaking it into smaller chunks of work. |
|
Correct. I already split up the commits, I just got busy at work last week. I'll create new PRs today |
|
This is the first PR. Just a trivial helper function: #32794 Then there are these 4 commits that will be applied on top. @jpkrohling How do you want me to do them? I can serially make each a PR, get it merged, then go on to the next. Or do you want to combine any?
|
**Description:** This will merge the metrics in mdB into mdA, trying to re-use resourceMetrics, scopeMetrics, and metric values as possible. This will be used to help implement the new feature for: #32513 **Link to tracking Issue:** #32513 / #32690 **Testing:** I created a unit test which tests various scenarios of how merge behavior should happen **Documentation:** The exported function is documented using standard golang style. And there are comments inside the code to explain what is going on and why --------- Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue is still valid. Currently waiting for this PR to be merged, and then I can create the final PR that will close this issue. |
|
The dependent PR has been merged! |
**Description:** This adds a new routing option for metrics: streamID. This routes datapoints based on their streamID. That's the unique hash of all it's attributes, plus the attributes and identifying information of its resource, scope, and metric data **Link to tracking Issue:** #32513 **Testing:** I added to the existing testing suites, testing the new routing, as well as adding to the benchmark suite **Documentation:** I updated the README to describe the new routingKey: `metricID`, and how it works
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
@RichieSams , can this be closed? |
|
Yup! Closed as of 12d071d. Thanks for the reminder |
Component(s)
exporter/loadbalancing
Is your feature request related to a problem? Please describe.
The LoadBalancing exporter is a great idea. It currently exposes a number of ways to route metrics coming in:
However, there are many situation where these routing methods can lead to non-uniform distributions of metrics. For example, if datapoints use lots of labels, or med to high cardinality labels. The finest grain routing we can currently do is metric name routing. But if the majority of the datapoints are all the same metric name, then this doesn't help.
Describe the solution you'd like
I would like to propose adding a new value to the routing enum:
streamIDRouting. This would route individual datapoints based on their unique ID. We could use the newinternal/exp/metrics/identitypackage: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/internal/exp/metrics/identity/stream.go#L29I'm willing to be the implementer
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: