[processor/deltatocumulative]: evicts all streams under heavy load #33014
Labels
Comments
|
Pinging code owners for processor/deltatocumulative: @sh0rez @RichieSams. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
1 similar comment
|
Pinging code owners for processor/deltatocumulative: @sh0rez @RichieSams. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
The proposed solution/expected behavior makes sense to me, and was filed by a code owner. Removing |
jpkrohling
pushed a commit
that referenced
this issue
May 16, 2024
changes eviction behavior at limit to only delete streams if they are actually stale. Current behavior just deletes the oldest, which leads to rapid deletion of all streams under heavy load, making this processor unusable. **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> **Link to tracking Issue:** Fixes #33014 **Testing:** <Describe what testing was performed and which tests were added.> **Documentation:** <Describe the documentation added.>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
processor/deltatocumulative
What happened?
Description
When running with a limit that is much lower than the ingested stream rate, the deltatocumulativeprocessor deletes all its streams from tracking.
This is because of its eviction logic, which deletes the oldest stream (whether stale or not) once the stream limit is exceeded.
On a high load this happens all the time, deleting all streams over and over again.
Steps to Reproduce
Expected Result
Only stale streams are deleted. New streams that don't fit are dropped, making the processor operate on the subset it can track within the limit.
Actual Result
The processor admits evicts live (non-stale) streams, admits a new one which is subsequently evicted as well.
Collector version
a133a8e
Environment information
No response
OpenTelemetry Collector configuration
No response
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: