Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capability to update tail sampling process configuration dynamically at runtime #10216

Open
gongcon opened this issue May 23, 2024 · 3 comments

Comments

@gongcon
Copy link

gongcon commented May 23, 2024

Important (read before submitting)

We are currently preparing for the upcoming 1.0 GA release. Feature requests that are not aligned with
the current roadmap and are not aimed at stabilizing and preparing the Collector for the release will
not be prioritized.

Delete this paragraph before submitting.

Is your feature request related to a problem? Please describe.
Tail sampling processor buffers spans in memory to make sampling decision of traces. When we update the config, sampling percentage for example, we update the yaml file and send a SIGHUP to the collector process. However, Otel collector restarts when it receives the SIGHUP. The buffered data are lost in the restart.

Describe the solution you'd like
Either the tail sampling process provides a way to dynamically update configuration at runtime, or Otel collector can do it in general. The later one is better since other processors, e.g. batch processor, may face the same issue.

It can be a feature of OpAMP as well as @cforce mentioned in #4205.

@cforce
Copy link
Contributor

cforce commented May 24, 2024

flushing of buffers from samplers could also be a solution. A friendly tear down before an opamp initiated restart of the collector would make sure buffered traces from tail sampling are not dropped but send to the exporters as eviction strategy. Maybe this is even a feature already with proper cfg - see https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md#dropped-traces

@gongcon
Copy link
Author

gongcon commented May 24, 2024

It's not a feature in tail sampling processor. It drops spans when the buffer is full, nothing to do with graceful shutdown.

@cforce
Copy link
Contributor

cforce commented May 24, 2024

Graceful shutdown related feature open-telemetry/opamp-go@528e802

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants