Capability to update otel configuration dynamically at runtime

[squaw077]

Is your feature request related to a problem? Please describe.
Need capability to update otel collector's configuration at runtime. In particular, need capability to add/remove exporters while collector is running.

Describe the solution you'd like
Solution in the form of api in otel collector that would allow to change configuration (add/remove exporters) remotely at runtime. Simple solution could be api that would allow to reload otel's collector configuration from file at runtime.

Describe alternatives you've considered
-Brute force solution involving restart of otel collector every time configuration changes.
-Building custom otel extension which will monitor configuration changes and apply those changes as they arrive

[bogdandrutu]

What distribution do you use? If you are talking about your own distro then this is already available

[squaw077]

I'm currently using otel collector binary w/ configuration file from: https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.29.0. Ans also can upgrade to latest version if necessary.

If you think that requested capability tis already available there, could you please point me to the reference describing how to use it. By some reason I sifted through official otel documentation and did not find any description there.

[bogdandrutu]

You are talking about an API to change the config, who is going to call that API? What is the trigger of the change?

[squaw077]

I have daemon running along w/ otel collector in the same container. This daemon is expected to communicate config changes to otel collector using API in question. It could be REST API endpoint or any other api that would allow to communicate config changes over to otel collector at runtime. In solution that I'm building, user triggers otel config changes. These changes have to be delivered over to otel collector at runtime.

[neelayu]

As far as I know, if you build your own otelcol distribution, you can implement the config.WatchableMapProvider interface, to watch for any configuration changes. Correct me if I am wrong, the default MapProvider structs do not implement it.

[bogdandrutu]

If you have a daemon already you can shutdown and start again the collector with the new file since that is what we do anyway internally.

[bogdandrutu]

Closing this because we will fix the file watch then you will be able to simply update the file.

[squaw077]

Just wanted to confirm if this feature has already been implemented. If so, please confirm which build it can be found in. Otherwise, please advise when are you planning to implement it.

[cforce]

@bogdandrutu You closed the ticket but am also puzzled as well as @squaw077 if and how this feature is implemented and searching for an proper example how to use properly. Would you be so kind and answer this questions, tx for support.

There are so many stories discussed and MR's around, its a jungle and impossible to understand what made in into the release and where its documented? Even if has been rolled back in terms of default enablement or not . Its not clear to me where we are.

• Add file watch to support config reload on file change #4454
• configmapprovider.File does not watch for file changes #4397
• Watch config file for changes #273
• Add a flag for enabling config watch #6300
• Configuration reloading logic and customizability #5966

Also the new new approach using opamp - if it will replace the other and how much this is dependent on Kubernetes

• New component: OpAMP Configuration Provider. opentelemetry-collector-contrib#15295
• Create opamp extension opentelemetry-collector-contrib#16462
• Configuration reloading needs rethinking from error handling perspective #6226
• OpAMP Agent Extension opentelemetry-collector-contrib#16594

[spencerwilson]

@cforce , another link for your collection: #6000, which was released in v0.65.0. I've not tested it so YMMV.

service/collector: Support SIGHUP configuration reloading (#5966)

[JellinekSara]

I would also be glad if someone could help me understand if there are any features that support configuration updates at runtime. Let it be a watcher or some callable endpoint, but I'm really missing this feature and have no idea how to get the service read my updated configs.

[gongcon]

Same asking here. Sending SIGHUP to the process causes it restarting internally. From the log I can see the service shutdown and restart. Although the restart time is short, it still causes issue in our high traffic environment.

[rohang98]

I'm also wondering about the same capability. Can we reopen this ticket?

[cforce]

Check the opamp docs and opam extension for collector in control project.
There are two options available:
a.) opamp extension in collector connect directly to omap server on another host also let you remote configure the cfg almost real time but opamp extension might restart the collector itself. It could be dangerous if something goes south and collector does not come back properly connected to the opamp backend again to give back con to to fix the buggy cfg
b.) The opamp extension of the collector connect to a same host or same cluster(k8) supervisor (also hosted in collector contrib project) as realy to the opmap backend. The supervisor as its name supervisees the collector by health during and after the restart which ts executed with pri9viledge by the supervisor as well. This gives the solution the advantage to recover and also continuously monitor the monitor from an "safe position" whatever goes south go stay in the driver seat, even to replace the binary of the collector

[gongcon]

Based on the doc, opamp is not for production yet (60% progress). @cforce, you mentioned that "opamp extension might restart the collector itself". Isn't this what we are trying to avoid? The major issue for us is the tail sampling process losses all in-memory traces when the collector restarts. Ideally, it should support a hot config swap like what Refinery does.

[cforce]

Then you shall add the tail sampling need as supervisor/opamp requirement . by a new feature request with the components supervisor/opamp extension tagged.
If you are right its indeed a limitation of the opamp concept which shall be tackled.

[mithunbelur]

I am facing the same issue. Otel collector is bundled with our own daemon/container.
Is there a way or pointers in the code to add/delete receiver configs and udpate the pipeline to use/remove this new receiver.
Or Is there a way to just add new pipeline with new set of receivers without touching other pipelines?
Both SIGHUP/ or calling otelcollector.Shutdown() followed by otelcollector.Run () does restart the existing pipeliens and their components.