Support OTel collector in flagd

[beeme1mr]

In order to get data into OTel, we need to be able to configure flagd to communicate with a collector. This entails:

• a new CLI arg for flagd, specifying the collector (taking the collector URI)
  • if CLI arg is set, configure OTLP in the OTel SDK (see example here)
  • deploy collector + configure it with Jaeger and confirm works as expected
• documentation and tests for the above

[Kavindu-Dodan]

I will start working on this :)

[Kavindu-Dodan]

I researched on this topic and now have a working solution inspired by OTEL demo [1]. With the knowledge, I have few questions to clarify so that I can finalize the first version of the implementation.

• Jaeger is specialized in tracing and currently, we do not record any tracing. So if we want to show Jaeger use-case, then we should add traces
• Our metrics are right now backed by Prometheus metrics [2], so if we need metrics at Collector (and then export to Prometheus), then we should also migrate the metric recorder to be backed by OTEL

A solution I see is to switch behavior based on the startup flag

If otel collector is setup, then expose metrics and traces to otel. Else, fallback to Prometheus metrics

And regarding traces, I will start with a simple trace which we can expand on later.

Let me know your opinion @beeme1mr @thisthat @toddbaert :)

[1] - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/examples/demo
[2] - https://github.com/open-feature/flagd/blob/main/core/pkg/runtime/from_config.go#L85

---

note - draft work (works on my machine 😉) #586

[thisthat]

I agree that at the moment, Jaeger is not necessary since we don't provide any trace.
On the metric point, we're using OpenTelemetry metrics with a Prometheus exporter. We could switch the exporter to use OTLP, but I suggest keeping the Prom exporter as the default one to be more flexible. The OpenTelemetry collector supports scraping Prometheus endpoints (see Prometheus-receiver).

To provide flexibility, we could use the env var OTEL_METRIC_EXPORTER to decide which exporter to use (doc). Quickly looking at the go-sdk, this looks like it is not yet implemented and there is a discussion around this topic (see spec). Hence, we must add that logic by ourselves.

[beeme1mr]

Hey @Kavindu-Dodan, there's another issue open for adding distributed traces to flagd. You can see that here.

I agree with @thisthat that we should leave the Prometheus exporter as the default. We just need to figure out a way to allow users to change metric exporter. Perhaps we could do something similar to the source configuration. That would make it easy to use cli flags when the use case is simple and a configuration when it becomes more complex. The OTEL_METRIC_EXPORTER env var may also be a good idea but I think it should behave like other flagd env vars.

[Kavindu-Dodan]

Thank you @thisthat & @beeme1mr for the feedback.

Can we have the following agreement then:

• Prometheus exporter will be the default metrics exporter
• Configurability to change the metric exporter - METRIC_EXPORTER or similar flag with an accepted value of otlp(handled manually) to switch to otel exporter
• Traces - Otel will be the default trace provider - will set the ground work for OTel Traces for evaluation in flagd #575 through this issue

[Kavindu-Dodan]

PR is ready for review - #586

I can start work on #575 if this is the next priority task :)