Will opentelemetry-operator support prometheus CRDs, like ServiceMonitor

[ifelseend]

Hi, guys

It is hard to manager prometheus receiver configs when there are many scrape jobs.
I want to know is it a plan to support prometheus ServiceMonitor CRDs, so that we can manage prometheus receiver configs with a better way.

[jpkrohling]

Perhaps a question for @alolita or @rakyll.

[Aneurysm9]

This is definitely something we'd like to support. I added support upstream so that we could re-use the config generation logic from the Prometheus operator. I think the remaining work would be to cause the config reconciliation to happen when any of the *Monitor CRs are changed. Since I'm not sure that the collector handles hot config reloading well (or at all) it might make sense to filter such changes through the recently added target allocation mechanism. It does handle hot config reloads and separates the collector from target discovery.

[jpkrohling]

Since I'm not sure that the collector handles hot config reloading well (or at all) it might make sense to filter such changes through the recently added target allocation mechanism. It does handle hot config reloads and separates the collector from target discovery.

The collector does not handle that. While I kinda understand why target allocator is doing that, I would rather do a regular Kubernetes rollout instead or restarting the container from the pod. This way, we protect the collector from crashing in case the config file can't be used (deprecated/new options).

[tejaswiniVadlamudi]

Does OTel Operator define its own CRDs and also support migration from Prometheus Operator CRDs to OTel CRDs?
or
Does OTel Operator support Prometheus Operator CRDs? => not inventing anything new

[secustor]

Initially the second option is more likely as we can reuse code from PrometheusOperator and their CRDs are widely used in the K8 ecosystem.
A minimal implementation of this is right now under review. #653

[yuriolisa]

@pavolloffay, @jpkrohling, @Aneurysm9, do you have any thoughts about that issue? I've seen there are many suggestions but I would like to define the way in which we are going to support the PRom CRDs.

[secustor]

I don't think this has been written down in it's entirety yet.

The current implementation idea, is to support target discovery CRDs only ( PodMonitor, ServiceMonitor ). This part is already implemented in the TargetAllocator.
But to become fully functional the collector needs a way to discovery dynamically new jobs from the TargetAllocator which has been discussed here.
There is also a draft for this implementation at the collector, which is blocked till the next release of Prometheus. open-telemetry/opentelemetry-collector-contrib#8055
If this PR is accepted, then some config changes are necessary to the Collectors deployed by the Operator.

[secustor]

Solved in Release v0.61.0