fix: bring back OTEL receiver metrics

[javiermolinar]

What this PR does:
This PR recovers the OTEL receiver metrics that Tempo previously exported.

The metrics stopped from being exported in this commit: #3148.
Doing a bit of code archeology here is where the OpenCensus code is removed open-telemetry/opentelemetry-collector#9102.

The way this works is by implementing a custom MeterProvider, adapted from go.opentelemetry.io/otel/metric/noop, to intercept the metrics emitted by obsreport.go and record them using our internal tooling.

Example of the metrics from the metrics endpoint:

# HELP tempo_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE tempo_receiver_accepted_spans counter
tempo_receiver_accepted_spans{receiver="tempo/otlp_receiver",transport="http"} 1
# HELP tempo_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE tempo_receiver_refused_spans counter
tempo_receiver_refused_spans{receiver="tempo/otlp_receiver",transport="http"} 0

Which issue(s) this PR fixes:
Fixes #3758

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Nice fix! One small perf tweak and we're good.

[joe-elliott]

if the name is not one we care to track we should just return a NopInt64Counter.

[zlin179]

Just wondering will we get a patch to 2.5 soon or we need to wait until 2.6 is released for this fix?

[joe-elliott]

Just wondering will we get a patch to 2.5 soon or we need to wait until 2.6 is released for this fix?

This will be released with 2.6. The release will likely occur towards the end of August

[zlin179]

Just wondering will we get a patch to 2.5 soon or we need to wait until 2.6 is released for this fix?

This will be released with 2.6. The release will likely occur towards the end of August

Thanks. Then is there any suggestion/workaround on checking logs/metrics on refused spans while receiver_refused_spans is missing? The only metrics I see is distributor_spans_received_total but it only shows the number of accepted_spans right?

[joe-elliott]

Then is there any suggestion/workaround on checking logs/metrics on refused spans while receiver_refused_spans is missing?

I would use: tempo_discarded_spans_total. It shows spans dropped for a number of reasons including ingestion reasons