[exporter/prometheusexporter] There are errors with prometheusexporter in otel collector log messages

[xwgao]

Component(s)

exporter/prometheus

What happened?

Description

I'm using prometheusexporter in my otel collector. I found the below errors with prometheusexporter in otel collector log messages.

2023-07-07T07:05:56.702Z	error	prometheusexporter@v0.80.0/log.go:23	error gathering metrics: collected metric "http_server_active_requests" { label:{name:"http_method" value:"GET"} label:{name:"http_scheme" value:"https"} label:{name:"job" value:"mas811-api"} label:{name:"net_host_name" value:"10.254.24.116"} label:{name:"net_host_port" value:"9443"} gauge:{value:0} timestamp_ms:1688713513547} was collected before with the same name and label values
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.80.0/log.go:23
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/http.go:144
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2122
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2500
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
	go.opentelemetry.io/collector/config/confighttp@v0.80.0/compression.go:111
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2122
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.42.0/handler.go:212
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/collector/config/confighttp@v0.80.0/clientinfohandler.go:28
net/http.serverHandler.ServeHTTP
	net/http/server.go:2936
net/http.(*conn).serve
	net/http/server.go:1995

I'm using auto-instrumentation for Java for my service, the instrumentation image is ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.26.0.

Steps to Reproduce

Use prometheusexporter in otel collector config. After otel collector pod is running, can find the error messages from the otel collector pod log.

Expected Result

There is no error with prometheusexporter found in otel collector log messages.

Actual Result

Found errors with prometheusexporter in otel collector log messages.

Collector version

0.80.0

Environment information

Environment

OpenShift 4.12.16

OpenTelemetry Collector configuration

receivers:
      otlp:
        protocols:
          grpc:
          http:
      otlp/spanmetrics:
        protocols:
          grpc:
            endpoint: 0.0.0.0:12346

    processors:
      attributes/delete:
        actions:
          - key: "http.target"
            action: delete
      batch:
        timeout: 10s
        send_batch_size: 10000
      metricstransform:
        transforms:
          - include: mas-optimizer.duration
            match_type: strict
            action: insert
            experimental_match_labels: {
              "service.name": "mas811-api",
              "span.name": "POST /jobs"
            }
            new_name: mas-optimizer.api_create_job_duration_seconds

    exporters:
      logging:
        verbosity: detailed
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-optimizer
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 4s, 6s, 8s, 10s, 20s, 30s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.url
          - name: http.route
          - name: http.host

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch, attributes/delete]
          exporters: [spanmetrics, logging]
        metrics:
          receivers: [otlp, otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging]

Log output

2023-07-07T07:05:56.702Z	error	prometheusexporter@v0.80.0/log.go:23	error gathering metrics: collected metric "http_server_active_requests" { label:{name:"http_method" value:"GET"} label:{name:"http_scheme" value:"https"} label:{name:"job" value:"mas811-api"} label:{name:"net_host_name" value:"10.254.24.116"} label:{name:"net_host_port" value:"9443"} gauge:{value:0} timestamp_ms:1688713513547} was collected before with the same name and label values
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.80.0/log.go:23
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/http.go:144
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2122
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2500
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
	go.opentelemetry.io/collector/config/confighttp@v0.80.0/compression.go:111
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2122
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.42.0/handler.go:212
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/collector/config/confighttp@v0.80.0/clientinfohandler.go:28
net/http.serverHandler.ServeHTTP
	net/http/server.go:2936
net/http.(*conn).serve
	net/http/server.go:1995

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[xwgao]

/label exporter/prometheus help-wanted -exporter/prometheus

[iblancasa]

was collected before with the same name and label values

In Prometheus, this error notice frequently appears after a metric with the identical name and label values has already been gathered. If a metric with the same name and labels is collected more than once, Prometheus will throw this error because it expects metric names and label combinations to be unique.

To prevent this error, make sure that each combination of metric names and label values is distinct, or change the metric names and/or label values to make them so. As an alternative, you can combine metrics with the same name and label combinations into a single metric by utilizing functions like sum() or avg().

[Aneurysm9]

@iblancasa is correct about the category of error. The interesting thing is that I don't see the instance label on the metric reported in the rejected metric labels. I would expect that to be taken from the service.instance.id resource attribute. This is an important attribute used to uniquely identify timeseries in Prometheus. If the spanmetrics connector does not produce this value then this error is likely to occur.

[xwgao]

@Aneurysm9 I didn't find service.instance.id resource attribute from my otel collector pod log. How to make the spanmetrics connector produce this value so that this error could be avoided? Thanks.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9
• needs: Github issue template generation code needs this to generate the corresponding labels.

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

@xwgao There's a reference here for using the spanmetrics connector with Prometheus.

Also, Prometheus metric naming normalization is now done by default in the latest release, so you could try to again on v0.85.0.

[atoulme]

Closing as inactive, please reopen if you still have issues.