oltp grpc exporter failing in k8s deployment using helm charts

[parth-purohit]

Hi Team,

I have deployed both otel-collector as daemonset and standalone deployment as per the README docs in below link

https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

agentCollector:
enabled: true
standaloneCollector:
enabled: true

The oltp grpc exporter in agent pod is failing with below error:

2022-01-25T05:53:25.907Z	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "basic", "service.instance.id": "ebb82220-7804-4da1-af63-1bd1055120e8", "service.version": "latest"}
2022-01-25T05:53:25.907Z	info	service/collector.go:230	Starting otelcontribcol...	{"Version": "v0.37.1", "NumCPU": 4}
2022-01-25T05:53:25.907Z	info	service/collector.go:132	Everything is ready. Begin running and processing data.
2022-01-25T05:53:37.500Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.810290538s"}
2022-01-25T05:53:56.408Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "7.534661199s"}
2022-01-25T05:54:07.502Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.816822688s"}
2022-01-25T05:54:26.217Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "16.060087817s"}
2022-01-25T05:54:37.525Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.261162066s"}

Steps followed:
helm install --set standaloneCollector.enabled=true my-opentelemetry-collector open-telemetry/opentelemetry-collector
kubectl logs -f daemonset/my-opentelemetry-collector-agent

Could you please help resolve this issue.

[TylerHelmuth]

Hey @parth-purohit do you still need help with this issue? If so, can you provide:

• Version of the collector having issues (looks like 0.37.1)
• Version of the charts being used
• The startup logs of the agent
• The startup logs of the standalone collector

Thanks!

[parth-purohit]

Hi @TylerHelmuth I tried using the latest version and still facing the issue with opentelemetry-collector-0.9.4

• Version of the collector having issues: 0.46.0
• Version of the charts being used: 0.9.4
• standalone-collector-logs.txt
• agent-logs.txt

Thanks!

[TylerHelmuth]

Thanks. I'll be on PTO until next week but I will look when I get back.

Can you include your values.yaml as well?

[TylerHelmuth]

Any chance you have a proxy in your environment that the agent would be interacting with?

[parth-purohit]

Hi @TylerHelmuth I tried running it on a different machine and it worked.

Previous machine had some rules blocking traffic so it was failing.

We can close the this issue.

Thanks :)

[TylerHelmuth]

@dmitryax Ready to close.