Kafka sasl_ssl/kerberos auth not working

[mguggi]

Component(s)

receiver/kafka

What happened?

Description

We have a Kafka cluster that only supports SASL_SSL. For this I tried to configure auth.tls in addition to auth.kerberos. But in my opinion already fetching the Kerberos ticket does not work.

The same configuration files (keytab and krb5.conf) work for example with librdkafka.

Expected Result

A connection to Kafka and receiving the spans from the configured topic.

Actual Result

As soon as the collector is deployed, the following message appears and the collector does not start.

Error: failed to build pipelines: failed to create "kafka" receiver for data type "traces": kafka: client has run out of available brokers to talk to: [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database

Collector version

0.82.0

Environment information

Environment

The collector runs as Kubernetes (1.25) deployment. The deployment was made with the official helm chart (0.65.0).

OpenTelemetry Collector configuration

config:
  receivers:
    jaeger: null
    prometheus: null
    zipkin: null
    otlp: null
    kafka:
      protocol_version: 2.4.0
      brokers: |-
        <broker-address>:9093
      topic: <my-topic>
      encoding: zipkin_json
      group_id: otel-collector-tracing
      auth:
        kerberos:
          service_name: kafka
          realm: <domain>
          username: <user>@<domain>
          use_keytab: true
          config_file: /kafka/etc/krb5.conf
          keytab_file: /kafka/auth/kafka-user.keytab
        tls:
          ca_file: /kafka/ssl/root-ca
          insecure: false
  exporters:
    otlp/tempo:
      endpoint: tempo-distributor:4317
      headers:
        x-scope-orgid: default
      tls:
        insecure: true
    zipkin:
      format: json
      default_service_name: unknown-service
    logging:
      verbosity: detailed
  processors:
    # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 20
    attributes:
      actions:
      - key: environment
        value: development
        action: insert
  service:
    pipelines:
      logs: null
      metrics: null
      traces:
        exporters:
          - logging
        processors:
          - memory_limiter
          - batch
          - attributes
        receivers:
          - kafka
    telemetry:
      logs:
        encoding: json
      metrics:
        level: detailed
        address: ${env:MY_POD_IP}:8888

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  metrics:
    enabled: true

Log output

Error: failed to build pipelines: failed to create "kafka" receiver for data type "traces": kafka: client has run out of available brokers to talk to: [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy
• needs: Github issue template generation code needs this to generate the corresponding labels.

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.