[Instrumentation] K8s metadata not set for some tags

[mat-rumian]

In case of auto-instrumentation for some of the tags like: k8s.node.name and k8s.pod.name values are not set correctly even if OTEL_RESOURCE_ATTRIBUTES_NODE_NAME and OTEL_RESOURCE_ATTRIBUTES_POD_NAME are set. OTel Collector with enabled k8s processor is not solving this issue.

Environment variables:

OTEL_RESOURCE_ATTRIBUTES_POD_NAME=sumologic-the-coffee-bar-bar-7bc584f678-tmw5q
OTEL_RESOURCE_ATTRIBUTES=service.name=the-coffee-bar,application=the-coffee-bar-app,k8s.container.name=the-coffee-bar,k8s.deployment.name=sumologic-the-coffee-bar-bar,k8s.namespace.name=the-coffee-bar-ns,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=sumologic-the-coffee-bar-bar-7bc584f678
OTEL_RESOURCE_ATTRIBUTES_NODE_NAME=ip-10-0-21-76.us-west-2.compute.internal

Current result:

k8s.node.name = $(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME)
k8s.pod.name = $(OTEL_RESOURCE_ATTRIBUTES_POD_NAME)

Expected result:

k8s.node.name = ip-10-0-21-76.us-west-2.compute.internal
k8s.pod.name = sumologic-the-coffee-bar-bar-7bc584f678-tmw5q

Environment:
OpenTelemetry-Operator 0.46.0
AWS EKS 1.19

[pavolloffay]

@mat-rumian could you please paste here the pod spec?

[pavolloffay]

As far as I know, the OTEL_RESOURCE_ATTRIBUTES_POD_NAME is not automatically added to the resource attributes. The feature request was not implemented open-telemetry/opentelemetry-specification#2135

[pavolloffay]

@mat-rumian any info for us?

[mat-rumian]

Hey @pavolloffay,
Sorry for late response. Here's the pod spec

apiVersion: v1
kind: Pod
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"
    kubectl.kubernetes.io/restartedAt: "2022-04-07T08:30:10Z"
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2022-04-07T09:59:35Z"
  generateName: sumologic-the-coffee-bar-bar-5c48ff68f8-
  labels:
    app: bar
    app.kubernetes.io/instance: sumologic-the-coffee-bar
    app.kubernetes.io/name: sumologic-the-coffee-bar
    pod-template-hash: 5c48ff68f8
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:instrumentation.opentelemetry.io/inject-python: {}
          f:kubectl.kubernetes.io/restartedAt: {}
        f:generateName: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/name: {}
          f:pod-template-hash: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"71779177-264d-4e41-8889-2124c3daadba"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:containers:
          k:{"name":"the-coffee-bar"}:
            .: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"OTEL_RESOURCE_ATTRIBUTES"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"OTEL_SERVICE_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:livenessProbe:
              .: {}
              f:exec:
                .: {}
                f:command: {}
              f:failureThreshold: {}
              f:initialDelaySeconds: {}
              f:periodSeconds: {}
              f:successThreshold: {}
              f:timeoutSeconds: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":8082,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
            f:readinessProbe:
              .: {}
              f:exec:
                .: {}
                f:command: {}
              f:failureThreshold: {}
              f:initialDelaySeconds: {}
              f:periodSeconds: {}
              f:successThreshold: {}
              f:timeoutSeconds: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:securityContext: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/tmp"}:
                .: {}
                f:mountPath: {}
                f:name: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:hostname: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:serviceAccount: {}
        f:serviceAccountName: {}
        f:terminationGracePeriodSeconds: {}
        f:volumes:
          .: {}
          k:{"name":"tls-certificate"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:secretName: {}
    manager: kube-controller-manager
    operation: Update
    time: "2022-04-07T09:59:35Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:initContainerStatuses: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.0.80.232"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: "2022-04-07T09:59:56Z"
  name: sumologic-the-coffee-bar-bar-5c48ff68f8-wqns7
  namespace: the-coffee-bar-ns
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: sumologic-the-coffee-bar-bar-5c48ff68f8
    uid: 71779177-264d-4e41-8889-2124c3daadba
  resourceVersion: "166327117"
  selfLink: /api/v1/namespaces/the-coffee-bar-ns/pods/sumologic-the-coffee-bar-bar-5c48ff68f8-wqns7
  uid: b93e3f9b-e330-4f81-8d05-640966429985
spec:
  containers:
  - command:
    - the-coffee-bar
    - --host=the-coffee-bar
    - --port=8082
    - --coffeemachine-host=the-coffee-machine
    - --coffeemachine-port=8083
    - --cashdesk-host=the-cashdesk
    - --cashdesk-port=8084
    env:
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: service.name=the-coffee-bar,application=the-coffee-bar-app,k8s.container.name=the-coffee-bar,k8s.deployment.name=sumologic-the-coffee-bar-bar,k8s.namespace.name=the-coffee-bar-ns,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=sumologic-the-coffee-bar-bar-5c48ff68f8
    - name: OTEL_SERVICE_NAME
      value: the-coffee-bar
    - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
      value: http://collection-sumologic-otelcol.sumologic:4318/v1/traces
    - name: PYTHONPATH
      value: /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation
    - name: OTEL_TRACES_EXPORTER
      value: otlp_proto_http
    - name: OTEL_APPLICATION_NAMESPACE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: OTEL_PROPAGATORS
      value: tracecontext,baggage,b3,xray
    - name: OTEL_TRACES_SAMPLER
      value: always_on
    image: public.ecr.aws/sumologic/the-coffee-bar:pythonAppsv0.2.0-ot1.9.1-0.28b1
    imagePullPolicy: Always
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - -e
        - |
          ps aux | grep python | grep -v grep
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    name: the-coffee-bar
    ports:
    - containerPort: 8082
      name: http
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /bin/sh
        - -c
        - -e
        - |
          ps aux | grep python | grep -v grep
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 250m
        memory: 400Mi
      requests:
        cpu: 100m
        memory: 50Mi
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: tls-certificate
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: sumologic-the-coffee-bar-token-tqthb
      readOnly: true
    - mountPath: /otel-auto-instrumentation
      name: opentelemetry-auto-instrumentation
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: the-coffee-bar
  initContainers:
  - command:
    - cp
    - -a
    - /autoinstrumentation/.
    - /otel-auto-instrumentation/
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.28b1
    imagePullPolicy: IfNotPresent
    name: opentelemetry-auto-instrumentation
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /otel-auto-instrumentation
      name: opentelemetry-auto-instrumentation
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: sumologic-the-coffee-bar-token-tqthb
      readOnly: true
  nodeName: ip-10-0-66-234.us-west-2.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: sumologic-the-coffee-bar
  serviceAccountName: sumologic-the-coffee-bar
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: tls-certificate
    secret:
      defaultMode: 420
      secretName: collection-sumologic-otelcol.sumologic-tls
  - name: sumologic-the-coffee-bar-token-tqthb
    secret:
      defaultMode: 420
      secretName: sumologic-the-coffee-bar-token-tqthb
  - emptyDir: {}
    name: opentelemetry-auto-instrumentation
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-04-07T09:59:37Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-04-07T09:59:56Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-04-07T09:59:56Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-04-07T09:59:35Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://8693f413ae34f1c2865acf81d070b64bd8fecb4e9f78eb33955d3258e2c2ae6c
    image: public.ecr.aws/sumologic/the-coffee-bar:pythonAppsv0.2.0-ot1.9.1-0.28b1
    imageID: docker-pullable://public.ecr.aws/sumologic/the-coffee-bar@sha256:46645a8acab222921d53df017f185b4f91e825f8f338b5c5b2336a3b30fdc236
    lastState: {}
    name: the-coffee-bar
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-04-07T09:59:38Z"
  hostIP: 10.0.66.234
  initContainerStatuses:
  - containerID: docker://8ff23d3051daf1cc351f607a45c0e5c591dae51de4df35e21f3f162722181c17
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.28b1
    imageID: docker-pullable://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python@sha256:0423f65cae67d18d9a3de5ff94ce46bd4f6a4c9c6d838b8fdb2b8194355ed1e6
    lastState: {}
    name: opentelemetry-auto-instrumentation
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://8ff23d3051daf1cc351f607a45c0e5c591dae51de4df35e21f3f162722181c17
        exitCode: 0
        finishedAt: "2022-04-07T09:59:37Z"
        reason: Completed
        startedAt: "2022-04-07T09:59:37Z"
  phase: Running
  podIP: 10.0.80.232
  podIPs:
  - ip: 10.0.80.232
  qosClass: Burstable
  startTime: "2022-04-07T09:59:35Z"

[mat-rumian]

Hey @pavolloffay ,
any updates on this issue? :) Is there something I can help with?

[mat-rumian]

The main reason for invalid attribute values (k8s.node.name and k8s.pod.name) is the order in which they are configured in the environment variable list. OTEL_RESOURCE_ATTRIBUTES which is using other environment variables has to be last on the list of environment variables.