Include_labels option in add_resource_metadata not working as expected

[gizas]

Description:
According to documentation users can configure include_labels option to specify and filter the labels being ingested.

The include_labels option can be configured under namespace and node resources, implying that users can configure seperately the labels they want per kind of resource

See eg.

add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
          #labels.dedot: true
          #annotations.dedot: true
        node:
          include_labels: ["nodelabel2"]
         ....

But the configuration of include_labels under namespace leads to node labels being lost.
All versions of 8.10.2 and above have been verified that showcase above functionality.

Details:

• Version of Elastic Agent: 8.10.2
• Version of elastic-agent-autodiscover: v0.6.2[https://github.com/elastic/elastic-agent/blob/1320210da633a0e00da6f5301fcc39c6de29bca8/go.mod#L16]

Steps to reproduce:

1. Install Elastic Stack 8.10.2
2. Install a Kubernetes Integration Policy with Kube-state-metrics datastreams enabled
3. Add a label to your namespace:

kubectl label namespace kube-system katsoulis=pasole

kubectl  describe namespace kube-system
Name:         kube-system
Labels:       katsoulis=pasole
              kubernetes-1=pasole
              kubernetes.io/metadata.name=kube-system

5. With the default configuration you will notice that all namespace and node labels are being collected
   

6. Edit the advanced options of state_pod dataset inside your Kubernetes Integration Policy
   

7. You will notice that include_labels condition applies to both nodes and namespace metadata, resulting to node labels being lost
   

[gizas]

cc @bturquet for prioritisation

[tetianakravchenko]

some findings:
I've tested it with simple metricbeat configuration - only 1 metricset (pod), such as:

    - module: kubernetes
      add_metadata: true
      add_resource_metadata:
        namespace:
          include_labels: ["labelName"]
      metricsets:
        - pod
....

Already after the config unpacking - UnpackConfig the configuration is wrong - Node part contains the include_labels (Namespace as well):

If add the same label to the node kubectl label node test-control-plane labelName=labelValue - kubernetes.node.labels.labelName will be added.

UnpackConfig calls the Unpack on the rawConfig - which is correct, then is called Unpack from the go-ucfg lib

This issue exists starting from the 17.17.7 (the version it was backported to - relevant PR). Seems that elastic/beats#29133 was not working as expected at least for the k8s module, will check it also for the add_kubernetes_metadata processor and the kubernetes provider

[tetianakravchenko]

configuration add_resource_metadata

1. k8s module - from the time it was introduced this was never working as expected.
2. add_kubernetes_metadata processor.
   tested filebeat with:

processors:
        - add_kubernetes_metadata:
            add_resource_metadata:
              namespace:
                include_labels: ["labelName"]
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

In https://github.com/elastic/beats/pull/29133/files was removed ..., in https://github.com/elastic/beats/pull/29329/files AddResourceMetadata was added to the default config.

So before the changes above, configuration contained only the values defined in the customer config, default values are not defined:

I've also tested the version 7.17.6 - defining add_resource_metadata leads to the full overwrite of the default value (hence the node labels are missing). so what is the expected behavior - merge or replace?

After the changes above - default values are initialized

and merged during the unpacking in https://github.com/elastic/go-ucfg/blob/main/reify.go#L434

Seems there is some issue during the merging.
I think the right approach would be to check the merge issue, or other option - use the replace merge strategie.

Update: replace merge strategie does not lead to the replacement: include_labels will be duplicated in both node and namespace

3. autodiscover - TODO

[tetianakravchenko]

related PRs:

• Fix default resource metadata config elastic-agent-autodiscover#72
• Upgrade elastic-agent-autodiscover dependency; use InitDefaults() to init default configuration beats#37458
• elastic-agent Bump elastic-agent-autodiscover version #3938

add_resource_metadata processor does not work for metricbeat

• delete add_kubernetes_metadata processor from documentation - it never worked as expected(?), recommended way - to use enricher

[tetianakravchenko]

@gizas FYI: I've tested locally #3938:

1. pod datastream add_resource_metadata config:

2. state_pod datastream add_resource_metadata config:

[tetianakravchenko]

Another issue:

• provider.kubernetes for elastic-agenthas still the same issue

Update: no issues there, see comment #3346 (comment)