[filebeat] Failed to parse kubernetes.labels.app

[dmitryzykov]

I'm using helm/stable/filebeat which is based on docker.elastic.co/beats/filebeat-oss:6.4.2
all other components are also oss:6.4.2

I'm having a problem when filebeat failed to parse kubernetes.labels.app for many containers.

This is filebeat log when I'm sending directly to elasticsearch:

2018-10-27T09:42:23.923Z	WARN	elasticsearch/client.go:520	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x12a6762a, ext:63676230135, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"version":"6.4.2", "name":"filebeat-tx7fz", "hostname":"filebeat-tx7fz"}, "offset":3227, "stream":"stderr", "message":"   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)", "kubernetes":common.MapStr{"container":common.MapStr{"name":"sink"}, "namespace":"default", "replicaset":common.MapStr{"name":"test-propertysite-sink-78549f4bb7"}, "labels":common.MapStr{"pod-template-hash":"3410590663", "app":common.MapStr{"kubernetes":common.MapStr{"io/name":"propertysite", "io/component":"sink", "io/instance":"test"}}}, "pod":common.MapStr{"name":"test-propertysite-sink-78549f4bb7-rzj84"}, "node":common.MapStr{"name":"aks-nodepool1-38193062-2"}}, "host":common.MapStr{"name":"filebeat-tx7fz"}, "meta":common.MapStr{"cloud":common.MapStr{"provider":"az", "instance_id":"0fdf45d7-7bfa-4f34-b5e6-9edc8dde83e6", "instance_name":"aks-nodepool1-38193062-2", "machine_type":"Standard_DS2_v2", "region":"westeurope"}}, "source":"/var/lib/docker/containers/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231-json.log", "prospector":common.MapStr{"type":"docker"}, "input":common.MapStr{"type":"docker"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4203b3040), Source:"/var/lib/docker/containers/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231-json.log", Offset:3464, Timestamp:time.Time{wall:0xbeed2a9fb6313090, ext:224243617813, loc:(*time.Location)(0x1f61860)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfe57e, Device:0x801}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [kubernetes.labels.app]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:419"}}

and this is the same error from logstash, when I'm sending from filebeat to logstash

[2018-10-26T19:38:14,673][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2018.10.26", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x21da5026>], :response=>{"index"=>{"_index"=>"filebeat-2018.10.26", "_type"=>"doc", "_id"=>"Ch3isWYBrdoKcExWXgp1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [kubernetes.labels.app]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:695"}}}}}

This is a part of my filebeat config file:

    filebeat.prospectors:
    - enabled: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - '*'
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            equals:
              kubernetes.container.name: filebeat
      type: docker

I noticed that parsing problems appear only for pods with such labels format(which is recommended for helm charts:

  labels:
    app.kubernetes.io/name: foo
    helm.sh/chart: foo-chart
    app.kubernetes.io/instance: foo
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/component: foo

and the problem doesn't appear for more simpler label format like:

  labels:
    name: bar
    chart: bar-chart
    instance: bar
    managed-by: Tiller
    component: bar

Expected behavior from filebeat is a successful parse of such labels and successful shipment to elasticsearch instead parse error and failure to send log.

For example Fluentd parse app.kubernetes.io/component and it appears in elasticsearch as kubernetes.labels.app_kubernetes_io/component

During bug research, I found similar problems

• https://discuss.elastic.co/t/mapping-for-kubernetes-labels-app-tried-to-parse-field-app-as-object-but-found-a-concrete-value/146490
• https://stackoverflow.com/questions/52760676/elasticsearch-error-kubernetes-labels-app

[lesaux]

I'm seeing this as well.

[wwyhy]

me too, is there any solutions? thanks

[exekias]

Hi everyone, thank you for your detailed report

This issue is caused by label/annotation dots (.) creating hierarchy in Elasticsearch documents. If an annotation has an structure like this: example.annotation.data: some-data and a different one contains example.anottation: value their mapping will conflict, as example.annotation is both an object and a keyword.

We have some counter measures in place to detect these situations when the annotations belong to the same object, but it seems it's not enough.

My thinking goes toward allowing replacing . with _. I will open a PR to do so (as a config flag initially)

In the meanwhile, the workaround would be to drop the offending annotations. You can use the drop_fields processor to do so:

processors:
  - drop_fields:
      when:
        has_fields: ['kubernetes.labels.app']
      fields:
        - 'kubernetes.labels.app'

[lesaux]

Thank you! this fix works in the meantime!

[kuzm1ch]

Hello, this workaround is also helped me, but I have almost the same problem with basic kubernetes component:
quay.io/calico/node:v2.6.7
quay.io/calico/cni:v1.11.2
quay.io/calico/kube-controllers:v1.0.3
and etc.

metricbeat: 6.4.2
kubernetes: v1.10.6
kube-state-metrics: k8s.gcr.io/addon-resizer:1.8.3

kubernetes.labels.role: 
{
  "kubernetes": {
    "io/networking": "1"
  }
}

(I think this metric should be like "kubernetes.labels.role.kubernetes.io/networking: 1" )
and therefore in some case it is parse as object, in anather as text
also create new issue related this problem
#8987

[choppedpork]

Oddly enough, I have two separate clusters with the same version (6.4.2) and identical (as far as filebeat.yml is concerned) config and this issue is only occurring in one of them.

In my case, this issue is only occurring for logs which are being parsed as JSON using json.keys_under_root.

[justinfiore]

We see this issue also with labels like:

labels:
    app.kubernetes.io/name: foo
    app.kubernetes.io/instance: foo
    app.kubernetes.io/component: foo

Is there a fix for this other than dropping the labels?

[sco11morgan]

@exekias Looking forward to your fix for label dots. Maybe like the Docker fix with labels.dedot

[dpankros]

Also getting this, but I'm seeing it primarily in metricbeat.

I just want to add that this label format is what is recommended by the Kubernetes team.

[exekias]

I've opened #9860 to tackle this issue, thank you everyone for your feedback

[vorozhko]

Hello @exekias,
I have installed filebeat using deploy/kubernetes templates and the issue still persist in 6.6.1.
{"type":"mapper_parsing_exception","reason":"failed to parse [kubernetes.labels.app]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:301"}
Dropping field didn't fix issue as well in new version, but was working with older version 6.5.

However the messages was logged in to ES.

[maggieghamry]

Hi @exekias , I am encountering the same error @vorozhko mentioned above, in 6.6.1. I see that 6.7.0 was just released today, but didn't explicitly see this fix in 6.7.0. Did it get merged in by chance, and I missed it? Should I expect an upgrade to 6.7.0 to fix this without a workaround?

[willejs]

The fix for metric beat made it in to 6.7.0, which is not enabled by default. I didn't read the code correctly and it does not fix filebeat at all from what I can tell. I think filebeat needs the config options exposed in the add_kubernetes_metadata processor?

[exekias]

Hi everyone, you should be able to enable the new behavior by passing these parameters:

processors:
 -  add_kubernetes_metadata:
      labels.dedot: true
      annotations.dedot: true

This will be the default starting with 7.0

[exekias]

For reference, this was fixed here: #9939, closing this

[Dr-kyle]

I think you can set elasticsearch index mapping or template to resolve this question, for example:
PUT your_index
{
"mappings": {
"_doc": {
"properties": {
"kubernetes": {
"properties": {
"labels": {
"properties": {
"app": {
"type": "object",
"enabled": false
}
}
}
}
}
}
}
}
}

[nerddelphi]

Hi everyone. That issue wasn't fixed at all. I'm getting the same behavior, even using the latest version of metricbeat (7.1.1) and config:

processors:
 -  add_kubernetes_metadata:
      labels.dedot: true
      annotations.dedot: true

[kaiyan-sheng]

@nerddelphi Thanks for the feedback! What error message are you seeing?

[nerddelphi]

@kaiyan-sheng These errors:

{"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id '6OPyamsBGJCb4HgtJtQp'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:264"}}

and

{"type":"mapper_parsing_exception","reason":"object mapping for [kubernetes.labels.app] tried to parse field [app] as object, but found a concrete value"}

[Dave-c-Ross]

Same here

[senthil13]

Same for me too, with latest 7.2 version

(status = 400): {
	"type": "mapper_parsing_exception",
	"reason": "failed to parse field [kubernetes.pod._module.labels.app] of type [keyword] in document with id 'DJv34WsBh5Hn_TLey_iS'",
	"caused_by": {
		"type": "illegal_state_exception",
		"reason": "Can't get text on a START_OBJECT at 1:673"
	}
}

[kaiyan-sheng]

Is labels.dedot set to be true in the config?

[senthil13]

Where to add ? i am using this - https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml

[kaiyan-sheng]

Ok you are using metricbeat, so for each k8s module entry, add labels.dedot: true like:

- module: kubernetes
  labels.dedot: true
  metricsets:
  ...

[dekimpew]

I can confirm this works with the 7.1 metricbeat kubernetes.yml deployment.

[bcouetil]

Hello 😄

I have the problem in version 6.8.1 with filebeat and the labels/annotations.dedot to true, is this expected in this version ?

[alastairs]

@bcouetil We found the same in v6.8.2. I'm now upgrading our Elastic stack to v7 in the hope this will resolve it...

[bcouetil]

I've done that successfully. Updating only filebeat works.

[alastairs]

Hmmm, I wasn’t able to run Filebeat 7 against Elastic 6. But thanks for confirming it’s properly fixed in v7! Things are looking happier on our end too ☺️

[vchan2002]

Just an FYI..., even adding the dedot doesn't necessary solve the issue 100%, as least not immediately. The problem seems to be that the kubernetes.labels.app, depending on which document makes it there 1st and establishing it as a keyword field or a hierarchy (as the daily index), will basic kick out the other kind... The key is that you'll see kubernetes.labels.app in Kibana as a "conflict" field in the index pattern, since some it might be one some day, it might be another another day.

I'd imagine the dedot will prevent this conflict when the daily index rolls over. Maybe I'm being impatient and not waiting to see what happens with the index rolling over on the next day.... I just added this at the bottom of the processor to get rid of that field.....

  - drop_fields:
      fields: ["kubernetes.labels.app"]

I think this describes the problem better....
https://discuss.elastic.co/t/filebeats-auto-mapping-of-kubernetes-labels-causing-big-issues/154718

[willemdh]

Hello, we have similar issues since 7.6.1 in metricbeat...

{"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.3scale] of type [keyword] in document

Etc...

[kaiyan-sheng]

@willemdh Did you add labels.dedot: true by any chance? If dedot is already enabled and you are still seeing this error, could you open a discuss forum ticket https://discuss.elastic.co/c/beats/filebeat please? Thanks!!

[willemdh]

@kaiyan-sheng Yes, we added dedot true a long time ago.. A colleague of me created a ticket in the meantime.