Filebeat 6.2.1 massive DOS on Kubernetes Apiserver

[towolf]

Today, for no particular visible reason, one filebeat instance started a request DOS on our 3 apiservers.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1beta1",
  "metadata": {
    "creationTimestamp": "2018-03-12T13:44:46Z"
  },
  "level": "Request",
  "timestamp": "2018-03-12T13:44:46Z",
  "auditID": "6fe10401-ceb8-401a-90ed-703c667b7baf",
  "stage": "ResponseStarted",
  "requestURI": "/api/v1/pods?fieldSelector=spec.nodeName%3Ds858&resourceVersion=35649380&watch=true",
  "verb": "watch",
  "user": {
    "username": "system:serviceaccount:kube-system:filebeat",
    "uid": "788801b4-f61b-11e7-8071-00259008c682",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "10.216.0.16"
  ],
  "objectRef": {
    "resource": "pods",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2018-03-12T13:44:46.085940Z",
  "stageTimestamp": "2018-03-12T13:44:46.086197Z"
}

The requestURI was the same throughout this episode:
"/api/v1/pods?fieldSelector=spec.nodeName%3Ds858&resourceVersion=35649380&watch=true"

Millions of times. The same request from the same filebeat instance.

For confirmed bugs, please report:

• Version: 6.2.1
• Operating System: Ubuntu 16.04 amd64 / Kubernetes v1.9.1
• Steps to Reproduce: not sure

[towolf]

At some point this took out one of the masters in the cluster. So, not pretty at all.

[exekias]

Hi Tobias, I'm afraid you hit #6503, this issue has been fixed and next release (6.2.3) will include the fix.

[exekias]

I forgot to mention that we have introduced backoff mechanism to our Kubernetes watcher code that should prevent this from happening again with any other future error.

For convenience, I've pushed a snapshot build with the fix: exekias/filebeat:6.2.3-snapshot

I'm closing this as it's a fixed issue

[yawboateng]

@exekias thanks for the snapshot build. when should we expect the 6.2.3 release? also have the same issue with masters going offline

[exekias]

We don't publish release dates, but release often. You can expect 6.2.3 in the following week(s)

[towolf]

Thanks for the info @exekias

I switched to your snapshot, because it cannot get worse.

Also wondering what the ETA of the release is ...

[exekias]

I'm sorry for the inconvenience, thank you for your reports

[yawboateng]

Can confirm the issue is resolved in the 6.2.3 snapshot build from @exekias. running for 3 days now with no issues 👍

[towolf]

Same, 0 Restarts on all our nodes, and no more infinite request spam.

Great work.

[exekias]

An update on this, new images for 6.2.3 are already available: https://www.docker.elastic.co/

[towolf]

Thanks for the notice. I've bumped the daemonset to the public release.