Update Filebeat Kubernetes manifests to use docker prospector

[exekias]

#5402 was released already, we should update Kubernetes manifests to take benefit from it.

This also fixes #5920

[zytek]

I can take it, but first problem that I noticed is that even though you configure container.path in docker prospector - filebeat still requires then not used 'paths' config option.

2017/12/20 12:37:25.099716 beat.go:635: CRIT Exiting: each prospector must have at least one path defined

should we drop this requirement?

[exekias]

Uhm, that should not happen when containers.ids config is provided, how did you reproduce that error?

[zytek]

Using this config map file:

  kubernetes.yml: |-
    - type: docker
      container.ids:
      - "*"
      #paths:
      #- "/var/lib/docker/containers/*/*.log"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}

produces error:

2017/12/20 13:00:23.425506 beat.go:635: CRIT Exiting: each prospector must have at least one path defined
Exiting: each prospector must have at least one path defined

what is important to note is that if I provide invalid but non empty "paths" config option then filebeat DOES NOT WORK as harvester is not tailing any log files. Seem that docker prospector doesn't properly configure harvester basing on it's own paths to follow.

2017/12/20 12:58:23.237201 prospector.go:111: DBG [prospector] File Configs: [/usr/share/filebeat/INVALID_PATH]
2017/12/20 12:58:23.237209 reload.go:258: INFO Starting 1 runners ...
2017/12/20 12:58:23.237216 prospector.go:87: INFO Starting prospector of type: docker; ID: 5082549983441638544
2017/12/20 12:58:23.237223 reload.go:265: DBG [cfgfile] New runner started: 5082549983441638544
2017/12/20 12:58:23.237229 reload.go:219: INFO Loading of config files completed.
2017/12/20 12:58:23.237244 prospector.go:147: DBG [prospector] Start next scan
2017/12/20 12:58:23.237273 prospector.go:168: DBG [prospector] Prospector states cleaned up. Before: 0, After: 0
2017/12/20 12:58:23.238493 podwatcher.go:83: INFO kubernetes: Watching API for pod events
2017/12/20 12:58:33.237484 prospector.go:124: DBG [prospector] Run prospector
2017/12/20 12:58:33.238232 prospector.go:147: DBG [prospector] Start next scan
2017/12/20 12:58:33.238271 prospector.go:168: DBG [prospector] Prospector states cleaned up. Before: 0, After: 0

Running now with proper path:

  kubernetes.yml: |-
    - type: docker
      container.ids:
      - "*"
      paths:
      - "/var/lib/docker/containers/*/*.log"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}

makes filebeat+docker prospector happy and my logs are streamed, with log events in "message" field.

[zytek]

let's fix that and I'll update default manifest

[exekias]

There is a typo there, container.ids -> containers.ids

[zytek]

Oh (facepalm) - works fine when I fixed the typo.

Anyway, there is definitely a UX problem here ;-) This should either complain about lack of containers.ids config option OR use a default '*' value and work out of the box, the hack I had to did with paths seems counter-intuitive upon lack of proper configuration.

Edit: documentation says that containers.ids is required but filebeat didn't complain about lack of it.

[exekias]

Agree with the UX issue, we have some ideas to fix that globally but nothing in place yet :), let me know if it works with the corrected field

[zytek]

It does work, gonna make PR later to change sample manifests.

[walktall]

btw, I think namespace: ${POD_NAMESPACE} is not used in code. Could we remove it from yaml?

[exekias]

You are correct, this was fixed with 6.1 too