Traefik module for Filebeat is not working as expected

[e8kor]

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.

For confirmed bugs, please report:

• Version: 6.1.1
• Operating System:
  Linux 4.4.0-31-generic [question] Failed to publish topology #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Steps to Reproduce:

Traefik access log example:

94.254.131.115 - - [19/Jan/2018:10:01:02 +0000] "GET /assets/52f8f2e711d235d76044799e/owners?oauth_token=ya29.GltABOXd_gtG-XVvYX2YhxXJiXVvbHRMXn9fbzc_mDfl2rDhqK0CrAlwuwwRWnNnEaMDwkmyI7-QGbRSB0Hzje2cc__FjTQ1iuiYTSIBaIPfxSWip5jx6zqvsVVo HTTP/1.1" 200 85 - "Android" 623112 "Host-api-wearerealitygames-com-2" "http://172.25.0.9:4140" 13ms
89.64.35.193 - - [19/Jan/2018:10:01:02 +0000] "GET /marketplace/tax?oauth_token=ya29.Gl0fBWnrJ7DcEU-tN-O3Vxn2XZVaz2I-hFTjP1JQzhYFVT-SKtlmo9hSzrx3n82LUwUxJ1s5lmU8U3Mc9gA_aCxBk49ShYEwvmYOWxJJyldDIJ7hY4us4LoiSY1OqAM HTTP/1.1" 200 150 - "Android" 623114 "Host-api-wearerealitygames-com-2" "http://172.25.0.6:4140" 8ms

Grok pattern:
link: https://github.com/elastic/beats/blob/master/filebeat/module/traefik/access/ingest/pipeline.json#L7
pattern:
%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP%{NUMBER:traefik.access.http_version}" %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( "%{DATA:traefik.access.referrer}")?( "%{DATA:traefik.access.agent}")?(?:%{NUMBER:traefik.access.request_count}|-)?( "%{DATA:traefik.access.frontend_name}")?( "%{DATA:traefik.access.backend_url}")?

Resut:
error: field [agent] not present as part of path [traefik.access.agent]

From what I understand is that current pattern is not working properly its not parsing data since user_agent field.

here is example of pattern that works for me:
initial (pattern for support of old version of traefik):
%{IPORHOST:traefik.access.remote_ip} - (?:-|"%{DATA:traefik.access.user_name}") \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) "(?:-|%{DATA:traefik.access.referrer})" "(?:-|%{DATA:traefik.access.agent})" (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|"%{DATA:traefik.access.frontend_name}") (?:-|"%{DATA:traefik.access.backend_url}") %{NUMBER:traefik.access.duration:int}ms
fallback (pattern for support of new version of traefik):
%{IPORHOST:traefik.access.remote_ip} - (?:-|"%{DATA:traefik.access.user_name}") \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) (?:-|"%{DATA:traefik.access.referrer}") (?:-|"%{DATA:traefik.access.agent}") (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|"%{DATA:traefik.access.frontend_name}") (?:-|"%{DATA:traefik.access.backend_url}") %{NUMBER:traefik.access.duration:int}ms

patterns are based on: https://github.com/containous/traefik/blob/5140bbe99a79b45f98c27fbb8e9b6833194af4cb/middlewares/accesslog/logger_formatters.go

related discussion: https://discuss.elastic.co/t/traefik-module-for-filbeat-dont-parse-traefik-logs/116170/3

Pipeline that works for me

pipeline.json

{
  "description": "Pipeline for parsing Traefik access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "%{IPORHOST:traefik.access.remote_ip} - (?:-|\"%{DATA:traefik.access.user_name}\") \\[%{HTTPDATE:traefik.access.time}\\] \"%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}\" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) \"(?:-|%{DATA:traefik.access.referrer})\" \"(?:-|%{DATA:traefik.access.agent})\" (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|\"%{DATA:traefik.access.frontend_name}\") (?:-|\"%{DATA:traefik.access.backend_url}\") %{NUMBER:traefik.access.duration:int}ms",
        "%{IPORHOST:traefik.access.remote_ip} - (?:-|\"%{DATA:traefik.access.user_name}\") \\[%{HTTPDATE:traefik.access.time}\\] \"%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}\" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) (?:-|\"%{DATA:traefik.access.referrer}\") (?:-|\"%{DATA:traefik.access.agent}\") (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|\"%{DATA:traefik.access.frontend_name}\") (?:-|\"%{DATA:traefik.access.backend_url}\") %{NUMBER:traefik.access.duration:int}ms"
      ],
      "ignore_missing": true
    }
  },{
    "remove":{
      "field": "message",
      "ignore_failure": true
    }
  }, {
    "rename": {
      "field": "@timestamp",
      "target_field": "read_timestamp"
    }
  }, {
    "date": {
      "field": "traefik.access.time",
      "target_field": "@timestamp",
      "formats": ["dd/MMM/YYYY:H:m:s Z"]
    }
  }, {
    "remove": {
      "field": "traefik.access.time",
      "ignore_failure": true
    }
  }, {
    "user_agent": {
      "field": "traefik.access.agent",
      "target_field": "traefik.access.user_agent",
      "ignore_failure": true
    }
  }, {
    "remove": {
      "field": "traefik.access.agent",
      "ignore_failure": true
    }
  }, {
    "geoip": {
      "field": "traefik.access.remote_ip",
      "target_field": "traefik.access.geoip"
    }
  }],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

[ruflin]

@e8kor Which version of Traefik are you using? Can you open a PR with the change so we support both options?

[e8kor]

hi, I'm using 1.4.6 and I believe that they will fix logging in 1.5.0. Pull request is coming.

[alastairs]

I came across this issue after hitting an error running filebeat with the traefik module against traefik 1.6 on Kubernetes:

field [traefik] not present as part of path [traefik.access.message]

Is the traefik module currently broken? I see the PR is somewhat stalled; is there anything I can do to help it along?

[ruflin]

Seems like we have 2 open PR's which could be related to this:

• Add traefik 1.5 formatted logs and update grok rule to handle them #6488
• fix grok expression for traefik #6136

We should push these forward to make it into the code.

[sayden]

Fixed in #8768