Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move dissect tokenizing to Ingest pipeline in Traefik Filebeat Module #9434

Closed
sayden opened this issue Dec 7, 2018 · 2 comments
Closed
Assignees
Labels

Comments

@sayden
Copy link
Contributor

sayden commented Dec 7, 2018

Some background can be read at the end of this PR #8768 (comment)

Traefik Filebeat module relies on two steps to process the incoming logs: An initial step on the dissect part covered by this file that tokenizes the first 8 fields in a Traefik log message which follows Combined Log Format as you can see in the example written here

NOTE: Traefik docs have a mistake by saying that their logs are in Common instead of Combined Log Format but as you can see here the format that appears in the example in Traefik docs doesn't match but instead it matches with Combined format here)

The second step in tokenization any log line, involves the pipeline.json file which receives the output from the dissect, done within Filebeat itself and tokenizes everything that wasn't tokenized in the dissect part: fields 9th, corresponding to traefik.access.body_sent.bytes in any log format and forward.

The expected result we want is to tokenize all incoming content in one place, directly in the pipeline.json file to effectively use the Ingest node for tokenization.

@sayden sayden added enhancement module Filebeat Filebeat Team:Integrations Label for the Integrations team labels Dec 7, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/infrastructure

@sayden
Copy link
Contributor Author

sayden commented Feb 4, 2019

Fixed here #10442

@sayden sayden closed this as completed Feb 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants