Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add filebeat module for server access logs to use s3 filebeat input #12880

Closed
3 tasks
kaiyan-sheng opened this issue Jul 11, 2019 · 1 comment · Fixed by #13170
Closed
3 tasks
Assignees
Labels
Filebeat Filebeat Team:Integrations Label for the Integrations team v7.4.0

Comments

@kaiyan-sheng
Copy link
Contributor

kaiyan-sheng commented Jul 11, 2019

After s3 input is added for filebeat, we can start using the logs retrieved by s3 input and create modules to parse them. This issue is to add module for S3 server access logs. These logs provides detailed records for the requests that are made to a bucket. They are useful for many applications. For example, access log information can be useful in security and access audits. It can also help you learn about your customer base and understand your Amazon S3 bill.

One s3 server access log file can include multiple log records. Each record represents one request and consists of space-delimited fields. For example:

79a59df900b9 awsexamplebucket [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79a59df900b9 3E57427F3EXAMPLE REST.GET.VERSIONING - "GET /awsexamplebucket?versioning HTTP/1.1" 200 - 113 - 7 - "-" "S3Console/0.4" - s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader awsexamplebucket.s3.amazonaws.com TLSV1.1

Filebeat module checklist

  • Test log files exist for the grok patterns
  • Generated output for at least 1 log file exists
  • Dashboard
@kaiyan-sheng kaiyan-sheng self-assigned this Jul 11, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team Filebeat Filebeat labels Jul 11, 2019
@exekias
Copy link
Contributor

exekias commented Jul 12, 2019

these look really similar to apache/nginx logs! plenty of space for a great dashboard here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat Team:Integrations Label for the Integrations team v7.4.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants