Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update: Add support for x_forwarded_for headers in apaches access logs #3251

Merged
merged 17 commits into from
Aug 29, 2022
Merged

Update: Add support for x_forwarded_for headers in apaches access logs #3251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c953563
Update: Add support for x_forwarded_for headers in apaches access logs
redcinelli May 2, 2022
0058ab5
Update: Pr link in the changelogs
redcinelli May 2, 2022
28ed558
Update: Painless check property exist before referencing it
redcinelli May 2, 2022
dbbe9a6
Update: simplify grok expression
redcinelli May 16, 2022
986bd02
Update: x-forwarded-for match only ip
redcinelli May 17, 2022
5904f15
Update: migrate ip adress string analysis to CIDR
redcinelli May 17, 2022
16579c1
Fix: support for vhost at the begining of the log
redcinelli May 17, 2022
96f1e32
Update: refactor try/catch, ctx can't be null, remove redundancy
redcinelli May 17, 2022
ac90936
Update: Updated format for apache logs
redcinelli May 18, 2022
3e5c9f8
Update: Fill client.ip instead of source.address
redcinelli May 23, 2022
73ecf83
Update: Add some documentation to explain supported format
redcinelli Jun 3, 2022
15d0614
Update packages/apache/_dev/build/docs/README.md
redcinelli Jun 7, 2022
1047981
Update: Build integration to generate Md doc
redcinelli Jun 7, 2022
499515f
Update: manifest.yml
redcinelli Jun 7, 2022
ded37f9
Merge branch 'main' into apache/support_x_forwarded_for_headers
redcinelli Jul 29, 2022
2c91c78
Fix: issue during merge
redcinelli Jul 29, 2022
3c9d133
Merge branch 'main' into apache/support_x_forwarded_for_headers
redcinelli Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions packages/apache/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,24 @@ Access logs collects the Apache access logs.

{{fields "access"}}

Supported format for the access logs are:

- [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format)
- Defined in apache `LogFormat` by :
>```%h %l %u %t \"%r\" %>s %b```
- Example:
> `127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326`
- Combined Log Format
- Defined in apache `LogFormat` by:
>```%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"```
- Example:
>```127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://datawarehouse.us.oracle.com/datamining/contents.htm" "Mozilla/4.7 [en] (WinNT; I)"```
- Combined Log Format + X-Forwarded-For header
- Defined in apache `LogFormat` by:
>```%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" X-Forwarded-For=\"%{X-Forwarded-For}i\"```
- Example:
>```127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://datawarehouse.us.oracle.com/datamining/contents.htm" "Mozilla/4.7 [en] (WinNT; I)" X-Forwarded-For="10.225.192.17, 10.2.2.121"```

### Error Logs

Error logs collects the Apache error logs.
Expand Down
4 changes: 2 additions & 2 deletions packages/apache/_dev/deploy/docker/httpd.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ LogLevel warn
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" X-Forwarded-For=\"%{X-Forwarded-For}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common

<IfModule logio_module>
Expand All @@ -296,7 +296,7 @@ LogLevel warn
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
CustomLog "/usr/local/apache2/logs/access.log" common
CustomLog "/usr/local/apache2/logs/access.log" combined

#
# If you prefer a logfile with access, agent, and referer information
Expand Down
5 changes: 5 additions & 0 deletions packages/apache/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.6.0"
changes:
- description: Add support for x_forwarded_for header
type: enhancement
link: https://github.com/elastic/integrations/pull/3251
- version: "1.5.1"
changes:
- description: Remove unused visualizations
Expand Down
5 changes: 4 additions & 1 deletion packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,7 @@
172.17.0.1 - - [29/May/2017:19:02:48 +0000] "GET /stringpatch HTTP/1.1" 404 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" "-"
monitoring-server - - [29/May/2017:19:02:48 +0000] "GET /status HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" "-"
127.0.0.1 - - [02/Feb/2019:05:38:45 +0100] "-" 408 152 "-" "-"
monitoring-server - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" "-"
monitoring-server - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="-"
89.160.20.112 - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="10.0.0.2,10.0.0.1"
ishleenk17 marked this conversation as resolved.
Show resolved Hide resolved
2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6 - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="10.225.192.17, 10.2.2.121"
monitoring-server - - [17/May/2022:21:41:43 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.79.1" X-Forwarded-For="192.168.0.2"
Loading