Skip to content

Commit

Permalink
Convert Filebeat haproxy.log to ECS (elastic#9117)
Browse files Browse the repository at this point in the history
- Convert many fields under `haproxy.*` to ECS. Previous field names are field aliases towards the new corresponding ECS field:
  - haproxy.client.port => source.port
  - haproxy.process_name => process.name
  - haproxy.pid => process.pid
  - haproxy.destination.ip => destination.ip
  - haproxy.destination.port => destination.port
- Convert pid and ports to integers in the event body. They used to be strings.
  • Loading branch information
webmat authored Nov 22, 2018
1 parent 04c951d commit f6bbfd5
Show file tree
Hide file tree
Showing 9 changed files with 255 additions and 196 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ https://github.com/elastic/beats/compare/v6.5.0...v7.0.0-alpha1[View commits]
- Rename many `system.syslog.*` fields to map to ECS. {pull}9135[9135]
- Rename many `iis.access.*` fields to map to ECS. {pull}9084[9084]
- IIS module's user agent string is no longer encoded (`+` replaced with spaces). {pull}9084[9084]
- Rename many `haproxy.*` fields to map to ECS. {pull}9117[9117]
*Metricbeat*
Expand Down
55 changes: 55 additions & 0 deletions dev-tools/ecs-migration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -197,3 +197,58 @@
to: http.request.referrer
alias: true
copy_to: false

- from: haproxy.client.port
to: source.port
alias: true
copy_to: false

- from: haproxy.process_name
to: process.name
alias: true
copy_to: false

- from: haproxy.pid
to: process.pid
alias: true
copy_to: false

- from: haproxy.destination.ip
to: destination.ip
alias: true
copy_to: false

- from: haproxy.destination.port
to: destination.port
alias: true
copy_to: false

- from: haproxy.geoip.continent_name
to: source.geo.continent_name
alias: true
copy_to: false

- from: haproxy.geoip.country_iso_code
to: source.geo.country_iso_code
alias: true
copy_to: false

- from: haproxy.geoip.location
to: source.geo.location
alias: true
copy_to: false

- from: haproxy.geoip.region_name
to: source.geo.region_name
alias: true
copy_to: false

- from: haproxy.geoip.city_name
to: source.geo.city_name
alias: true
copy_to: false

- from: haproxy.geoip.region_iso_code
to: source.geo.region_iso_code
alias: true
copy_to: false
196 changes: 90 additions & 106 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3393,66 +3393,6 @@ haproxy Module
[float]
== destination fields
Destination information
*`haproxy.destination.port`*::
+
--
type: long
Port of the destination host
--
*`haproxy.destination.ip`*::
+
--
IP of the destination host
--
*`haproxy.process_name`*::
+
--
Name of the process
--
*`haproxy.pid`*::
+
--
type: long
PID of the process
--
[float]
== client fields
Information about the client doing the request
*`haproxy.client.ip`*::
+
--
IP address of the client which initiated the TCP connection to haproxy.
--
*`haproxy.client.port`*::
+
--
type: long
TCP port of the client which initiated the connection.
--
*`haproxy.frontend_name`*::
+
--
Expand Down Expand Up @@ -3562,131 +3502,175 @@ The HAProxy source of the log
--
*`haproxy.termination_state`*::
+
--
Condition the session was in when the session ended.
--
*`haproxy.mode`*::
+
--
type: text
mode that the frontend is operating (TCP or HTTP)
--
[float]
== geoip fields
== connections fields
Contains various counts of connections active in the process.
Contains GeoIP information gathered based on the client.ip field. Only present if the GeoIP Elasticsearch plugin is available and used.
*`haproxy.connections.active`*::
+
--
type: long
Total number of concurrent connections on the process when the session was logged.
--
*`haproxy.geoip.continent_name`*::
*`haproxy.connections.frontend`*::
+
--
type: keyword
type: long
Name of the continent.
Total number of concurrent connections on the frontend when the session was logged.
--
*`haproxy.geoip.country_iso_code`*::
*`haproxy.connections.backend`*::
+
--
type: keyword
type: long
Country ISO code.
Total number of concurrent connections handled by the backend when the session was logged.
--
*`haproxy.geoip.location`*::
*`haproxy.connections.server`*::
+
--
type: geo_point
type: long
Represents a geopoint with the longitude and latitude.
Total number of concurrent connections still active on the server when the session was logged.
--
*`haproxy.geoip.region_name`*::
*`haproxy.connections.retries`*::
+
--
type: keyword
type: long
Name of the region
Number of connection retries experienced by this session when trying to connect to the server.
--
*`haproxy.geoip.city_name`*::
[float]
== client fields
Information about the client doing the request
*`haproxy.client.ip`*::
+
--
type: keyword
IP address of the client which initiated the TCP connection to haproxy.
If connection is via unix socket, socket path is in this field.
City name.
--
*`haproxy.geoip.region_iso_code`*::
*`source.port`*::
+
--
type: keyword
type: alias
--
ISO code of the region
*`process.name`*::
+
--
type: alias
--
*`haproxy.termination_state`*::
*`process.pid`*::
+
--
Condition the session was in when the session ended.
type: alias
--
[float]
== connections fields
== destination fields
Contains various counts of connections active in the process.
Destination information
*`haproxy.connections.active`*::
*`destination.port`*::
+
--
type: long
Total number of concurrent connections on the process when the session was logged.
type: alias
--
*`haproxy.connections.frontend`*::
*`destination.ip`*::
+
--
type: long
Total number of concurrent connections on the frontend when the session was logged.
type: alias
--
*`haproxy.connections.backend`*::
[float]
== geoip fields
Contains GeoIP information gathered based on the client.ip field. Only present if the GeoIP Elasticsearch plugin is available and used.
*`source.geo.continent_name`*::
+
--
type: long
Total number of concurrent connections handled by the backend when the session was logged.
type: alias
--
*`haproxy.connections.server`*::
*`source.geo.country_iso_code`*::
+
--
type: long
Total number of concurrent connections still active on the server when the session was logged.
type: alias
--
*`haproxy.connections.retries`*::
*`source.geo.location`*::
+
--
type: long
type: alias
Number of connection retries experienced by this session when trying to connect to the server.
--
*`source.geo.region_name`*::
+
--
type: alias
*`haproxy.mode`*::
--
*`source.geo.city_name`*::
+
--
type: text
type: alias
mode that the frontend is operating (TCP or HTTP)
--
*`source.geo.region_iso_code`*::
+
--
type: alias
--
Expand Down
2 changes: 1 addition & 1 deletion filebeat/include/fields.go

Large diffs are not rendered by default.

Loading

0 comments on commit f6bbfd5

Please sign in to comment.