Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove convert_timezone option from filebeat system module #12410

Merged
merged 3 commits into from
Jun 6, 2019
Merged

Remove convert_timezone option from filebeat system module #12410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ae43ca3
Remove `convert_timezone` option from filebeat system module
jsoriano Jun 4, 2019
aae341c
Update filebeat x-pack reference config
jsoriano Jun 4, 2019
c0e0053
Add documentation for timezones
jsoriano Jun 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005]
- Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810]
- Add read_buffer configuration option. {pull}11739[11739]
- `convert_timezone` option is removed and locale is always added to the event so timezone is used when parsing the timestamp, this behaviour can be overriden with processors. {pull}12410[12410]

*Heartbeat*

Expand Down
17 changes: 17 additions & 0 deletions filebeat/docs/include/timezone-support.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[float]
==== Timezone support

This module parses logs that don't contain timezone information. For these logs,
Filebeat reads the local timezone and uses it when parsing to convert the
timestamp to UTC. The timezone to be used for parsing is included in the event
in the `event.timezone` field.

To disable this conversion, the `event.timezone` field can be removed with
the `drop_fields` processor.

If logs are originated from systems or applications with a different timezone to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that is for the syslog case etc?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example. But this can really happen in any case where the application generating the logs is configured with a different timezone than metricbeat.

the local one, the `event.timezone` field can be overwritten with the original
timezone using the `add_fields` processor.

See <<filtering-and-enhancing-data>> for information about specifying
processors in your config.
4 changes: 1 addition & 3 deletions filebeat/docs/modules/system.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,14 +69,12 @@ include::../include/config-option-intro.asciidoc[]

include::../include/var-paths.asciidoc[]

include::../include/var-convert-timezone.asciidoc[]

[float]
==== `auth` fileset settings

include::../include/var-paths.asciidoc[]

include::../include/var-convert-timezone.asciidoc[]
include::../include/timezone-support.asciidoc[]

:has-dashboards!:

Expand Down
6 changes: 0 additions & 6 deletions filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,6 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
Expand All @@ -36,9 +33,6 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
Expand Down
6 changes: 0 additions & 6 deletions filebeat/module/system/_meta/config.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
Expand All @@ -22,9 +19,6 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
6 changes: 0 additions & 6 deletions filebeat/module/system/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,10 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Authorization logs
auth:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false
4 changes: 1 addition & 3 deletions filebeat/module/system/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,12 @@ include::../include/config-option-intro.asciidoc[]

include::../include/var-paths.asciidoc[]

include::../include/var-convert-timezone.asciidoc[]

[float]
==== `auth` fileset settings

include::../include/var-paths.asciidoc[]

include::../include/var-convert-timezone.asciidoc[]
include::../include/timezone-support.asciidoc[]

:has-dashboards!:

Expand Down
2 changes: 0 additions & 2 deletions filebeat/module/system/auth/config/auth.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,5 @@ exclude_files: [".gz$"]
multiline:
pattern: "^\\s"
match: after
{{ if .convert_timezone }}
processors:
- add_locale: ~
{{ end }}
7 changes: 0 additions & 7 deletions filebeat/module/system/auth/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,6 @@ var:
# ssh logs to files
- /var/log/secure.log*
os.windows: []
- name: convert_timezone
default: false
# if ES < 6.1.0, this flag switches to false automatically when evaluating the
# pipeline
min_elasticsearch_version:
version: 6.1.0
value: false

ingest_pipeline: ingest/pipeline.json
input: config/auth.yml
2 changes: 0 additions & 2 deletions filebeat/module/system/syslog/config/syslog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,5 @@ exclude_files: [".gz$"]
multiline:
pattern: "^\\s"
match: after
{{ if .convert_timezone }}
processors:
- add_locale: ~
{{ end }}
7 changes: 0 additions & 7 deletions filebeat/module/system/syslog/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,6 @@ var:
os.darwin:
- /var/log/system.log*
os.windows: []
- name: convert_timezone
default: false
# if ES < 6.1.0, this flag switches to false automatically when evaluating the
# pipeline
min_elasticsearch_version:
version: 6.1.0
value: false

ingest_pipeline: ingest/pipeline.json
input: config/syslog.yml
6 changes: 0 additions & 6 deletions filebeat/modules.d/system.yml.disabled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,10 @@
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Authorization logs
auth:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false
6 changes: 0 additions & 6 deletions x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,6 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
Expand All @@ -36,9 +33,6 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:
Expand Down