-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
b53e50c
commit 66636e7
Showing
2 changed files
with
369 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,367 @@ | ||
[[config-file-format]] | ||
== Config file format | ||
|
||
Beats config files are based on http://www.yaml.org[YAML], a file format that is | ||
easier to read and write than other common data formats like XML or JSON. | ||
|
||
In beats all YAML files start with an dictionary, an unordered collection of | ||
name/value pairs. In addition to dictionaries, YAML also supports lists, numbers, | ||
strings, and many other data types. All members of the same list or dictionary must | ||
have the same indentation level. | ||
|
||
Dictionaries are represented by simple `key: value` pairs all having the same | ||
indentation level. The colon after `key` must be followed by a space. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
name: John Doe | ||
age: 34 | ||
country: Canada | ||
------------------------------------------------------------------------------ | ||
|
||
Lists are introduced by dashes `- `. All list members will be lines beginning | ||
with `- ` at the same indentation level. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
- Red | ||
- Green | ||
- Blue | ||
------------------------------------------------------------------------------ | ||
|
||
Lists and dictionaries are used in beats to build structured configurations. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
filebeat: | ||
prospectors: | ||
- input_type: log | ||
paths: | ||
- /var/log/*.log | ||
multiline: | ||
pattern: '^[' | ||
match: after | ||
------------------------------------------------------------------------------ | ||
|
||
Lists and dictionaries can also be represented in abbreviated form. Abbreviated | ||
form is somewhat similar to JSON using `{}` for dictionaries and `[]` for lists: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
person: \{name: "John Doe", age: 34, country: "Canada"} | ||
colors: ["Red", "Green", "Blue"] | ||
------------------------------------------------------------------------------ | ||
|
||
|
||
[[config-file-format-namespacing]] | ||
=== Namespacing | ||
|
||
All settings are structured using dictionaries and lists. Those are collapsed | ||
into "namespaced" settings, by creating a setting using the full path of the | ||
settings name and it's parent structures names, when reading the configuration | ||
file. | ||
|
||
For example this setting: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
output: | ||
elasticsearch: | ||
index: 'beat-%{+yyyy.MM.dd}' | ||
------------------------------------------------------------------------------ | ||
|
||
gets collapsed into `output.elasticsearch.index: 'beat-%{+yyyy.MM.dd}'`. The | ||
full name of a setting is based on all parent structures involved. | ||
|
||
Lists create numeric names starting with 0. | ||
|
||
For example this filebeat setting: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
filebeat: | ||
prospectors: | ||
- input_type: log | ||
------------------------------------------------------------------------------ | ||
|
||
Gets collapsed into `filebeat.prospectors.0.input_type: log`. | ||
|
||
Alternatively to using indentation, setting names can be used in collapsed form too. | ||
|
||
Note: having two settings with same fully collapsed path is invalid. | ||
|
||
Simple filebeat example with partially collapsed setting names and use of compact form: | ||
|
||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
filebeat.prospectors: | ||
- input_type: log | ||
paths: ["/var/log/*.log"] | ||
multiline.pattern: '^[' | ||
multiline.match: after | ||
output.elasticsearch.hosts: ["http://localhost:9200"] | ||
------------------------------------------------------------------------------ | ||
|
||
[[config-file-format-type]] | ||
=== Config file Data Types | ||
|
||
Values of configuration settings are interpreted as required by beats. | ||
If a value can not be correctly interpreted as the required type - for example a | ||
string is given when a number is required - the beat will fail to start up. | ||
|
||
==== Boolean | ||
|
||
Boolean values can be either `true` or `false`. Alternative names for `true` are | ||
`yes` and `on`. Instead of `false` the values `no` and `off` can be used. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
enabled: true | ||
disabled: false | ||
------------------------------------------------------------------------------ | ||
|
||
==== Number | ||
|
||
Number values require you to enter the number to use without using single or | ||
double quotes. Some settings only support a restricted number range though. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
integer: 123 | ||
negative: -1 | ||
float: 5.4 | ||
------------------------------------------------------------------------------ | ||
|
||
==== String | ||
|
||
In YAML[http://www.yaml.org], multiple styles of string definitions are supported: | ||
double-quoted, single-quoted, unquoted. | ||
|
||
The double-quoted style is specified by surrounding the string with `"`. This | ||
style provides support for escaping unprintable characters using `\`, but comes | ||
at the cost of having to escape `\` and `"` characters. | ||
|
||
The single-quoted style is specified by surrounding the string with `'`. This | ||
style supports no escaping (use `''` to quote a single quote). Only printable | ||
characters can be used when using this form. | ||
|
||
Unquoted style requires no quotes, but does not support any escaping plus care | ||
needs to be taken to not use any symbol that has a speacial meaning in YAML. | ||
|
||
Note: Single-quoted style is recommended when defining regular expressions, | ||
event format strings, windows file paths, or non-alphabetical symbolic characters. | ||
|
||
==== Duration | ||
|
||
Durations require a numeric value with optional fraction and required unit. | ||
Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based | ||
on durations can be disabled by using zero or negative durations. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
duration1: 2.5s | ||
duration2: 6h | ||
duration_disabled: -1s | ||
------------------------------------------------------------------------------ | ||
|
||
==== Regular Expression | ||
|
||
Regular expressions are special strings getting compiled into regular | ||
expressions at load time. | ||
|
||
As regular expressions and YAML use `\` for escaping | ||
characters in strings, it's highly recommended to use single quoted strings when | ||
defining regular expressions. When single quoted strings are used, `\` character | ||
is not interpreted by YAML parser as escape symbol. | ||
|
||
==== Format String (sprintf) | ||
|
||
Format strings enable you to refer to event field values creating a string based | ||
on the current event being processed. Variable expansions are enclosed in | ||
expansion braces `%{<accessor>:default value}`. Event fields are accessed using | ||
field references `[fieldname]`. Optional default values can be specified in case the | ||
field name is missing from the event. | ||
|
||
You can also format time stored in the | ||
`@timestamp` field using the `+FORMAT` syntax where FORMAT is a valid (time | ||
format)[https://godoc.org/github.com/elastic/beats/libbeat/common/dtfmt]. | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
constant-format-string: 'constant string' | ||
field-format-string: '%{[fieldname]} string' | ||
format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}' | ||
------------------------------------------------------------------------------ | ||
|
||
|
||
[[config-file-format-env-vars]] | ||
=== Environment Variables | ||
|
||
Beats support use of environment variables in config files to set values that | ||
need to be configurable during deployment. Environment variable expansion is | ||
introduced using `${VAR}`, where `VAR` is the name of the environment variable. | ||
|
||
Note: Only values can be set using environment variables. Environment variables | ||
usage in namespace and setting names are not supported. | ||
|
||
Variable references are replaced when settings are read by beats. The | ||
replacement is case-sensitive and occurs after the YAML file itself has been | ||
parsed. References to undefined variables will lead to errors when dereferenced | ||
and no default value is specified. To specify a default value, use: | ||
|
||
`${VAR:default_value}` | ||
|
||
Where `default_value` is the value to use if the environment variable is | ||
undefined. | ||
|
||
If you need to use a literal `${` in your configuration file then you can write | ||
`$${` to escape the expansion. The `$` symbol can be used to escape other | ||
characters in the default_value like using `$}` in order to generate a `}` | ||
character without closing the variable expansion. | ||
|
||
After changing the value of an environment variable, the beat needs to be | ||
restarted to pick up the new value. | ||
|
||
[float] | ||
==== Examples | ||
|
||
Here are some examples of configurations that use environment variables | ||
and what each configuration looks like after replacement: | ||
|
||
[options="header"] | ||
|================================== | ||
|Config source |Environment setting |Config after replacement | ||
|`name: ${NAME}` |`export NAME=elastic` |`name: elastic` | ||
|`name: ${NAME:beats}` |no setting |`name: beats` | ||
|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic` | ||
|================================== | ||
|
||
[[config-gile-format-refs]] | ||
=== Reference Variables | ||
|
||
Beats settings can reference other settings splicing multiple optionally custom | ||
named settings into new values. References use the same syntax as | ||
<<config-file-format-env-vars>> do. Only fully collapsed setting names can be | ||
referenced to. | ||
|
||
For example the filebeat registry file defaults to: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
filebeat.registry: $\{path.data}/registry | ||
------------------------------------------------------------------------------ | ||
|
||
With `path.data` being an implicit config setting, that is overwritable from | ||
command line, as well as in the configuration file. | ||
|
||
Example referencing `es.host` in `output.elasticsearch.hosts`: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
es.host: '$\{ES_HOST:localhost}' | ||
output.elasticsearch: | ||
hosts: ['http://$\{es.host}:9200'] | ||
------------------------------------------------------------------------------ | ||
|
||
Introducing `es.host`, the host can be overwritten from command line using | ||
`-E es.host=another-host`. | ||
|
||
Plain references, having no default value and are not spliced with other | ||
references or strings can reference complete namespaces. | ||
|
||
These setting with duplicate content: | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
namespace1: | ||
subnamespace: | ||
host: localhost | ||
sleep: 1s | ||
namespace2: | ||
subnamespace: | ||
host: localhost | ||
sleep: 1s | ||
------------------------------------------------------------------------------ | ||
|
||
can be rewritten to | ||
|
||
["source","yaml",subs="attributes"] | ||
------------------------------------------------------------------------------ | ||
namespace1: $\{shared} | ||
namespace2: $\{shared} | ||
shared: | ||
subnamespace: | ||
host: localhost | ||
sleep: 1s | ||
------------------------------------------------------------------------------ | ||
|
||
when using plain references. | ||
|
||
|
||
[[config-file-format-cli]] | ||
=== Command line arguments | ||
|
||
Config files to load are set using the `-c` flag on command line. If no flag is | ||
given, a beat and OS pecific default file path will be assumed. | ||
|
||
You can specify multiple configuration files by repeating the `-c` flag. You can | ||
use this, for example, for setting defaults in a base configuration file, and | ||
overwrite settings via local configuration files. | ||
|
||
In addition to overwriting settings using multiple configuration files, | ||
individual settings can be overwritten using `-E <setting>=<value>`. | ||
|
||
[[config-file-format-tips]] | ||
=== YAML Tips and Gotchas | ||
|
||
When you edit the configuration file, there are a few things that you should know. | ||
|
||
[float] | ||
==== Use Spaces for Indentation | ||
|
||
Indentation is meaningful in YAML. Make sure that you use spaces, rather than | ||
tab characters, to indent sections. | ||
|
||
In the default configuration files and in all the examples in the documentation, | ||
we use 2 spaces per indentation level. We recommend you do the same. | ||
|
||
[float] | ||
==== Look at the Default Config File for Structure | ||
|
||
The best way to understand where to define a configuration option is by looking | ||
at the provided sample configuration files. The configuration files contain most | ||
of the default configurations that are available per beat. To change a setting, | ||
simply uncomment the line and change the values. | ||
|
||
[float] | ||
==== Test Your Config File | ||
|
||
You can test your configuration file to verify that the structure is valid. Simply run | ||
your in the foreground with the `-configtest` flag specified. For example: | ||
|
||
["source","yaml",subs="attributes,callouts"] | ||
---------------------------------------------------------------------- | ||
filebeat -c filebeat.yml -configtest | ||
---------------------------------------------------------------------- | ||
|
||
You'll see a message if an error in the configuration file is found. | ||
|
||
[float] | ||
==== Wrap Regular Expressions in Single Quotation Marks | ||
|
||
If you need to specify a regular expression in a YAML file, it's a good idea to | ||
wrap the regular expression in single quotation marks to work around YAML's | ||
tricky rules for string escaping. | ||
|
||
For more information about YAML, see http://yaml.org/. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters