Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document complex object support in env vars #4200

Merged
merged 3 commits into from
Jun 9, 2017
Merged

Document complex object support in env vars #4200

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2f3cdd7
Document complex ojbect support in env vars
dedemorton May 4, 2017
89f0f7d
Fixes from the review
dedemorton May 5, 2017
723cf58
More changes from the review
dedemorton Jun 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions filebeat/docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ include::./multiline.asciidoc[]

include::../../libbeat/docs/shared-config-ingest.asciidoc[]

:standalone:
include::../../libbeat/docs/shared-env-vars.asciidoc[]

include::./multiple-prospectors.asciidoc[]
Expand Down
1 change: 1 addition & 0 deletions heartbeat/docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[]
//points to shared topic because configuring-logstash.asciidoc is just a wrapper
include::./configuring-logstash.asciidoc[]

:standalone:
include::../../libbeat/docs/shared-env-vars.asciidoc[]

:standalone:
Expand Down
83 changes: 41 additions & 42 deletions libbeat/docs/config-file-format.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Simple filebeat example with partially collapsed setting names and use of compac

filebeat.prospectors:
- type: log
paths: ["/var/log/*.log"]
paths: ["/var/log/*.log"]
multiline.pattern: '^['
multiline.match: after

Expand Down Expand Up @@ -213,44 +213,7 @@ format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
[[config-file-format-env-vars]]
=== Environment Variables

Beats support use of environment variables in config files to set values that
need to be configurable during deployment. Environment variable expansion is
introduced using `${VAR}`, where `VAR` is the name of the environment variable.

Note: Only values can be set using environment variables. Environment variables
usage in namespace and setting names are not supported.

Variable references are replaced when settings are read by beats. The
replacement is case-sensitive and occurs after the YAML file itself has been
parsed. References to undefined variables will lead to errors when dereferenced
and no default value is specified. To specify a default value, use:

`${VAR:default_value}`

Where `default_value` is the value to use if the environment variable is
undefined.

If you need to use a literal `${` in your configuration file then you can write
`$${` to escape the expansion. The `$` symbol can be used to escape other
characters in the default_value like using `$}` in order to generate a `}`
character without closing the variable expansion.

After changing the value of an environment variable, the beat needs to be
restarted to pick up the new value.

[float]
==== Examples

Here are some examples of configurations that use environment variables
and what each configuration looks like after replacement:

[options="header"]
|==================================
|Config source |Environment setting |Config after replacement
|`name: ${NAME}` |`export NAME=elastic` |`name: elastic`
|`name: ${NAME:beats}` |no setting |`name: beats`
|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
|==================================
include::shared-env-vars.asciidoc[]

[[config-gile-format-refs]]
=== Reference Variables
Expand Down Expand Up @@ -333,15 +296,15 @@ When installed via an RPM or DEB package, the config file at
`/etc/{beatname}/{beatname}.yml` will have the proper owner and permissions. The
file is owned by `root` and has file permissions of `0644` (`-rw-r--r--`).

You may encounter the following errors if your config file fails these checks:
You may encounter the following errors if your config file fails these checks:

["source","sh"]
--------------------------------------------------------------------------------
Exiting: error loading config file: config file ("{beatname}.yml") must be
owned by the beat user (uid=501) or root
--------------------------------------------------------------------------------

To correct this problem you can use either `chown root {beatname}.yml` or
To correct this problem you can use either `chown root {beatname}.yml` or
`chown 501 {beatname}.yml` to change the owner of the configuration file.

["source","sh"]
Expand Down Expand Up @@ -370,7 +333,43 @@ use this, for example, for setting defaults in a base configuration file, and
overwrite settings via local configuration files.

In addition to overwriting settings using multiple configuration files,
individual settings can be overwritten using `-E <setting>=<value>`.
individual settings can be overwritten using `-E <setting>=<value>`. The
`<value>` can be either a single value or a complex object, such as a list or
dictionary.

For example, given the following configuration:

["source","yaml"]
--------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
username: username
password: password
--------------------------------------------------------------------------------

You can disable the Elasticsearch output and write all events to the console by
setting:

["source","sh"]
--------------------------------------------------------------------------------
-E output='{elasticsearch.enabled: false, console.pretty: true}'
--------------------------------------------------------------------------------
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe include sample config and final config here?

e.g. with -E output='{elasticsearch.enabled: false, console.pretty: true}'

and a config file like:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: username
  password: password

then the final configuration presented to the beat becomes:

output.elasticsearch:
  enabled: false
  hosts: ["http://localhost:9200"]
  username: username
  password: password

output.console:
  pretty: true

That is, overwriting fields with complex objects will merge the object with the original configuration, not changing/overwriting any other settings.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added


Any complex objects that you specify at the command line are merged with the
original configuration, and the following configuration is passed to the Beat:

["source","yaml"]
--------------------------------------------------------------------------------
output.elasticsearch:
enabled: false
hosts: ["http://localhost:9200"]
username: username
password: password

output.console:
pretty: true
--------------------------------------------------------------------------------


[[config-file-format-tips]]
=== YAML Tips and Gotchas
Expand Down
75 changes: 67 additions & 8 deletions libbeat/docs/shared-env-vars.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,20 @@
//// Use the appropriate variables defined in the index.asciidoc file to
//// resolve Beat names: beatname_uc and beatname_lc.
//// Use the following include to pull this content into a doc file:
//// :standalone:
//// include::../../libbeat/docs/shared-env-vars.asciidoc[]
//// Specify :standalone: when this file is pulled into and index. When
//// the file is embedded in another file, do no specify :standalone:
//////////////////////////////////////////////////////////////////////////

ifdef::standalone[]

[[using-environ-vars]]
== Using Environment Variables in the Configuration

You can use environment variable references in the +{beatname_lc}.yml+ file to
endif::[]

You can use environment variable references in the config file to
set values that need to be configurable during deployment. To do this, use:

`${VAR}`
Expand All @@ -22,18 +29,36 @@ Where `VAR` is the name of the environment variable.
Each variable reference is replaced at startup by the value of the environment
variable. The replacement is case-sensitive and occurs before the YAML file is
parsed. References to undefined variables are replaced by empty strings unless
you specify a default value. To specify a default value, use:
you specify a default value or custom error text.

To specify a default value, use:

`${VAR:default_value}`

Where `default_value` is the value to use if the environment variable is
undefined.

To specify custom error text, use:

`${VAR:?error_text}`

Where `error_text` is custom text that will be prepended to the error
message if the environment variable cannot be expanded.

If you need to use a literal `${` in your configuration file then you can write
`$${` to escape the expansion.

After changing the value of an environment variable, you need to restart
{beatname_uc} to pick up the new value.
the Beat to pick up the new value.

[NOTE]
==================================
You can also specify environment variables when you override a config
setting from the command line by using the `-E` option. For example:

`-E name=${NAME}`

==================================

[float]
=== Examples
Expand All @@ -43,9 +68,43 @@ and what each configuration looks like after replacement:

[options="header"]
|==================================
|Config source |Environment setting |Config after replacement
|`name: ${NAME}` |`export NAME=elastic` |`name: elastic`
|`name: ${NAME}` |no setting |`name:`
|`name: ${NAME:beats}` |no setting |`name: beats`
|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
|Config source |Environment setting |Config after replacement
|`name: ${NAME}` |`export NAME=elastic` |`name: elastic`
|`name: ${NAME}` |no setting |`name:`
|`name: ${NAME:beats}` |no setting |`name: beats`
|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
|`name: ${NAME:?You need to set the NAME environment variable}` |no setting | None. Returns an error message that's prepended with the custom text.
|`name: ${NAME:?You need to set the NAME environment variable}` |`export NAME=elastic` | `name: elastic`
|==================================

[float]
=== Specifying Complex Objects in Environment Variables

You can specify complex objects, such as lists or dictionaries, in environment
variables by using a JSON-like syntax.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we want to be more explicit here, as JSON-like syntax is pretty unspecific on syntax limitations + move the pointers from line 93pp up here.

e.g.:

Like with JSON, dictionaries and lists are constructed using {} and []. But in comparison to JSON, the syntax allows for trailing commas and slightly different string quotation rules. Strings can be unquoted, single-quoted, or double-quoted, as a convenience for simple settings and to make it easier for you to mix quotation usage in the shell. Arrays at the top-level do not require brackets ([]).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea. fixed.


As with JSON, dictionaries and lists are constructed using `{}` and `[]`. But
unlike JSON, the syntax allows for trailing commas and slightly different string
quotation rules. Strings can be unquoted, single-quoted, or double-quoted, as a
convenience for simple settings and to make it easier for you to mix quotation
usage in the shell. Arrays at the top-level do not require brackets (`[]`).

For example, the following environment variable is set to a list:

[source,yaml]
-------------------------------------------------------------------------------
ES_HOSTS="10.45.3.2:9220,10.45.3.1:9230"
-------------------------------------------------------------------------------

You can reference this variable in the config file:

[source,yaml]
-------------------------------------------------------------------------------
output.elasticsearch:
hosts: '${ES_HOSTS}'
-------------------------------------------------------------------------------

When the Beat loads the config file, it resolves the environment variable and
replaces it with the specified list before reading the `hosts` setting.

NOTE: Do not use double-quotes (`"`) to wrap regular expressions, or the backslash (`\`) will be interpreted as an escape character.
13 changes: 8 additions & 5 deletions libbeat/docs/yaml.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
//// Use the appropriate variables defined in the index.asciidoc file to
//// resolve Beat names: beatname_uc and beatname_lc.
//// Use the following include to pull this content into a doc file:
//// :standalone:
//// include::../../libbeat/docs/yaml.asciidoc[]
//// Specify :standalone: when this file is pulled into and index. When
//// the file is embedded in another file, do no specify :standalone:
//////////////////////////////////////////////////////////////////////////

ifdef::standalone[]
Expand Down Expand Up @@ -40,7 +43,7 @@ simply uncomment the line and change the values.

You can test your configuration file to verify that the structure is valid.
Simply change to the directory where the binary is installed, and run
the Beat in the foreground with the `-configtest` flag specified. For example:
the Beat in the foreground with the `-configtest` flag specified. For example:

ifdef::allplatforms[]

Expand All @@ -65,7 +68,7 @@ You'll see a message if the Beat finds an error in the file.
[float]
=== Wrap Regular Expressions in Single Quotation Marks

If you need to specify a regular expression in a YAML file, it's a good idea to wrap the regular expression in single quotation marks to work around YAML's tricky rules for string escaping.
If you need to specify a regular expression in a YAML file, it's a good idea to wrap the regular expression in single quotation marks to work around YAML's tricky rules for string escaping.

For more information about YAML, see http://yaml.org/.

Expand All @@ -74,9 +77,9 @@ For more information about YAML, see http://yaml.org/.
=== Wrap Paths in Single Quotation Marks

Windows paths in particular sometimes contain spaces or characters, such as drive
letters or triple dots, that may be misinterpreted by the YAML parser.
letters or triple dots, that may be misinterpreted by the YAML parser.

To avoid this problem, it's a good idea to wrap paths in single quotation marks.
To avoid this problem, it's a good idea to wrap paths in single quotation marks.

[float]
[[avoid-leading-zeros]]
Expand All @@ -85,7 +88,7 @@ To avoid this problem, it's a good idea to wrap paths in single quotation marks.
If you use a leading zero (for example, `09`) in a numeric field without
wrapping the value in single quotation marks, the value may be interpreted
incorrectly by the YAML parser. If the value is a valid octal, it's converted
to an integer. If not, it's converted to a float.
to an integer. If not, it's converted to a float.

To prevent unwanted type conversions, avoid using leading zeros in field values,
or wrap the values in single quotation marks.
Expand Down
1 change: 1 addition & 0 deletions metricbeat/docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[]

include::./configuring-logstash.asciidoc[]

:standalone:
include::../../libbeat/docs/shared-env-vars.asciidoc[]

:standalone:
Expand Down
3 changes: 3 additions & 0 deletions packetbeat/docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ include::./configuring-logstash.asciidoc[]

include::./flows.asciidoc[]

:standalone:
include::../../libbeat/docs/shared-env-vars.asciidoc[]

include::./thrift.asciidoc[]

:standalone:
Expand Down
1 change: 1 addition & 0 deletions winlogbeat/docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ include::./winlogbeat-filtering.asciidoc[]

include::../../libbeat/docs/shared-config-ingest.asciidoc[]

:standalone:
include::../../libbeat/docs/shared-env-vars.asciidoc[]

:standalone:
Expand Down