Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Add module usage info #7826

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions docs/index-shared2.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ include::static/managing-multiline-events.asciidoc[]
:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/glob-support.asciidoc
include::static/glob-support.asciidoc[]

// Working with Logstash Modules

include::static/modules.asciidoc[]

// Working with Filebeat Modules

:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/filebeat-modules.asciidoc
include::static/filebeat-modules.asciidoc[]

// Data resiliency

:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/resiliency.asciidoc
Expand All @@ -37,11 +46,6 @@ include::static/persistent-queues.asciidoc[]
:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/dead-letter-queues.asciidoc
include::static/dead-letter-queues.asciidoc[]

// Working with Filebeat Modules

:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/filebeat-modules.asciidoc
include::static/filebeat-modules.asciidoc[]

// Transforming Data

:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/transforming-data.asciidoc
Expand Down
118 changes: 118 additions & 0 deletions docs/static/modules.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
[[logstash-modules]]
== Working with Logstash Modules

Logstash modules provide a quick, end-to-end solution for ingesting data and
visualizing it with purpose-built dashboards.

Each module comes pre-packaged with Logstash configurations, Kibana dashboards,
and other meta files that make it easier for you to set up the Elastic Stack for
specific use cases or data sources.

[float]
[[running-logstash-modules]]
=== Running modules

When you run a module, Logstash creates and loads the pipeline configurations
required to read and parse the data. It also loads the index pattern,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

index pattern and field mappings will be directly on ES, should we also make that explicit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My inclination is to keep this statement simple here. If we start seeing questions about this on the forum, we can clarify.

field definitions, searches, visualizations, and dashboards required to
visualize your data in Kibana.

To run a module, you use the `--modules` option:

[source,shell]
----
bin/logstash --modules MODULE_NAME [-M CONFIG_SETTINGS]
----


//TODO: For 6.0, show how to run mutliple modules

Where `MODULE_NAME` is the name of Logstash module and `CONFIG_SETTINGS`
is one or more optional configuration settings. `CONFIG_SETTINGS` are only
required when the default configuration doesn't meet your needs, or you need to
override settings specified in the `logstash.yml` settings file.

For example, the following command runs the Netflow module with the default
settings:

[source,shell]
----
bin/logstash --modules netflow
----

The following command runs the Netflow module and overrides the Elasticsearch
`host` setting:

[source,shell]
----
bin/logstash --modules netflow -M "netflow.var.elasticsearch.host=es.mycloud.com"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@suyograo missed this one initially, but this should be plural "hosts" right? and its also missing the port?

----


See <<overriding-logstash-module-settings>> for more info about overriding settings.

[float]
[[configuring-logstash-modules]]
=== Configuring modules

To configure a module, you can either
<<setting-logstash-module-config,specify configuration settings>> in the
`logstash.yml` <<logstash-settings-file,settings file>>, or use command-line overrides to
<<overriding-logstash-module-settings,specify settings at the command line>>.

[float]
[[setting-logstash-module-config]]
==== Specify module settings in `logstash.yml`

To specify module settings in the `logstash.yml`
<<logstash-settings-file,settings file>> file, you add a module definition to
the modules array. Each module definition begins with a dash (-) and is followed
by `name: module_name` then a series of name/value pairs that specify module
settings. For example:

[source,shell]
----
modules:
- name: netflow
var.output.elasticsearch.host: "es.mycloud.com"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@suyograo same here as the comment above...

var.output.elasticsearch.user: "foo"
var.output.elasticsearch.password: "password"
var.input.tcp.port: 5606
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@suyograo also, does netflow has a tcp setting? i thought it was udp.

----

For a list of available module settings, see the documentation for the module.

[float]
[[overriding-logstash-module-settings]]
==== Specify module settings at the command line

You can override module settings by specifying one or more configuration
overrides when you start Logstash. To specify an override, you use the `-M`
command line option:

[source,shell]
----
-M MODULE_NAME.var.PLUGINTYPE1.PLUGINNAME1.KEY1=VALUE
----

Notice that the fully-qualified setting name includes the module name.

You can specify multiple overrides. Each override must start with `-M`.

The following command runs the Netflow module and overrides both the
Elasticsearch `host` setting and the `udp.port` setting:

[source,shell]
----
bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" -M "netflow.var.elasticseach.host=my-es-cloud"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@suyograo same comment here as above.

----

Any settings defined in the command line are ephemeral and will not persist across
subsequent runs of Logstash. If you want to persist a configuration, you need to
set it in the `logstash.yml` <<logstash-settings-file,settings file>>.

Settings that you specify at the command line are merged with any settings
specified in the `logstash.yml` file. If an option is set in both
places, the value specified at the command line takes precedence.