Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add queue settings to docs #4884

Merged
merged 6 commits into from
Oct 9, 2017
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions auditbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ The following topics describe how to configure {beatname_uc}:
* <<configuration-{beatname_lc}>>
* <<configuration-general-options>>
* <<{beatname_lc}-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand All @@ -46,6 +47,8 @@ include::./auditbeat-general-options.asciidoc[]
include::./reload-configuration.asciidoc[]

:allplatforms:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions filebeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ The following topics describe how to configure Filebeat:
* <<multiline-examples>>
* <<configuration-general-options>>
* <<filebeat-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand Down Expand Up @@ -50,6 +51,8 @@ include::./filebeat-general-options.asciidoc[]
include::./reload-configuration.asciidoc[]

:allplatforms:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions heartbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ The following topics describe how to configure Heartbeat:

* <<configuration-heartbeat-options>>
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand All @@ -43,6 +44,8 @@ include::./heartbeat-options.asciidoc[]
include::./heartbeat-general-options.asciidoc[]

:allplatforms:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down
37 changes: 13 additions & 24 deletions libbeat/docs/outputconfig.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -283,21 +283,14 @@ However big batch sizes can also increase processing times, which might result i
API errors, killed connections, timed-out publishing requests, and, ultimately, lower
throughput.

Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
Beats that publish single events (such as Packetbeat) send each event directly to
Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
spooler size.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.

===== `timeout`

The http request timeout in seconds for the Elasticsearch request. The default is 90.

===== `flush_interval`

The number of seconds to wait for new events between two bulk API index requests.
If `bulk_max_size` is reached before this interval expires, additional bulk index
requests are made.

===== `ssl`

Configuration options for SSL parameters like the certificate authority to use
Expand Down Expand Up @@ -533,10 +526,10 @@ However big batch sizes can also increase processing times, which might result i
API errors, killed connections, timed-out publishing requests, and, ultimately, lower
throughput.

Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
Beats that publish single events (such as Packetbeat) send each event directly to
Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
spooler size.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.


[[kafka-output]]
=== Configure the Kafka output
Expand Down Expand Up @@ -731,10 +724,6 @@ The ACK reliability level required from broker. 0=no response, 1=wait for local

Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error.

===== `flush_interval`

The number of seconds to wait for new events between two producer API calls.

===== `ssl`

Configuration options for SSL parameters like the root CA for Kafka connections. See
Expand Down Expand Up @@ -934,11 +923,9 @@ of sending events. However big batch sizes can also increase processing times,
which might result in API errors, killed connections, timed-out publishing
requests, and, ultimately, lower throughput.

Setting `bulk_max_size` to values less than or equal to 0 disables buffering in
libbeat. When buffering is disabled, Beats that publish single events (such as
Packetbeat) send each event directly to Redis. Beats that publish
data in batches (such as Filebeat) send events in batches based on the spooler
size.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.

===== `ssl`

Expand Down Expand Up @@ -1068,7 +1055,9 @@ The maximum number of events to buffer internally during publishing. The default
Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this
setting does not affect how events are published.

Setting `bulk_max_size` to 0 disables buffering in libbeat.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.

[[configuration-output-codec]]
=== Configure the ouput codec
Expand Down
74 changes: 74 additions & 0 deletions libbeat/docs/queueconfig.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
[[configuring-internal-queue]]
== Configure the internal queue

{beatname_uc} uses an internal queue to store events before publishing them. The
queue is responsible for buffering and combining events into batches that can
be consumed by the outputs. The outputs will use bulk operations to send a
batch of events in one transaction.

You can configure the type and behavior of the internal queue by setting options in the `queue` section of the +{beatname_lc}.yml+ config file.


Example configuration:

[source,yaml]
------------------------------------------------------------------------------
queue.mem:
events: 4096
------------------------------------------------------------------------------

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a [float] tag here to keep the info about the memory queue from becoming a separate topic. If you don't use the [float] tag, you get this:

image

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Change did require me to add [float] about everywhere for the doc build not to fail.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yah, I really dislike [float] tags. We use them all over the library. They seem like such a hack, but are impossible to avoid without re-architecting all the content...and even so, you would still end up having to use them in some places.

[float]
[[configuration-internal-queue-memory]]
=== Configure the memory qeueue

The memory queue keeps all events in memory. It is the only queue type
supported right now. By default no flush interval is configured. All events
published to this queue will be directly consumed by the outputs.
The output's `bulk_max_size` setting limits the number of events being processed at once.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this works!


The memory queue is waiting for the output to acknowledge or drop an events. If
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence has a typo "an events". The verb tense also seems a little off. Should probably say:

The memory queue waits for the output to acknowledge or drop events.

the queue is full, no new events can be inserted into the memeory queue. Only
after the signal from the output will the queue free up space for more events to be accepted.

To enforce spooling in the queue, set the `flush.min_events` and `flush.timeout` options.

This sample configuration forwards events to the output if 512 events are
available or the oldest available event is already waiting for 5s in the queue:

[source,yaml]
------------------------------------------------------------------------------
queue.mem:
events: 4096
flush.min_events: 512
flush.timeout: 5s
------------------------------------------------------------------------------

[float]
==== Configuration options

You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file:

[float]
===== `events`

Number of events the queue can store.

The default value is 4096 events.

[float]
===== `flush.min_events`

Minimum number of events required for publishing. If this value is set to 0, the
output can start publishing events without additional waiting times. Otherwise
the output has to wait for more events to become available.

The default value is 0.

[float]
===== `flush.timeout`

Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events
will be immediately available for consumption.

The default values is 0s.

3 changes: 3 additions & 0 deletions metricbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ The following topics describe how to configure Metricbeat:
* <<configuration-metricbeat>>
* <<configuration-general-options>>
* <<metricbeat-configuration-reloading>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand All @@ -44,6 +45,8 @@ include::./metricbeat-general-options.asciidoc[]
include::./reload-configuration.asciidoc[]

:allplatforms:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions packetbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ The following topics describe how to configure Packetbeat:
* <<configuration-protocols>>
* <<configuration-processes>>
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand All @@ -45,6 +46,8 @@ include::./packetbeat-options.asciidoc[]
include::./packetbeat-general-options.asciidoc[]

:allplatforms:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down
3 changes: 3 additions & 0 deletions winlogbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ The following topics describe how to configure Winlogbeat:

* <<configuration-winlogbeat-options>>
* <<configuration-general-options>>
* <<configuring-internal-queue>>
* <<configuring-output>>
* <<configuration-ssl>>
* <<filtering-and-enhancing-data>>
Expand All @@ -38,6 +39,8 @@ include::./winlogbeat-options.asciidoc[]
include::./winlogbeat-general-options.asciidoc[]

:win:
include::../../libbeat/docs/queueconfig.asciidoc[]

include::../../libbeat/docs/outputconfig.asciidoc[]

include::../../libbeat/docs/shared-ssl-config.asciidoc[]
Expand Down