Add queue settings to docs

auditbeat/docs/configuring-howto.asciidoc

@@ -22,6 +22,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-{beatname_lc}>>
 * <<configuration-general-options>>
 * <<{beatname_lc}-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -46,6 +47,8 @@ include::./auditbeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

filebeat/docs/configuring-howto.asciidoc

@@ -23,6 +23,7 @@ The following topics describe how to configure Filebeat:
 * <<multiline-examples>>
 * <<configuration-general-options>>
 * <<filebeat-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -50,6 +51,8 @@ include::./filebeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

heartbeat/docs/configuring-howto.asciidoc

@@ -23,6 +23,7 @@ The following topics describe how to configure Heartbeat:
 
 * <<configuration-heartbeat-options>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -43,6 +44,8 @@ include::./heartbeat-options.asciidoc[]
 include::./heartbeat-general-options.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

libbeat/docs/outputconfig.asciidoc

@@ -283,21 +283,14 @@ However big batch sizes can also increase processing times, which might result i
 API errors, killed connections, timed-out publishing requests, and, ultimately, lower
 throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
-Beats that publish single events (such as Packetbeat) send each event directly to
-Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
-spooler size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 ===== `timeout`
 
 The http request timeout in seconds for the Elasticsearch request. The default is 90.
 
-===== `flush_interval`
-
-The number of seconds to wait for new events between two bulk API index requests.
-If `bulk_max_size` is reached before this interval expires, additional bulk index
-requests are made.
-
 ===== `ssl`
 
 Configuration options for SSL parameters like the certificate authority to use
@@ -533,10 +526,10 @@ However big batch sizes can also increase processing times, which might result i
 API errors, killed connections, timed-out publishing requests, and, ultimately, lower
 throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
-Beats that publish single events (such as Packetbeat) send each event directly to
-Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
-spooler size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
+
 
 [[kafka-output]]
 === Configure the Kafka output
@@ -731,10 +724,6 @@ The ACK reliability level required from broker. 0=no response, 1=wait for local
 
 Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error.
 
-===== `flush_interval`
-
-The number of seconds to wait for new events between two producer API calls.
-
 ===== `ssl`
 
 Configuration options for SSL parameters like the root CA for Kafka connections. See
@@ -934,11 +923,9 @@ of sending events. However big batch sizes can also increase processing times,
 which might result in API errors, killed connections, timed-out publishing
 requests, and, ultimately, lower throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in
-libbeat. When buffering is disabled, Beats that publish single events (such as
-Packetbeat) send each event directly to Redis. Beats that publish
-data in batches (such as Filebeat) send events in batches based on the spooler
-size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 ===== `ssl`
 
@@ -1068,7 +1055,9 @@ The maximum number of events to buffer internally during publishing. The default
 Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this
 setting does not affect how events are published.
 
-Setting `bulk_max_size` to 0 disables buffering in libbeat.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 [[configuration-output-codec]]
 === Configure the ouput codec

libbeat/docs/queueconfig.asciidoc

@@ -0,0 +1,69 @@
+[[configuring-internal-queue]]
+== Configure the internal queue
+
+The {beatname_uc} uses an internal queue to store events before publishing them. The
+queue is responsible for buffering and combining events into batches that can
+be consumed by the outputs. The outputs will use bulk operations to send a
+batch of events in one transaction.
+
+You can configure the type and behavior of the internal queue by setting options in the `queue` section of the +{beatname_lc}.yml+ config file.
+
+
+Example configuration:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+------------------------------------------------------------------------------
+
+[[configuration-internal-queue-memory]]
+=== Configure the memory qeueue
+
+The memory queue keeps all events in memory. It is the only queue type
+supported right now.  By default no flush interval is configured. All events
+published to this queue will be directly consumed by the outputs.
+The output's `bulk_max_size` setting limits the number of events being processed at once.
+
+The memory queue is waiting for the output to acknowledge or drop an events. If
+the queue is full, no new events can be inserted into the memeory queue. Only
+after the signal from the output will the queue free up space for more events to be accepted.
+
+To enforce spooling in the queue, set the `flush.min_events` and `flush.timeout` options.
+
+This sample configuration forwards events to the output if 512 events are
+available or the oldest available event is already waiting for 5s in the queue:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+  flush.min_events: 512
+  flush.timeout: 5s
+------------------------------------------------------------------------------
+
+==== Configuration options
+
+You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file:
+
+===== `events`
+
+Number of events the queue can store. 
+
+The default value is 4096 events.
+
+===== `flush.min_events`
+
+Minimum number of events required for publishing. If this value is set to 0, the
+output can start publishing events without additional waiting times. Otherwise
+the output has to wait for more events to become available.
+
+The default value is 0.
+
+===== `flush.timeout`
+
+Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events
+will be immediately available for consumption.
+
+The default values is 0s.
+

metricbeat/docs/configuring-howto.asciidoc

@@ -22,6 +22,7 @@ The following topics describe how to configure Metricbeat:
 * <<configuration-metricbeat>>
 * <<configuration-general-options>>
 * <<metricbeat-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -44,6 +45,8 @@ include::./metricbeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

packetbeat/docs/configuring-howto.asciidoc

@@ -24,6 +24,7 @@ The following topics describe how to configure Packetbeat:
 * <<configuration-protocols>>
 * <<configuration-processes>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -45,6 +46,8 @@ include::./packetbeat-options.asciidoc[]
 include::./packetbeat-general-options.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

winlogbeat/docs/configuring-howto.asciidoc

@@ -19,6 +19,7 @@ The following topics describe how to configure Winlogbeat:
 
 * <<configuration-winlogbeat-options>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -38,6 +39,8 @@ include::./winlogbeat-options.asciidoc[]
 include::./winlogbeat-general-options.asciidoc[]
 
 :win:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]