From 677a95277c8d6d6d9565e652a2f84350f994a146 Mon Sep 17 00:00:00 2001 From: urso Date: Fri, 11 Aug 2017 19:26:08 +0200 Subject: [PATCH 1/6] Add queue settings to docs --- libbeat/docs/generalconfig.asciidoc | 67 +++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/libbeat/docs/generalconfig.asciidoc b/libbeat/docs/generalconfig.asciidoc index a3f7426160a..e22c090cea4 100644 --- a/libbeat/docs/generalconfig.asciidoc +++ b/libbeat/docs/generalconfig.asciidoc @@ -103,3 +103,70 @@ processors in your config. Sets the maximum number of CPUs that can be executing simultaneously. The default is the number of logical CPUs available in the system. + +[float] +[[configuration-internal-queue]] +=== Configure the internal queue + +The `queue` namespace configures the queue type to be used. + +The Elastic Beats employ an internal queue for events to be published. The +queue is responsible for buffering and combining events into batches, which can +be consumed by the outputs. The outputs will use bulk operations to send a +batch of events in one transaction. + + +Example configuration: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 +------------------------------------------------------------------------------ + +==== Configure the Memory Qeueue +[[configuration-internal-queue-memory]] + +The memory queue keeps all events in memory. By default no flush interval is +configured. All events published to this queue will be directly consumed by the +outputs. An output will consume up to the outputs `bulk_max_size` events at once. + +Only after an event is dropped or acknowledged by the output, a new slot will +be available in the queue. + +By setting `flush.min_events` and `flush.timeout`, spooling in the queue is +enforced. In this case the output has to wait for a minimal number of events +to be available. + +This sample configuration forwards events to the output if 512 events are +available or the oldest available event is already waiting for 5s in the queue: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 + flush.min_events: 512 + flush.timeout: 5s +------------------------------------------------------------------------------ + +===== `events` + +Number of events the queue can store. + +The default value is 4096 events. + +===== `flush.min_events` + +Minimum number of events required for publishing. If this value is set to 0, the +output can start publishing events without additional waiting times. Otherwise +the output has to wait for more events to become available. + +The default value is 0. + +===== `flush.timeout` + +Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events +will be immediately available for consumption. + +The default values is 0s. + From 387376ab606a3a62f5c193fc80f473ca368752db Mon Sep 17 00:00:00 2001 From: urso Date: Sun, 13 Aug 2017 17:20:39 +0200 Subject: [PATCH 2/6] Move queue setting to TOC + address some review --- auditbeat/docs/configuring-howto.asciidoc | 3 + filebeat/docs/configuring-howto.asciidoc | 3 + heartbeat/docs/configuring-howto.asciidoc | 3 + libbeat/docs/generalconfig.asciidoc | 67 --------------------- libbeat/docs/queueconfig.asciidoc | 69 ++++++++++++++++++++++ metricbeat/docs/configuring-howto.asciidoc | 3 + packetbeat/docs/configuring-howto.asciidoc | 3 + winlogbeat/docs/configuring-howto.asciidoc | 3 + 8 files changed, 87 insertions(+), 67 deletions(-) create mode 100644 libbeat/docs/queueconfig.asciidoc diff --git a/auditbeat/docs/configuring-howto.asciidoc b/auditbeat/docs/configuring-howto.asciidoc index 54a63e86ce4..e16447de3cf 100644 --- a/auditbeat/docs/configuring-howto.asciidoc +++ b/auditbeat/docs/configuring-howto.asciidoc @@ -22,6 +22,7 @@ The following topics describe how to configure {beatname_uc}: * <> * <> * <<{beatname_lc}-configuration-reloading>> +* <> * <> * <> * <> @@ -46,6 +47,8 @@ include::./auditbeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc index 745b8ab45f5..b6268485426 100644 --- a/filebeat/docs/configuring-howto.asciidoc +++ b/filebeat/docs/configuring-howto.asciidoc @@ -23,6 +23,7 @@ The following topics describe how to configure Filebeat: * <> * <> * <> +* <> * <> * <> * <> @@ -50,6 +51,8 @@ include::./filebeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/heartbeat/docs/configuring-howto.asciidoc b/heartbeat/docs/configuring-howto.asciidoc index a46909a50f9..3774c62e983 100644 --- a/heartbeat/docs/configuring-howto.asciidoc +++ b/heartbeat/docs/configuring-howto.asciidoc @@ -23,6 +23,7 @@ The following topics describe how to configure Heartbeat: * <> * <> +* <> * <> * <> * <> @@ -43,6 +44,8 @@ include::./heartbeat-options.asciidoc[] include::./heartbeat-general-options.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/libbeat/docs/generalconfig.asciidoc b/libbeat/docs/generalconfig.asciidoc index e22c090cea4..a3f7426160a 100644 --- a/libbeat/docs/generalconfig.asciidoc +++ b/libbeat/docs/generalconfig.asciidoc @@ -103,70 +103,3 @@ processors in your config. Sets the maximum number of CPUs that can be executing simultaneously. The default is the number of logical CPUs available in the system. - -[float] -[[configuration-internal-queue]] -=== Configure the internal queue - -The `queue` namespace configures the queue type to be used. - -The Elastic Beats employ an internal queue for events to be published. The -queue is responsible for buffering and combining events into batches, which can -be consumed by the outputs. The outputs will use bulk operations to send a -batch of events in one transaction. - - -Example configuration: - -[source,yaml] ------------------------------------------------------------------------------- -queue.mem: - events: 4096 ------------------------------------------------------------------------------- - -==== Configure the Memory Qeueue -[[configuration-internal-queue-memory]] - -The memory queue keeps all events in memory. By default no flush interval is -configured. All events published to this queue will be directly consumed by the -outputs. An output will consume up to the outputs `bulk_max_size` events at once. - -Only after an event is dropped or acknowledged by the output, a new slot will -be available in the queue. - -By setting `flush.min_events` and `flush.timeout`, spooling in the queue is -enforced. In this case the output has to wait for a minimal number of events -to be available. - -This sample configuration forwards events to the output if 512 events are -available or the oldest available event is already waiting for 5s in the queue: - -[source,yaml] ------------------------------------------------------------------------------- -queue.mem: - events: 4096 - flush.min_events: 512 - flush.timeout: 5s ------------------------------------------------------------------------------- - -===== `events` - -Number of events the queue can store. - -The default value is 4096 events. - -===== `flush.min_events` - -Minimum number of events required for publishing. If this value is set to 0, the -output can start publishing events without additional waiting times. Otherwise -the output has to wait for more events to become available. - -The default value is 0. - -===== `flush.timeout` - -Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events -will be immediately available for consumption. - -The default values is 0s. - diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc new file mode 100644 index 00000000000..ea050e6440c --- /dev/null +++ b/libbeat/docs/queueconfig.asciidoc @@ -0,0 +1,69 @@ +[[configuring-internal-queue]] +== Configure the internal queue + +The {beatname_uc} uses an internal queue to store events before publishing them. The +queue is responsible for buffering and combining events into batches that can +be consumed by the outputs. The outputs will use bulk operations to send a +batch of events in one transaction. + +You can configure the type and behavior of the internal queue by setting options in the `queue` section of the +{beatname_lc}.yml+ config file. + + +Example configuration: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 +------------------------------------------------------------------------------ + +[[configuration-internal-queue-memory]] +=== Configure the memory qeueue + +The memory queue keeps all events in memory. It is the only queue type +supported right now. By default no flush interval is configured. All events +published to this queue will be directly consumed by the outputs. An output +will consume up to the outputs `bulk_max_size` events at once. + +Only after an event is dropped or acknowledged by the output, a new slot will +be available in the queue. + +To enforce spooling in the queue, set the `flush.min_events` and +`flush.timeout` options. + +This sample configuration forwards events to the output if 512 events are +available or the oldest available event is already waiting for 5s in the queue: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 + flush.min_events: 512 + flush.timeout: 5s +------------------------------------------------------------------------------ + +==== Configuration options + +You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file: + +===== `events` + +Number of events the queue can store. + +The default value is 4096 events. + +===== `flush.min_events` + +Minimum number of events required for publishing. If this value is set to 0, the +output can start publishing events without additional waiting times. Otherwise +the output has to wait for more events to become available. + +The default value is 0. + +===== `flush.timeout` + +Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events +will be immediately available for consumption. + +The default values is 0s. + diff --git a/metricbeat/docs/configuring-howto.asciidoc b/metricbeat/docs/configuring-howto.asciidoc index 27fdf740058..2df113c9763 100644 --- a/metricbeat/docs/configuring-howto.asciidoc +++ b/metricbeat/docs/configuring-howto.asciidoc @@ -22,6 +22,7 @@ The following topics describe how to configure Metricbeat: * <> * <> * <> +* <> * <> * <> * <> @@ -44,6 +45,8 @@ include::./metricbeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/packetbeat/docs/configuring-howto.asciidoc b/packetbeat/docs/configuring-howto.asciidoc index a324997beab..3fafba41be3 100644 --- a/packetbeat/docs/configuring-howto.asciidoc +++ b/packetbeat/docs/configuring-howto.asciidoc @@ -24,6 +24,7 @@ The following topics describe how to configure Packetbeat: * <> * <> * <> +* <> * <> * <> * <> @@ -45,6 +46,8 @@ include::./packetbeat-options.asciidoc[] include::./packetbeat-general-options.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/winlogbeat/docs/configuring-howto.asciidoc b/winlogbeat/docs/configuring-howto.asciidoc index 8a6fa05ebce..dd1e94d2748 100644 --- a/winlogbeat/docs/configuring-howto.asciidoc +++ b/winlogbeat/docs/configuring-howto.asciidoc @@ -19,6 +19,7 @@ The following topics describe how to configure Winlogbeat: * <> * <> +* <> * <> * <> * <> @@ -38,6 +39,8 @@ include::./winlogbeat-options.asciidoc[] include::./winlogbeat-general-options.asciidoc[] :win: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] From 92c457fb1c0e824632b72a688d3dea2f8c03fd38 Mon Sep 17 00:00:00 2001 From: urso Date: Tue, 15 Aug 2017 21:11:46 +0200 Subject: [PATCH 3/6] Remove flush_interval and spooler from outputconfig --- libbeat/docs/outputconfig.asciidoc | 37 +++++++++++------------------- 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index 751dd55650a..af88e0d4cde 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -283,21 +283,14 @@ However big batch sizes can also increase processing times, which might result i API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled, -Beats that publish single events (such as Packetbeat) send each event directly to -Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the -spooler size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. ===== `timeout` The http request timeout in seconds for the Elasticsearch request. The default is 90. -===== `flush_interval` - -The number of seconds to wait for new events between two bulk API index requests. -If `bulk_max_size` is reached before this interval expires, additional bulk index -requests are made. - ===== `ssl` Configuration options for SSL parameters like the certificate authority to use @@ -533,10 +526,10 @@ However big batch sizes can also increase processing times, which might result i API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled, -Beats that publish single events (such as Packetbeat) send each event directly to -Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the -spooler size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. + [[kafka-output]] === Configure the Kafka output @@ -731,10 +724,6 @@ The ACK reliability level required from broker. 0=no response, 1=wait for local Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error. -===== `flush_interval` - -The number of seconds to wait for new events between two producer API calls. - ===== `ssl` Configuration options for SSL parameters like the root CA for Kafka connections. See @@ -934,11 +923,9 @@ of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in -libbeat. When buffering is disabled, Beats that publish single events (such as -Packetbeat) send each event directly to Redis. Beats that publish -data in batches (such as Filebeat) send events in batches based on the spooler -size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. ===== `ssl` @@ -1068,7 +1055,9 @@ The maximum number of events to buffer internally during publishing. The default Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this setting does not affect how events are published. -Setting `bulk_max_size` to 0 disables buffering in libbeat. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. [[configuration-output-codec]] === Configure the ouput codec From 756ca3f80cac3ee8ece150b279f3573605b7bde3 Mon Sep 17 00:00:00 2001 From: urso Date: Tue, 15 Aug 2017 21:38:39 +0200 Subject: [PATCH 4/6] remove 'slot' --- libbeat/docs/queueconfig.asciidoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc index ea050e6440c..ab142c029f2 100644 --- a/libbeat/docs/queueconfig.asciidoc +++ b/libbeat/docs/queueconfig.asciidoc @@ -22,14 +22,14 @@ queue.mem: The memory queue keeps all events in memory. It is the only queue type supported right now. By default no flush interval is configured. All events -published to this queue will be directly consumed by the outputs. An output -will consume up to the outputs `bulk_max_size` events at once. +published to this queue will be directly consumed by the outputs. +The output's `bulk_max_size` setting limits the number of events being processed at once. -Only after an event is dropped or acknowledged by the output, a new slot will -be available in the queue. +The memory queue is waiting for the output to acknowledge or drop an events. If +the queue is full, no new events can be inserted into the memeory queue. Only +after the signal from the output will the queue free up space for more events to be accepted. -To enforce spooling in the queue, set the `flush.min_events` and -`flush.timeout` options. +To enforce spooling in the queue, set the `flush.min_events` and `flush.timeout` options. This sample configuration forwards events to the output if 512 events are available or the oldest available event is already waiting for 5s in the queue: From 6128f32969736245115d22626c7d1b4a288b40bf Mon Sep 17 00:00:00 2001 From: urso Date: Wed, 16 Aug 2017 15:51:10 +0200 Subject: [PATCH 5/6] more review changes - remove The - add [float] --- libbeat/docs/queueconfig.asciidoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc index ab142c029f2..4680492a38d 100644 --- a/libbeat/docs/queueconfig.asciidoc +++ b/libbeat/docs/queueconfig.asciidoc @@ -1,7 +1,7 @@ [[configuring-internal-queue]] == Configure the internal queue -The {beatname_uc} uses an internal queue to store events before publishing them. The +{beatname_uc} uses an internal queue to store events before publishing them. The queue is responsible for buffering and combining events into batches that can be consumed by the outputs. The outputs will use bulk operations to send a batch of events in one transaction. @@ -17,6 +17,7 @@ queue.mem: events: 4096 ------------------------------------------------------------------------------ +[float] [[configuration-internal-queue-memory]] === Configure the memory qeueue @@ -42,16 +43,19 @@ queue.mem: flush.timeout: 5s ------------------------------------------------------------------------------ +[float] ==== Configuration options You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file: +[float] ===== `events` Number of events the queue can store. The default value is 4096 events. +[float] ===== `flush.min_events` Minimum number of events required for publishing. If this value is set to 0, the @@ -60,6 +64,7 @@ the output has to wait for more events to become available. The default value is 0. +[float] ===== `flush.timeout` Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events From 9662217c7aede6085c43ba0f1bf05347bdc482c5 Mon Sep 17 00:00:00 2001 From: Nicolas Ruflin Date: Wed, 4 Oct 2017 11:27:37 +0200 Subject: [PATCH 6/6] Update queueconfig.asciidoc --- libbeat/docs/queueconfig.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc index 4680492a38d..3c4f77f4c74 100644 --- a/libbeat/docs/queueconfig.asciidoc +++ b/libbeat/docs/queueconfig.asciidoc @@ -26,7 +26,7 @@ supported right now. By default no flush interval is configured. All events published to this queue will be directly consumed by the outputs. The output's `bulk_max_size` setting limits the number of events being processed at once. -The memory queue is waiting for the output to acknowledge or drop an events. If +The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memeory queue. Only after the signal from the output will the queue free up space for more events to be accepted.