Document load balancing modes in filebeat (#2780)

* Document load balancing modes in filebeat

add new file

filebeat/docs/index.asciidoc

@@ -38,6 +38,8 @@ include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]
 
+include::./load-balancing.asciidoc[]
+
 :allplatforms:
 include::../../libbeat/docs/yaml.asciidoc[]
 

filebeat/docs/load-balancing.asciidoc

@@ -0,0 +1,93 @@
+[[load-balancing]]
+== Load Balancing
+
+Filebeat provides configuration options that you can use to fine
+tune load balancing when sending events to multiple hosts. 
+
+To enable load balancing, you specify `loadbalance: true` when you configure
+the output. For example:
+
+[source,yaml]
+-------------------------------------------------------------------------------
+output.logstash:
+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+-------------------------------------------------------------------------------
+
+The `loadbalance` option is available for Redis, Logstash, and Elasticsearch
+outputs. The Kafka output handles load balancing internally.
+
+Filebeat can send events in a few different modes:
+
+* **Send events to one host after another:**
++
+By default, when you configure Filebeat to send events to multiple hosts
+(`loadbalance: true`), Filebeat will send the events to one host after
+another. This mode requires the least memory and CPU usage. This is not
+true load balancing in the sense that Filebeat does account for the load
+being processed by each host.
++
+The load balancer also supports multiple workers per host. The default is
+`worker: 1`. If you increase the number of workers, additional network
+connections will be used.  The total number of workers participating
+in load balancing is `number of hosts * workers`.
++
+Example: 
++
+[source,yaml]
+-------------------------------------------------------------------------------
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/*.log
+output.logstash:
+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+  worker: 2
+-------------------------------------------------------------------------------
++
+In this example, there are 4 workers participating in load balancing. 
+
+* **Send events to `N` hosts in lock-step:**
++
+You can configure Filebeat to send events to `N` hosts in lock-step by setting
+`spool_size = N * bulk_max_size`. This mode requires more memory and CPU usage
+than the previous mode.
++
+Example:
++
+[source,yaml]
+-------------------------------------------------------------------------------
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/*.log
+filebeat.spool_size: 4096
+output.logstash:
+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+  bulk_max_size: 2048
+-------------------------------------------------------------------------------
+
+* **Send events in parallel and asynchronously:**
++
+You can configure Filebeat to send events in parallel and asynchronously by
+setting `publish_async: true`. With this setting, Filebeat pushes a batch of
+lines and then prepares a new batch of lines while waiting for the output to
+ACK. This mode can improve load-balancing throughput, but requires the most
+memory and CPU usage. 
++
+Example:
++
+[source,yaml]
+-------------------------------------------------------------------------------
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/*.log
+filebeat.publish_async: true
+output.logstash:
+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+-------------------------------------------------------------------------------
+

filebeat/docs/reference/configuration/filebeat-options.asciidoc

@@ -468,13 +468,15 @@ the specified value.
 filebeat.spool_size: 2048
 -------------------------------------------------------------------------------------
 
+See <<load-balancing>> for more information about how this setting affects load balancing.
 
 ===== publish_async
 
 If enabled, the publisher pipeline in Filebeat operates in async mode preparing
 a new batch of lines while waiting for ACK. This option can improve load-balancing
 throughput at the cost of increased memory usage. The default value is false.
 
+See <<load-balancing>> for more information about how this setting affects load balancing.
 
 ===== idle_timeout