Document load balancing modes in filebeat #2780

dedemorton · 2016-10-14T00:04:00Z

Added to resolve #1852

tsg · 2016-10-14T09:26:37Z

Let's better wait on @urso to review this one.

urso

Some more details how load balancing works internally: https://github.com/elastic/beats/blob/master/libbeat/outputs/mode/lb/lb.go#L13

urso · 2016-10-17T12:36:30Z

filebeat/docs/load-balancing.asciidoc

+[[load-balancing]]
+== Configuring Load Balancing
+
+//REVIEWERS: Which outputs support load balancing? I only see the loadbalance option described under Logstash and Redis. Are any other outputs supported?


redis, logstash and elasticsearch.

Note: Kafka internally uses the same load-balancer module, but only for asynchronous error handling. Loadbalancing for kafka is internally solved (a little different) by client library used.

urso · 2016-10-17T12:38:23Z

filebeat/docs/load-balancing.asciidoc

+* **Send events to one host after another:**
+
+By default, when you configure Filebeat to send events to multiple hosts
+(`load_balance: true`), Filebeat will send the events to one host after


s/load_balance/loadbalance/

plus:

filebeat.publish_async: false

filebeat.spool_size = output.X.bulk_max_size

The host choosen for the next batch is kind of random.

This is not really load-balancing from client point of view, but can still balance out network usage somehow if multiple filebeat instances are connected to one cluster.

Better option might be to set loadbalance: false, though. In this mode filebeat chooses on logstash only by random + there is a chance different filebeat instances choosing different logstash instances in a logstash cluster still somehow balancing some load.

@urso I'm not sure what you want me to change or add here. I don't think I need to mention publish_async until later because the default is false. Also, I'm not sure if it makes sense to recommend changing spool_size until later in this topic. I've changed the intro to say, "Filebeat can send events in a few different modes" since it sounds like the first approach is not truly load balancing (even though the loadbalance option is enabled). I'll push my changes in a bit, and maybe you can tell me specifically what I need to add or change to make the section right.

urso · 2016-10-17T12:43:26Z

filebeat/docs/load-balancing.asciidoc

+You can configure Filebeat to send events to `N` hosts in lock-step by setting
+`spool_size = N * bulk_max_size`. This mode requires more memory and CPU usage
+than the previous mode.
+


third mode:

set filebeat.publish_async: true and filebeat will try to push batches and wait for ACKs asynchronously. In this mode, filebeat will try to fully load-balance batches collected by spooler. LS instances being slower might handle less batches. This mode should get the highest throughput at the cost of highest memory usage: Batches are kept sorted in memory until ACKed by output. ACK might be out of order, but cleanup is done in order, potentially keeping ACKed batches in memory until older still active batches are ACKed.

@urso I'm not sure what you want me to change (vs what you are offering here as explanation). My inclindation is to not talk about how we manage batches in memory because that's an internal implementation detail that users don't really want to know (and we might want to change someday). Best not to document unless users can configure it in someway.

I got a little confused with asciidoc a little. I did think the third mode to be a separate section and didn't see it initially. My bad.

urso · 2016-10-17T12:52:37Z

filebeat/docs/load-balancing.asciidoc

+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+  bulk_max_size: 2048
+-------------------------------------------------------------------------------


the loadbalancer supports multiple workers per host. By default worker: 1, but if number of workers is increased, more network connections will be in use.

e.g. one can configure on logstash host with worker: 2 :

output.logstash: hosts: ["localhost:5044"] loadbalance: true worker: 2 bulk_max_size: 2048

@urso I'm adding this to the section where you mention it, but I'm wondering if it should go in the first section about enabling loadbalance.

Yes, agree it should go in the first section about enabling loadbalance. The total number of workers participating in load balancing is #hosts * worker.

urso · 2016-10-17T12:54:07Z

filebeat/docs/load-balancing.asciidoc

+-------------------------------------------------------------------------------
+
+
+//REVIEWERS: Please confirm the accuracy of the config examples here. I'm guessing at the config and haven't tested these examples. It would probably be better to use a more realistic path for /var/log/*.log (please suggest one). I'd like to show the full config because it helps users understand where the settings fit into the overall yaml file.


/var/log/*.log is even our default and should be fine on linux.

urso · 2016-10-17T12:56:20Z

filebeat/docs/load-balancing.asciidoc

+
+//REVIEWERS: Please confirm the accuracy of the config examples here. I'm guessing at the config and haven't tested these examples. It would probably be better to use a more realistic path for /var/log/*.log (please suggest one). I'd like to show the full config because it helps users understand where the settings fit into the overall yaml file.
+
+//REVIEWERS: I'm a bit confused in the above example because bulk_max_size does not appear under output.logstash in filebeat.full.yml. Is this intentional? It does show up as an option in https://www.elastic.co/guide/en/beats/filebeat/5.0/logstash-output.html, so I assume it's valid, correct?


bulk_max_size should be added to libbeat/_meta/config*yml . There is another ticket for checking for potentially missing output settings. Only docs have been synced most recently.

urso · 2016-10-17T12:57:57Z

filebeat/docs/load-balancing.asciidoc

+- input_type: log
+  paths:
+    - /var/log/*.log
+filebeat.publish_async: false


s/false/true

Oh, this is the third mode, hehe.

dedemorton · 2016-10-24T23:06:42Z

@urso I forgot to remove the in progress label for the PR. Is this one good to go?

urso · 2016-10-25T12:15:44Z

filebeat/docs/load-balancing.asciidoc

+  hosts: ["localhost:5044", "localhost:5045"]
+  loadbalance: true
+  bulk_max_size: 2048
+-------------------------------------------------------------------------------


Yes, agree it should go in the first section about enabling loadbalance. The total number of workers participating in load balancing is #hosts * worker.

urso · 2016-10-25T12:16:29Z

filebeat/docs/load-balancing.asciidoc

+You can configure Filebeat to send events to `N` hosts in lock-step by setting
+`spool_size = N * bulk_max_size`. This mode requires more memory and CPU usage
+than the previous mode.
+


I got a little confused with asciidoc a little. I did think the third mode to be a separate section and didn't see it initially. My bad.

urso · 2016-10-25T12:17:25Z

filebeat/docs/load-balancing.asciidoc

+
+By default, when you configure Filebeat to send events to multiple hosts
+(`loadbalance: true`), Filebeat will send the events to one host after
+another. This mode requires the least memory and CPU usage.


Maybe add a note about this not really being load balancing?

add new file

dedemorton added docs review labels Oct 14, 2016

urso reviewed Oct 17, 2016

View reviewed changes

dedemorton added in progress Pull request is currently in progress. and removed in progress Pull request is currently in progress. labels Oct 19, 2016

urso approved these changes Oct 25, 2016

View reviewed changes

dedemorton force-pushed the config_load_balancing branch from 2117928 to fedf7b5 Compare October 25, 2016 23:41

dedemorton added 3 commits October 25, 2016 16:49

Document load balancing modes in filebeat

9e59452

add new file

Fix issues from review

919c6a1

More changes from review

cd9d3c8

dedemorton force-pushed the config_load_balancing branch from fedf7b5 to cd9d3c8 Compare October 25, 2016 23:51

urso merged commit d92113b into elastic:master Oct 25, 2016

dedemorton mentioned this pull request Oct 26, 2016

Backport: doc changes 2852, 2851, 2850, adn 2780 into 5.0 #2863

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document load balancing modes in filebeat #2780

Document load balancing modes in filebeat #2780

dedemorton commented Oct 14, 2016

tsg commented Oct 14, 2016

urso left a comment

urso Oct 17, 2016

urso Oct 17, 2016

dedemorton Oct 21, 2016

urso Oct 17, 2016

dedemorton Oct 21, 2016

urso Oct 25, 2016

urso Oct 17, 2016

dedemorton Oct 21, 2016

urso Oct 25, 2016

urso Oct 17, 2016

urso Oct 17, 2016

urso Oct 17, 2016

dedemorton commented Oct 24, 2016

urso Oct 25, 2016

urso Oct 25, 2016

urso Oct 25, 2016

		-------------------------------------------------------------------------------


		//REVIEWERS: Please confirm the accuracy of the config examples here. I'm guessing at the config and haven't tested these examples. It would probably be better to use a more realistic path for /var/log/*.log (please suggest one). I'd like to show the full config because it helps users understand where the settings fit into the overall yaml file.


		//REVIEWERS: Please confirm the accuracy of the config examples here. I'm guessing at the config and haven't tested these examples. It would probably be better to use a more realistic path for /var/log/*.log (please suggest one). I'd like to show the full config because it helps users understand where the settings fit into the overall yaml file.

		//REVIEWERS: I'm a bit confused in the above example because bulk_max_size does not appear under output.logstash in filebeat.full.yml. Is this intentional? It does show up as an option in https://www.elastic.co/guide/en/beats/filebeat/5.0/logstash-output.html, so I assume it's valid, correct?

Document load balancing modes in filebeat #2780

Document load balancing modes in filebeat #2780

Conversation

dedemorton commented Oct 14, 2016

tsg commented Oct 14, 2016

urso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedemorton commented Oct 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment