Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate per event and bulk-event queues #649

Merged
merged 2 commits into from
Jan 7, 2016
Merged

Conversation

urso
Copy link

@urso urso commented Jan 6, 2016

Have separate queues with different buffer sizes
for single events and bulk events in publisher.

Default queues size for single events is 1000 elements and for bulk events is set to 0.

Default values dramatically reduce total number of buffered events and memory usage in case elasticsearch/logstash become unresponsive.

Introduce (hidden) configuration in shipper section:

  • queue_size
  • bulk_queue_size

When using async publisher number of buffered events is about given by:

Q_e = queue size (default 1000)
Q_b = bulk queue size (default 0)
B = bulk_max_size
W = number of load balancing workers (only if load balancing is enabled)
N = number of events in memory

N = 2*(Q_e) + (2 + Q_b + W)*B

using default values:
Q_e = 1000
Q_b = 0
B = 2048

  • topbeat/packetbeat with default values + single output:

    N = 2*1000 + 2*2048 = 6096 events
    
  • topbeat/packetbeat with default values + load balancing with 4 workers:

    N = 2*1000 + 6*2048 = 14288 events
    

@urso
Copy link
Author

urso commented Jan 6, 2016

resolves #516
resolves elastic/libbeat#337

@urso urso mentioned this pull request Jan 6, 2016
@urso urso added the review label Jan 7, 2016
@@ -79,6 +79,10 @@ type ShipperConfig struct {
Topology_expire int
Tags []string
Geoip common.Geoip

// internal publisher queue sizes
QueueSize *int `yaml:"queue_size"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm +1 on making this also visible in the config file but with a warning, that people that change it must understand what they are doing.

@ruflin
Copy link
Member

ruflin commented Jan 7, 2016

LGTM. CHANGELOG should be updated. See my comment above.

urso added 2 commits January 7, 2016 21:47
Have separate queues with different buffer sizes
for single events and bulk events in publisher.

Default queues size for single events is 1000 elements and for bulk events is set to 0.

Default values dramatically reduce total number of buffered events and memory usage in case elasticsearch/logstash become unresponsive.

Introduce (hidden) configuration in shipper section:
  - queue_size
  - bulk_queue_size

When using async publisher number of buffered events is about given by:

Q_e = queue size (default 1000)
Q_b = bulk queue size (default 0)
B = bulk_max_size
W = number of load balancing workers (only if load balancing is enabled)
N = number of events in memory

    N = 2*(Q_e) + (2 + Q_b + W)*B

using default values:
   Q_e = 1000
   Q_b = 0
   B = 2048

  - topbeat/packetbeat with default values + single output:

      N = 2*1000 + 2*2048 = 6096 events

  - topbeat/packetbeat with default values + load balancing with 4 workers:

      N = 2*1000 + 6*2048 = 14288 events
ruflin added a commit that referenced this pull request Jan 7, 2016
Separate per event and bulk-event queues
@ruflin ruflin merged commit 4304cc7 into elastic:1.1 Jan 7, 2016
@urso urso deleted the enh/pub-queues branch February 1, 2016 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants