Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default bulk sizes #628

Merged
merged 4 commits into from
Jan 7, 2016
Merged

default bulk sizes #628

merged 4 commits into from
Jan 7, 2016

Conversation

urso
Copy link

@urso urso commented Jan 5, 2016

related to #587

  • set default spooler size in filebeat to 2048, in order to increase number of
    lines to be combined into one bulk request
  • set default bulk max size to 2048, to guarantee not to split bigger bulk
    requests into too many small requests.

Tested different spooler and bulk sizes using NASA-HTTP logs:

spooler          bulk  | cpu(%)   mem-res(MB)  throughput(1k lines/s)
---------------------------------------------------------------------
 1024            200   |  26        20              2.1
 1024            1024  |  55        20             15.7
 2048            2048  |  66        25             18.2
 3072            3072  |  66        28             18.5
 4096            4096  |  70        33             18.7

@tsg
Copy link
Contributor

tsg commented Jan 5, 2016

Doesn't this potentially cause memory issues for Packetbeat? It can be 1000 * 2048 transactions stored in memory, right?

@urso
Copy link
Author

urso commented Jan 5, 2016

We already have some memory issues in topbeat and packetbeat. I have another PR in the pipeline to address this issue.

@tsg
Copy link
Contributor

tsg commented Jan 5, 2016

Ok, then we just need to make sure both changes are merged in all the same branches. If the other change is small, might then make sense to put it in this PR?

@dedemorton
Copy link
Contributor

Should we also update the doc to show the new default in the example here? https://www.elastic.co/guide/en/beats/filebeat/1.0.1/filebeat-configuration-details.html#_spool_size

@urso
Copy link
Author

urso commented Jan 5, 2016

@dedemorton good point. thanks

@urso urso force-pushed the enh/default-bulk-sizes branch from b1fa7ba to 12e0e36 Compare January 6, 2016 15:59
@urso
Copy link
Author

urso commented Jan 6, 2016

see PR #649 for limiting total number of buffered events

@ruflin
Copy link
Contributor

ruflin commented Jan 7, 2016

LGTM

urso added 4 commits January 7, 2016 22:40
- set default spooler size in filebeat to 2048, in order to increase number of
  lines to be combined into one bulk request

- set default bulk max size to 2048, to guarantee not to split bigger bulk
  requests into too many small requests.

Tested different spooler and bulk sizes using NASA-HTTP logs:

spooler          bulk  | cpu(%)   mem-res(MB)  throughput(1k lines/s)
---------------------------------------------------------------------
 1024            200   |  26        20              2.1
 1024            1024  |  55        20             15.7
 2048            2048  |  66        25             18.2
 3072            3072  |  66        28             18.5
 4096            4096  |  70        33             18.7
- add changelog entries about spool_size and bulk_max_size defaults
@urso urso force-pushed the enh/default-bulk-sizes branch from 0b61e39 to cb6eea1 Compare January 7, 2016 21:41
ruflin added a commit that referenced this pull request Jan 7, 2016
@ruflin ruflin merged commit 9177c74 into elastic:1.1 Jan 7, 2016
@urso urso deleted the enh/default-bulk-sizes branch January 27, 2016 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants