default bulk sizes #628
Merged
default bulk sizes #628
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Doesn't this potentially cause memory issues for Packetbeat? It can be 1000 * 2048 transactions stored in memory, right? |
|
We already have some memory issues in topbeat and packetbeat. I have another PR in the pipeline to address this issue. |
|
Ok, then we just need to make sure both changes are merged in all the same branches. If the other change is small, might then make sense to put it in this PR? |
|
Should we also update the doc to show the new default in the example here? https://www.elastic.co/guide/en/beats/filebeat/1.0.1/filebeat-configuration-details.html#_spool_size |
|
@dedemorton good point. thanks |
urso
force-pushed
the
enh/default-bulk-sizes
branch
from
b1fa7ba
to
12e0e36
Compare
|
see PR #649 for limiting total number of buffered events |
|
LGTM |
added 4 commits
January 7, 2016 22:40
- set default spooler size in filebeat to 2048, in order to increase number of lines to be combined into one bulk request - set default bulk max size to 2048, to guarantee not to split bigger bulk requests into too many small requests. Tested different spooler and bulk sizes using NASA-HTTP logs: spooler bulk | cpu(%) mem-res(MB) throughput(1k lines/s) --------------------------------------------------------------------- 1024 200 | 26 20 2.1 1024 1024 | 55 20 15.7 2048 2048 | 66 25 18.2 3072 3072 | 66 28 18.5 4096 4096 | 70 33 18.7
- add changelog entries about spool_size and bulk_max_size defaults
urso
force-pushed
the
enh/default-bulk-sizes
branch
from
0b61e39
to
cb6eea1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
related to #587
lines to be combined into one bulk request
requests into too many small requests.
Tested different spooler and bulk sizes using NASA-HTTP logs: