Provide bytesize filtering capability

[ppf2]

Discussed with the LS team earlier today. There are cases where the events coming in can be of varied sizes, from the regular 3-4Kb per event, to something that is potentially huge like 100+Mb (eg. application log dump of a xml response). While some users may be able to simply drop the events, some users do have to retain them for compliancy. One optimization idea being implemented is to create a custom ruby filter in LS to calculate the bytesize of the message field of an event, and send it through to separate topics/lists in a queuing system and have separate LS indexers to handle abnormally large events and have other LS indexers handle the regular sized events. @suyograo suggests filing an enhancement request for beats for it is best to perform such byte-size filtering further upstream.

[ruflin]

It seems like there are several options that could be pursued her from the beats side:

• Drop events which are too large. This should be part of filtering: Proposal Generic Filtering (Phase 2) #451
• Reduce events which are too big. For this max_bytes can already be used: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_max_bytes

A third option could be routing of events to different instances of LS. Currently from my point of view I don't feel like that is something that should happen on the beats side as when it comes to sending events we should keep the complexity to a minimum on the beats side.

[monicasarbu]

This should be easily done with generic filtering feature: #451

[ppf2]

Any progress on this? This will help users who want the ability to filter out, tag and route occasional large/gigantic events closer to the source.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.