Filebeat message rate limit or throttle

[zez3]

See Graylog2/graylog2-server#6939

[zez3]

elastic/beats#17775

[mpfz0r]

Thanks for your input. I think we should handle this on the server side.
graylog has the ability to throttle inputs if the journal gets too full,
it's just not trivial to implement this for the beats input :-/

[zez3]

This(throttling) already happens but does not bring much. https://docs.graylog.org/en/latest/pages/sending_data.html#input-throttling
Or you mean something else?

My node journal is filled because of the load and speed with which the messages get from filebeat+sidecars and then the message ingress moves to the second node and so forth. Both GL nodes and the agent host are CPU skyrocketing during this initial state of ingress.
Perhaps ES will bring a solution to their filebeat.
Should I rename this to rate limiting?

[zez3]

What works today and would aid on the agent side of the problem is to set 1 core

https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-general-options.html#_max_procs

And reported here
elastic/beats#2789 (comment)

[mpfz0r]

This(throttling) already happens but does not bring much. https://docs.graylog.org/en/latest/pages/sending_data.html#input-throttling
Or you mean something else?

That's the kind of throttling I'm referring to, yes.
It doesn't work because it's not implemented for the Beats input (yet).

[zez3]

It doesn't work because it's not implemented for the Beats input (yet).

I see, it's the 3rd criteria point that would be used for this case.

https://docs.graylog.org/en/latest/pages/sending_data.html#throttling-criteria

3.Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).

Still it would be nice to be able to change this value of 20k mess/sec

[zez3]

Looks like Elastic fixed it. From Filebeat 7.11 onward there is a new message limit that you can set.
https://www.elastic.co/guide/en/beats/filebeat/master/rate-limit.html

[zez3]

"Events that exceed the rate limit are dropped"