[201803] Set a rate limit on syslog messages from all Docker containers

[jleveque]

Enable rate limiting on rsyslog messages originating from each Docker container. Each container is limited to 10,000 messages per 5-minute interval.

[nikos-github]

Can you please explain why we need to do this?

[jleveque]

@nikos-github: This is to prevent overloading remote syslog servers if a process starts spamming the syslog. For example, if certain ASICs present a parity error that does not get auto-corrected, the syncd container can be responsible for sending > 200,000 syslog messages every 5 minutes.

[nikos-github]

@jleveque What is the effect of this on debug level setting on daemons in containers? How did you determine the numbers? Are they meant to be per PID? If so, it implies that we think 30 msgs per second for a PID is the limit. Is that actually enough at boot time? Why not implement the rate limiting at the source/syncd for the example event that you mentioned or filter it and not send it to syslog server once you know what to look for? I'm not convinced this is the right approach since there are valid cases where we may want to generate and see more than 30 msgs per second from a PID.

[jleveque]

@nikos-github:

What is the effect of this on debug level setting on daemons in containers?

Debug level settings were not taken into effect for this PR. This PR is based off the level of log messages of an image built with current default settings.

How did you determine the numbers? Are they meant to be per PID?

The numbers are not per-PID, but rather per-container. The numbers were an average of a few switches configured with 120+ ports. During normal operation, the highest rate of messages occurs at boot. I found with these devices, the average maximum rate of total messages (base image + all containers) is ~12,000 per 5 minute interval. The average maximum rate per container at boot is as follows:

Container	Max Messages Per 5-Minute Interval
syncd	3871
swss	2761
teamd	2097
lldp	415
snmp	325
pmon	110
dhcp_relay	67
bgp	31
radv	13
database	9

Thus, setting the per-container rate limit to 10,000 messages per 5-minute interval should not interfere with boot messages. If we are concerned at all, I think we could increase the limit to 20,000 or even a bit higher. The idea here is simply to prevent a rogue process from spamming a remote syslog server, so there is a bit of flexibility.