[security] syslog floods during DoS attack on REST server

[jusherma]

Description

Sending malicious traffic to the REST server causes log flooding. These logs can rapidly fill the syslog. During tests, more than 1.5 MB of these messages were written per minute of DoS attack. This could be used to force log rotation, concealing earlier malicious activity from logs. It could also cause a system outage by filling up the disk.

Steps to reproduce the issue:

Run these commands in parallel from Kali Linux, where 172.31.0.2 is the IP address of an interface on SONiC and 172.31.0.6 is an unassigned IP address

arpspoof -i eth1 -t 172.31.0.2 172.31.0.6 > /tmp/arpspoof 2>&1 &
srvr -SAa -i eth1 172.31.0.6 > /tmp/srvr 2>&1 &
hping3 172.31.0.2 -p 443 -S -a 172.31.0.6 -i u10000 -q

Describe the results you received:

Within about 2 minutes, warnings about TLS handshake timeouts will start rapidly flooding /var/log/syslog:

Dec  3 08:42:34.123901 sonic INFO mgmt-framework#/supervisord: rest-server 2022/12/03 08:42:34 http: TLS handshake error from 172.31.0.4:1558: read tcp 172.31.0.2:443->172.31.0.4:1558: read: connection timed out

Describe the results you expected:

1. REST server should only be listening to the mgmt interface
2. Minor errors, especially those triggered by unauthenticated users, should not be logged or should be rate-limited
3. fail2ban should be used to block abusive or malicious clients

[adyeung]

@jeff-yin this is relating to RBAC, pls help followup with your team

[jeff-yin]

@adyeung I don't think this is a role-based access control issue, as it has to do with unauthenticated users. This is more related to the configuration and logging done by the REST server.

1. REST server should only be listening to the mgmt interface

The REST server should be configurable to do this, or a separate mgmt VRF can be configured as well.

2. Minor errors, especially those triggered by unauthenticated users, should not be logged or should be rate-limited

Need to take a look at how the REST server is passing error messages to the syslog. But even so, some of these logs are worth keeping for troubleshooting purposes.

3. fail2ban should be used to block abusive or malicious clients

I think this is more important than point 2. We should configure this on the host right?

[jeff-yin]

Sorry, unassigned myself thinking I could re-assign. But I can't. Feel free to re-assign to me for now.

[adyeung]

Discussing offline over email, I will move assignee once we reach a conclusion offline

[adyeung]

@sachinholla from BRCM working on a fix

[jeff-yin]

Thanks @adyeung and @sachinholla

[sachinholla]

hi @jusherma,

How to install the "srvr" tool? I did not find it in kali tools page.

About REST server listening to the mgmt interface only -- most of the deployments I have seen wanted inband management. Will keep the current behavior as is and consider supporting a bind interface/vrf option in future.
Note: REST server followed the behavior of the telemetry service which is already in use in many production environments today. Telemetry server also listens on all interfaces.

About fail2ban -- if I understand correctly, this will be a new service monitoring the logs and adjusting he firewall rules (may be ACLs??).. It should be discussed & tracked as a separate security enhancement for sonic. Not specific to REST server.

[jusherma]

hi @jusherma,

How to install the "srvr" tool? I did not find it in kali tools page.

I think you should be able to reproduce this even without the srvr tool. Let me know if you cannot

About REST server listening to the mgmt interface only -- most of the deployments I have seen wanted inband management. Will keep the current behavior as is and consider supporting a bind interface/vrf option in future. Note: REST server followed the behavior of the telemetry service which is already in use in many production environments today. Telemetry server also listens on all interfaces.

Binding a server to all interfaces by default is poor security posture. It's ok to allow users to bind telemetry or REST server to all interfaces, but using that as a default is a security risk. I already have opened another issue about telemetry's use of binding to 0.0.0.0.

About fail2ban -- if I understand correctly, this will be a new service monitoring the logs and adjusting he firewall rules (may be ACLs??).. It should be discussed & tracked as a separate security enhancement for sonic. Not specific to REST server.

You're correct--fail2ban monitors log files and uses iptable rules to temporarily block abusive clients.

fail2ban needs to both be (a) added to SONIC and (b) enabled for each service we wish to protect. Regardless of how this task is tracked, work will need to be done to enable it for the REST server specifically.

[adyeung]

sonic-net/sonic-mgmt-framework#102