Detect and log control plane drops #20481
Conversation
prabhataravind
force-pushed
the
paravind/control_plane_drops
branch
3 times, most recently
from
df35e9e
to
5a23584
Compare
Signed-off-by: Prabhat Aravind <paravind@microsoft.com>
prabhataravind
force-pushed
the
paravind/control_plane_drops
branch
from
5a23584
to
c65adf1
Compare
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| # Drop count is in the second column for each CPU | ||
| if len(stat) > 1: | ||
| drop_count += int(stat[1], 16) | ||
|
|
There was a problem hiding this comment.
Close all the file descriptors?
There was a problem hiding this comment.
I am using "with open", so explicit close is not needed for file descriptors.
| last_drop_count = get_softnet_dropped_count() | ||
| with open(drop_count_stash, 'w') as f: | ||
| f.write(str(last_drop_count)) | ||
| return False |
There was a problem hiding this comment.
Any option to reset the last_drop_counter during config reload or warmboot??
There was a problem hiding this comment.
During warmboot, the counter will automatically be reset (after kexec to new kernel). For config reload case, we might have a stale value for potentially 1 cycle and then it will fix itself. The subsequent read will clear up the drop counter stash.
There was a problem hiding this comment.
Also verified that during "config reload", /proc/net/softnet_stat drop count does increase a bit, so we don't need to reset the last dropped counter after "config reload".
For eg: before config reload, the second column had a count of 0 and after config reload, the second column had a count of 00000015 (hex).
| for line in f: | ||
| if line.strip(): # Ensure the line is not empty | ||
| stat = line.split() | ||
| # Drop count is in the second column for each CPU |
There was a problem hiding this comment.
I think second column in softnet_stat represents drop counter in all CPU architectures, please add a note.
There was a problem hiding this comment.
Okay, I'll confirm for all cpu architectures.
There was a problem hiding this comment.
Per https://github.com/torvalds/linux/blob/v5.10/net/core/net-procfs.c#L153, the dropped count seems to be always in column 2 independent of the cpu architecture.
There was a problem hiding this comment.
Please include this reference in the comment.
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| f.write(str(current_drop_count)) | ||
|
|
||
| if current_drop_count > last_drop_count: | ||
| write_syslog("control_plane_drop_check: Kernel packet drops detected") |
There was a problem hiding this comment.
Can we add the drop count in the log?
Signed-off-by: Prabhat Aravind <paravind@microsoft.com>
Why I did it
Packet drops in kernel can go unnoticed for a long while until they cause protocols like BGP/LACP to timeout potentially causing a big impact. The change here is to detect that proactively and alert using monit
Work item tracking
How I did it
By adding a monit script that runs every 5 minutes and checks if drop counters reported by /proc/net/softnet_stat is increasing or not.
How to verify it
By running " sudo monit status controlPlaneDropCheck" and verifying that the status aligns with the drops reported in /proc/net/softnet_stat
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Master
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)