FRR related error message output during nightly test #19047
Labels
Triaged
this issue has been triaged
Comments
8 tasks
StormLiangMS
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this issue
May 23, 2024
What is the motivation for this PR? Many cases failed during teardown due to loganalyzer with below FRR related error messages. The failure only happen on the device which runs slim image. bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: no fpm use-next-hop-groups bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 12: fpm address 127.0.0.1 ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [MVZKX-EG443][EC 33554452] bgp_process_packet: BGP OPEN receipt failed for peer: XXX ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[50]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[49]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[49]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#zebra[36]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No route to host, type=RTM_NEWROUTE(24), seq=95412, pid=3576249171 ERR bgp#bgpd[65]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#zebra[36]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No route to host, type=RTM_NEWROUTE(24), seq=95495, pid=3576249171 ERR bgp#bgpd[65]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[65]: [XETTR-D5MR0][EC 100663316] Attempting to process an I/O event but for fd: 66(8) no thread to handle this! ERR bgp#bgpd[53]: [H4B4J-DCW2R][EC 33554455] XXX [Error] bgp_read_packet error: Connection reset by peer ERR bgp#zebra[62]: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: no fpm use-next-hop-groups ERR bgp#zebra[62]: [SHWNK-NWT5S][EC 100663304] No such command on config line 12: fpm address 127.0.0.1 ERR bgp#zebra[62]: [RFREB-PAV4B][EC 100663299] vty_read: read error on vty client fd 26, closing: Connection reset by peer ERR bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: no fpm use-next-hop-groups ERR bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 12: fpm address 127.0.0.1 ERR bgp#bgpd[51]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[49]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#bgpd[51]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF PortChannel999 in VRF 0 ERR bgp#bgpd[51]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF PortChannel999 in VRF 0 ERR bgp#bgpd[48]: [P3GYW-PBKQG][EC 33554466] XXX [FSM] unexpected packet received in state OpenSent ERR bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: no fpm use-next-hop-groups ERR bgp#zebra[36]: [SHWNK-NWT5S][EC 100663304] No such command on config line 12: fpm address 127.0.0.1 How did you do it? Slim image's rsyslog config has been modified by sonic-net/sonic-buildimage#17905 by mistake. So the FRR error message would output into syslog and cause teardown failure. Created a github issue sonic-net/sonic-buildimage#19047 to track the issue and ignore these related error message until the issue fixed to make the nightly build more stable. r, ".* ERR bgp#bgpd.* unexpected packet received in state OpenSent" r, ".* ERR bgp#bgpd.* INTERFACE_STATE: Cannot find IF .*" r, ".* ERR bgp#bgpd.* bgp_process_packet: BGP OPEN receipt failed for peer.*" r, ".* ERR bgp#bgpd.* bgp_read_packet error: Connection reset by peer.*" r, ".* ERR bgp#bgpd.* Attempting to process an I/O event but for fd.*" r, ".* ERR bgp#zebra.* No such command on config line .*" r, ".* ERR bgp#zebra.* error: No route to host.*" r, ".* ERR bgp#zebra.* read error on vty client fd .*" How did you verify/test it? NA Any platform specific information?
|
Does bgp session come up or stay down? |
thanks for asking, bgp sessions are up. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
There are error messages output in the following cases with the 202311 image.
<style> </style>Describe the results you received:
One of the failure log:
failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1"
Exception:
match: 12
expected_match: 0
expected_missing_match: 0
Match Messages:
May 20 09:00:19.304424 str-a7060cx-acs-10 ERR bgp#bgpd[51]: [P3GYW-PBKQG][EC 33554466] 10.0.0.5 [FSM] unexpected packet received in state OpenSent
May 20 09:00:20.310659 str-a7060cx-acs-10 ERR bgp#bgpd[51]: [P3GYW-PBKQG][EC 33554466] 10.0.0.5 [FSM] unexpected packet received in state OpenSent
May 20 09:00:21.315669 str-a7060cx-acs-10 ERR bgp#bgpd[51]: [P3GYW-PBKQG][EC 33554466] 10.0.0.5 [FSM] unexpected packet received in state OpenSent
Describe the results you expected:
Output of
show version:20231110.10
Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: