S6100 - T0 and T1 - Zebra crash seen with continous cold reboot- Master 253

[mini-nair-dell]

Observe Zebra crash in S6100- T0 and T1 topology with continuous reboot. The issue is not seen with a single reboot.

We can see this if we reboot the dut twice

Pls find the sylog and the backtrace attached

#0 0x00007fb09d189fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb09d18b42a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fb09e1d73bf in ?? () from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#3
#4 0x00007fb09e1e00ca in route_node_delete ()
from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#5 0x00007fb09e1e02f1 in route_next ()
from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#6 0x00007fb09b10a849 in ?? ()
from /usr/lib/x86_64-linux-gnu/frr/modules/zebra_fpm.so
#7 0x00007fb09e1e4a70 in thread_call ()
from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#8 0x00007fb09e1b4fe8 in frr_run ()
from /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0
#9 0x00005651f74aea93 in main ()

Thanks
Mini
zebra-syslog-1.docx
zebra-syslog.docx

[xinliu-seattle]

Will need core file to help with the investigation.

[xinliu-seattle]

Please provide >show tech file.

[mini-nair-dell]

sonic_dump_sonic-s6100-10_20200430_091155.tar.gz

[mini-nair-dell]

zebra.1588237631.40.core.gz

[mini-nair-dell]

syslog.2.gz

[mini-nair-dell]

Pls find the Core file and show tech attached.

Thanks
Mini

[xinliu-seattle]

Still see this issue on the master on 5/22. By continuous reboot - it is more 2 times.

[pavel-shirshov]

@xinliu-seattle Can you please define continuous reboot? I can't reproduce the crash.

[mini-nair-dell]

Hi @xinliu-seattle

Unable to test in the latest master image, as we are blocked with #4682.

By continuous reboot, i mean we got to reboot the dut > 2 times, and we see this issue. I have seen this with both T0 and T1.

THe script we use to test this is :
https://github.com/Azure/sonic-mgmt/blob/master/ansible/roles/test/tasks/continuous_reboot.yml

THanks
MIni

[mini-nair-dell]

Hi,

The issue is not seen in the master image 306 and Nov image 88. Zebra crash is not seen after continuous reboot

Thanks
Mini

[pavel-shirshov]

We haven't fixed anything.
Reopen to find a reason when zebra crashed on sonic restart.

[mini-nair-dell]

The issue not seen in the latest builds.

[pavel-shirshov]

zebra.1597718028.47.core.gz  zebra.1597804061.53.core.gz  zebra.1597946187.54.core.gz
zebra.1597736218.47.core.gz  zebra.1597804181.52.core.gz  zebra.1597946634.52.core.gz
zebra.1597782933.52.core.gz  zebra.1597804347.52.core.gz  zebra.1597947004.53.core.gz
zebra.1597783172.52.core.gz  zebra.1597804951.53.core.gz  zebra.1597948140.56.core.gz
zebra.1597783818.55.core.gz  zebra.1597805261.53.core.gz  zebra.1597949103.54.core.gz
zebra.1597784183.55.core.gz  zebra.1597836030.52.core.gz  zebra.1597949309.54.core.gz
zebra.1597784262.57.core.gz  zebra.1597836232.54.core.gz  zebra.1597949353.52.core.gz
zebra.1597784700.53.core.gz  zebra.1597836480.52.core.gz  zebra.1597949472.52.core.gz
zebra.1597784738.53.core.gz  zebra.1597836641.53.core.gz  zebra.1597949717.52.core.gz
zebra.1597784971.52.core.gz  zebra.1597837433.54.core.gz  zebra.1597949959.54.core.gz
zebra.1597785304.55.core.gz  zebra.1597838092.52.core.gz  zebra.1597950801.53.core.gz
zebra.1597785868.52.core.gz  zebra.1597838302.55.core.gz  zebra.1597951904.53.core.gz
zebra.1597786198.53.core.gz  zebra.1597838956.53.core.gz  zebra.1597952262.52.core.gz
zebra.1597786490.52.core.gz  zebra.1597839161.52.core.gz  zebra.1597952341.55.core.gz
zebra.1597786534.54.core.gz  zebra.1597839324.58.core.gz  zebra.1597953018.53.core.gz
zebra.1597786649.54.core.gz  zebra.1597840244.53.core.gz  zebra.1597957280.54.core.gz
zebra.1597786769.53.core.gz  zebra.1597840798.54.core.gz  zebra.1597957993.57.core.gz
zebra.1597786809.53.core.gz  zebra.1597841176.58.core.gz  zebra.1597958716.57.core.gz
zebra.1597787011.53.core.gz  zebra.1597841976.52.core.gz  zebra.1597959174.57.core.gz
zebra.1597794266.53.core.gz  zebra.1597842364.53.core.gz  zebra.1597959657.60.core.gz
zebra.1597794507.53.core.gz  zebra.1597842642.52.core.gz  zebra.1597960204.59.core.gz
zebra.1597794675.55.core.gz  zebra.1597842723.53.core.gz  zebra.1597960289.61.core.gz
zebra.1597794870.55.core.gz  zebra.1597843739.52.core.gz  zebra.1597960331.60.core.gz
zebra.1597796002.59.core.gz  zebra.1597844297.54.core.gz  zebra.1597960420.56.core.gz
zebra.1597796358.53.core.gz  zebra.1597934507.53.core.gz  zebra.1597960799.58.core.gz
zebra.1597796518.53.core.gz  zebra.1597934546.52.core.gz  zebra.1597960840.58.core.gz
zebra.1597797130.57.core.gz  zebra.1597935754.55.core.gz  zebra.1597961262.58.core.gz
zebra.1597797698.53.core.gz  zebra.1597936366.53.core.gz  zebra.1597962612.54.core.gz
zebra.1597797977.53.core.gz  zebra.1597936487.52.core.gz  zebra.1597962653.61.core.gz
zebra.1597798381.55.core.gz  zebra.1597937082.54.core.gz  zebra.1597963057.58.core.gz
zebra.1597799183.58.core.gz  zebra.1597937478.53.core.gz  zebra.1597963180.57.core.gz
zebra.1597799222.53.core.gz  zebra.1597938488.57.core.gz  zebra.1597963519.57.core.gz
zebra.1597799545.53.core.gz  zebra.1597940620.52.core.gz  zebra.1597963607.59.core.gz
zebra.1597800217.52.core.gz  zebra.1597940657.53.core.gz  zebra.1597963974.57.core.gz
zebra.1597800744.54.core.gz  zebra.1597941127.53.core.gz  zebra.1597964223.60.core.gz
zebra.1597800898.52.core.gz  zebra.1597944208.49.core.gz  zebra.1597964515.62.core.gz
zebra.1597801440.53.core.gz  zebra.1597944617.53.core.gz  zebra.1597964982.58.core.gz
zebra.1597801710.52.core.gz  zebra.1597944741.53.core.gz  zebra.1597965284.58.core.gz
zebra.1597801876.58.core.gz  zebra.1597945138.56.core.gz  zebra.1597965494.55.core.gz
zebra.1597802629.52.core.gz  zebra.1597945181.52.core.gz  zebra.1597965828.57.core.gz
zebra.1597803198.55.core.gz  zebra.1597945540.52.core.gz  zebra.1597966085.53.core.gz
zebra.1597803280.53.core.gz  zebra.1597945789.55.core.gz  zebra.1599007441.53.core.gz
zebra.1597803637.54.core.gz  zebra.1597946105.54.core.gz