S6100-T0-syncd crash seen on cold reboot- HEAD.253-2872d802

[mini-nair-dell]

Description:

• Load the s6100 with the latest master in T0 topology
• Reboot the node
• We can see syncd crashes, once the switch comes up.

root@sonic-s6100-07:~# exit
root@sonic-s6100-07:/var/core# ls -l
total 458308
-rw-rw-rw- 1 root root 10455628 Apr 15 05:30 syncd.1586928610.29.core.gz
-rw-rw-rw- 1 root root 10455365 Apr 15 05:39 syncd.1586929172.30.core.gz
-rw-rw-rw- 1 root root 10469609 Apr 15 05:53 syncd.1586930017.29.core.gz
-rw-rw-rw- 1 root root 10438043 Apr 15 05:59 syncd.1586930362.29.core.gz
-rw-rw-rw- 1 root root 427474944 Apr 15 06:45 syncd.1586933129.29.core
root@sonic-s6100-07:/var/core#

The back trace :

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/syncd --diag -u -p /etc/sai.d/sai.profile'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f4d7a860fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f4d846af380 (LWP 29))]
(gdb) bt
#0 0x00007f4d7a860fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4d7a86242a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f4d7a89ec00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f4d7a8a4fc6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f4d7a8a580e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f4d7d362d2d in CHECK_FREE_CLEAR_SIZE () from /usr/lib/libsai.so.1
#6 0x00007f4d7d3831a9 in _brcm_sai_indexed_data_free2 ()
from /usr/lib/libsai.so.1
#7 0x00007f4d7d344d16 in _brcm_sai_free_hostif () from /usr/lib/libsai.so.1
#8 0x00007f4d7d3ad94e in _brcm_sai_dm_fini () from /usr/lib/libsai.so.1
#9 0x00007f4d7d3213a3 in ?? () from /usr/lib/libsai.so.1
#10 0x000056138113e628 in ?? ()
#11 0x00005613810cb168 in ?? ()
#12 0x00005613810da458 in ?? ()
#13 0x00005613810c78ea in ?? ()
#14 0x00005613810c6f8e in ?? ()
#15 0x00007f4d7a84e2e1 in __libc_start_main ()
from /lib/x86_64-linux-gnu/libc.so.6
#16 0x00005613810c750a in ?? ()
(gdb)

The syslogs are attached

syslog-syncd.docx

[xinliu-seattle]

need >show tech file. Seems SAI is crashing. please check with SAI sanity.

[BrcmSAI]

It looks like the crash happens at shutdown path. The attached syslog doesn't contain any message in shutdown path. It is possible that the corresponding log entries has been rolled into syslog.1. or even older. Can you attach them as well?

[mini-nair-dell]

syncd.1588229327.30.core.gz
sonic_dump_sonic-s6100-07_20200430_065821.tar.gz

[mini-nair-dell]

syslog.docx
syslog1.docx

[mini-nair-dell]

Have uploaded the core file/show tech and syslog from the master - 259 for the issue

Thanks
Mini

[xinliu-seattle]

Close this as dup of #4429, which is on a different platform.