S6100: Orchagent crash seen on loading the latest master image - master.362-ee4197e9

[mini-nair-dell]

On loading the recent master image, orchagent crashes. Pls find the logs below.

root@sonic-s6100-10:/var/log# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3689568e97d2 docker-snmp:latest "/usr/bin/supervisord" 46 seconds ago Up 44 seconds snmp
88b0117917de docker-sonic-telemetry:latest "/usr/bin/supervisord" 4 minutes ago Up 4 minutes telemetry
6df223e07d9d docker-router-advertiser:latest "/usr/bin/docker-ini…" 4 minutes ago Up 4 minutes radv
cd138339c12f docker-dhcp-relay:latest "/usr/bin/docker_ini…" 4 minutes ago Up 4 minutes dhcp_relay
b76cec6abb6e docker-sonic-mgmt-framework:latest "/usr/bin/supervisord" 4 minutes ago Up 4 minutes mgmt-framework
741cffad8bbd docker-nat:latest "/usr/bin/supervisord" 4 minutes ago Up 4 minutes nat
b010e8d0af9d docker-lldp:latest "/usr/bin/docker-lld…" 4 minutes ago Up 4 minutes lldp
956f9359b2c9 docker-syncd-brcm:latest "/usr/bin/supervisord" 4 minutes ago Up 4 minutes syncd
1198f2e9549d docker-teamd:latest "/usr/bin/supervisord" 4 minutes ago Up 4 minutes teamd
a710a6443af4 docker-orchagent:latest "/usr/bin/docker-ini…" 4 minutes ago Up 4 minutes swss
47e043a926ad docker-fpm-frr:latest "/usr/bin/docker_ini…" 4 minutes ago Up 4 minutes bgp
8c71c7b10be7 docker-platform-monitor:latest "/usr/bin/docker_ini…" 4 minutes ago Up 4 minutes pmon
3af00073eca2 docker-database:latest "/usr/local/bin/dock…" 4 minutes ago Up 4 minutes database
root@sonic-s6100-10:/var/log# cd ../core
root@sonic-s6100-10:/var/core# ls
orchagent.1596079859.46.core.gz

Core file extract :

root@sonic-s6100-10:~# gdb /usr/bin/orchagent orchagent.1596079859.46.core
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/orchagent...(no debugging symbols found)...done.
[New LWP 61]
[New LWP 60]
[New LWP 46]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/orchagent -d /var/log/swss -b 8192 -m 54:bf:64:af:b8:42'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff18c8957bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7ff187fff700 (LWP 61))]
(gdb)
(gdb) bt
#0 0x00007ff18c8957bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff18c880535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff18cc4a983 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ff18cc508c6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ff18cc50901 in std::terminate() ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ff18cc50b34 in __cxa_throw ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ff18d1b1d7c in ?? ()
from /usr/lib/x86_64-linux-gnu/libswsscommon.so.0
#7 0x00007ff18cd82739 in sai_serialize_object_meta_key[abi:cxx11](_sai_object_meta_key_t const&) () from /usr/lib/x86_64-linux-gnu/libsaimetadata.so.0
#8 0x00007ff18cd98677 in saimeta::SaiObjectCollection::objectExists(_sai_object_meta_key_t const&) const () from /usr/lib/x86_64-linux-gnu/libsaimetadata.so.0
#9 0x00007ff18cd9f5a7 in saimeta::Meta::meta_sai_on_switch_shutdown_request(unsigned long) () from /usr/lib/x86_64-linux-gnu/libsaimetadata.so.0
#10 0x00007ff18d2806e3 in sairedis::NotificationSwitchShutdownRequest::processMetadata(std::shared_ptrsaimeta::Meta) const ()
from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#11 0x00007ff18d26242a in sairedis::RedisRemoteSaiInterface::syncProcessNotification(std::shared_ptrsairedis::Notification) ()
from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#12 0x00007ff18d2490b6 in sairedis::Sai::handle_notification(std::shared_ptr<sai--Type for more, q to quit, c to continue without paging--
redis::Notification>, sairedis::Context*) ()
from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#13 0x00007ff18d252345 in std::_Function_handler<_sai_switch_notifications_t (std::shared_ptrsairedis::Notification, sairedis::Context*), std::_Bind<std::_Mem_fn<_sai_switch_notifications_t (sairedis::Sai::)(std::shared_ptrsairedis::Notification, sairedis::Context)> (sairedis::Sai*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::shared_ptrsairedis::Notification&&, sairedis::Context*&&) ()
from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#14 0x00007ff18d26c897 in sairedis::Context::handle_notification(std::shared_ptrsairedis::Notification) () from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#15 0x00007ff18d26d320 in std::_Function_handler<_sai_switch_notifications_t (std::shared_ptrsairedis::Notification), std::_Bind<std::_Mem_fn<_sai_switch_notifications_t (sairedis::Context::)(std::shared_ptrsairedis::Notification)> (sairedis::Context, std::_Placeholder<1>)> >::_M_invoke(std::_Any_data const&, std::shared_ptrsairedis::Notification&&) ()
from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#16 0x00007ff18d2621ee in sairedis::RedisRemoteSaiInterface::handleNotification(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11:--Type for more, q to quit, c to continue without paging--
:basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) () from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#17 0x00007ff18d27ca61 in sairedis::RedisChannel::notificationThreadFunction()
() from /usr/lib/x86_64-linux-gnu/libsairedis.so.0
#18 0x00007ff18cc79b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ff18d4b2fa3 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007ff18c9574cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) quit

Attachement

Attached the syslogs at the time of crash

[mini-nair-dell]

syslog.docx
syslog1.docx

[lguohan]

you need this commit [sonic-sairedis] Update Submodule (#5052)

which is only available after build 364. the crash is expected on broadcom platform.