Fixes problems with database - swss - syncd synchronization. #110

vitaliy-senchyshyn · 2017-02-03T15:57:29Z

This PR fixes problem with database - swss - syncd synchronization.

There are two problems:

The database is flushed in swss.service ExecStartPre without check that redis server is started already. Sometimes the flush is executed too early which makes swss failing. Here is the error log of this case:

Feb 2 12:56:23 switch2 INFO docker[798]: Could not connect to Redis at 127.0.0.1:6379: Connection refused
Feb 2 12:56:23 switch2 INFO docker[798]: Could not connect to Redis at 127.0.0.1:6379: Connection refused
Feb 2 12:56:23 switch2 NOTICE systemd[1]: swss.service: control process exited, code=exited status=1
Feb 2 12:56:23 switch2 ERR systemd[1]: Failed to start switch state service container.
Feb 2 12:56:23 switch2 NOTICE systemd[1]: Unit swss.service entered failed state.

In order to solve this to swss.service is added a bash loop which checks that redis server is up using redis-cli ping command. If it's not the loop sleeps for a second before the next try.

The second problem is related to a race condition between database flush performed in swss.service and HIDDEN variable set by syncd. Both the services are started simultaneously and sometimes the flush in swss is performed later than the set in syncd. When after this orchagent starting script checks the HLEN of HIDDEN variable it's always zero so the orchagent just doesn't start.

As a solution syncd.service is made dependant on swss.service and should be executed after the last one is started.

…race condition with database flush/set

…ng the database

lguohan · 2017-02-03T16:40:50Z

ansible/roles/sonicv2/templates/etc/systemd/system/swss.j2

@@ -5,6 +5,8 @@ After=database.service

 [Service]
 User={{ sonicadmin_user }}
+# Wait for redis server start before database clean by checking the server listening port 6379
+ExecStartPre=/bin/bash -c "while true; do if [ -n \"$(netstat -l | grep 6379)\" ]; then break; fi; sleep 1; done"


using "nc -z -w 5 127.0.0.1 6379" to check if the port is open? there could a port 36379 that match your criteria.

@lguohan netsat output looks as follows on the box:

admin@switch2:~$ netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:6379 : LISTEN

So I guess it's better to check for ":6379". The it will cover all the cases.

Redis CLI has ping command

okay, let's check with PING

lguohan · 2017-02-03T17:17:57Z

on my box there are lots of more ports.

acsadmin@CCPSCH01030BBLF:~$ Active Internet connections Proto Recv-Q Send-Q Local Address tcp 0 0 :ssh tcp 0 0 localhost:4000 tcp 0 0 localhost:zebra tcp 0 0 localhost:6379 tcp 0 0 localhost:bgpd tcp 0 0 :bgp tcp6 0 0 [::]:ssh tcp6 0 0 [::]:bgp udp 0 0 *:33222 udp 0 0 localhost:syslog udp 0 0 *:57900 udp 0 0 *:37743 udp 0 0 *:50129 udp 0 0 *:42391 udp 0 0 *:58961 udp 0 0 *:22173 udp 0 0 *:42721 udp 0 0 :47221 udp 0 0 :43373 udp 0 0 :59820 udp 0 0 :bootpc udp 0 udp 0 0 localhost:ntp udp 0 udp 0 udp 0 0 localhost:snmp udp6 0 0 [::]:53888 udp6 0 0 [::]:38078 udp6 0 0 localhost:ntp raw6 0 0 [::]:ipv6-icmp Active UNIX domain Proto RefCnt Flags unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] unix 2 [ ACC ] sudo netstat -l
(only servers)
Foreign Address State
: LISTEN
: LISTEN
: LISTEN
: LISTEN
: LISTEN
: LISTEN
[::]: LISTEN
[::]: LISTEN
:
:
:
:
:
:
:
:
:
:
:
:
:
0 ccpsch01030bblf.phx:ntp :
:
0 ccpsch01030bblf-lo:snmp :
0 ccpsch01030bblf.ph:snmp :
:
[::]:
[::]:
[::]:
[::]: 7
sockets (only servers)
Type State I-Node Path
STREAM LISTENING 1290 /run/systemd/private
STREAM LISTENING 11289 /var/run/docker.sock
STREAM LISTENING 9042 /var/lib/docker/network/files/738130dd349f83861d47520786267cc26128056961ac2ae54121c475a5f99610.sock
SEQPACKET LISTENING 1315 /run/udev/control
STREAM LISTENING 1318 /run/systemd/journal/stdout
STREAM LISTENING 182052 /var/run/supervisor.sock.1
STREAM LISTENING 181439 /var/run/quagga/zserv.api
STREAM LISTENING 181441 /var/run/quagga/zebra.vty
STREAM LISTENING 180016 /var/run/lldpd.socket
STREAM LISTENING 180768 /var/run/quagga/bgpd.vty
STREAM LISTENING 149167 /var/run/redis/redis.sock
STREAM LISTENING 27286466 /var/run/sswsyncd/sswsyncd.socket
STREAM LISTENING 181808 /var/run/supervisor.sock.1
STREAM LISTENING 10462 /var/run/docker/libcontainerd/docker-containerd.sock
STREAM LISTENING 181191 /var/agentx/master
STREAM LISTENING 36911 /var/opt/quest/vas/vasd/.vasd_62
STREAM LISTENING 35980 /var/opt/quest/vas/vasd/.vasd40_ipc_sock
STREAM LISTENING 34857 /var/opt/quest/vas/vasd/.vasd_63
STREAM LISTENING 34563 /var/opt/quest/vas/vasd/.vasd_65
STREAM LISTENING 35982 /var/opt/quest/vas/vasd/.vasd_64

vitaliy-senchyshyn · 2017-02-03T17:32:37Z

On my too. I've just posted a part of the log.

stcheng · 2017-02-03T22:58:13Z

ansible/roles/sonicv2/templates/etc/systemd/system/syncd.j2

-Requires=database.service
-After=database.service
+Requires=database.service swss.service
+After=database.service swss.service


syncd is not depending on swss
and syncd should start before swss
@vitaliy-senchyshyn @lguohan

swss depends on syncd and swss starts after syncd

swss.service is clearing the database

ok I will give a test on this

picked this change from sonic-mgmt repo. sonic-net/sonic-mgmt#110

vitaliy-senchyshyn added 2 commits February 3, 2017 17:30

[syncd.service] Make syncd starting after swss in order to eliminate …

8063317

…race condition with database flush/set

[swss.service] Make swss waiting for redis server start before cleani…

98740b1

…ng the database

lguohan reviewed Feb 3, 2017

View reviewed changes

Fixed review comments for changes in swss service file

2bf45ea

lguohan approved these changes Feb 3, 2017

View reviewed changes

lguohan mentioned this pull request Feb 3, 2017

Revert flushdb logic to syncd, to make brcm work #112

Closed

lguohan merged commit a2ab261 into sonic-net:master Feb 3, 2017

stcheng reviewed Feb 3, 2017

View reviewed changes

NStetskovych-zz mentioned this pull request Feb 8, 2017

[sonicdocker] Change container task order due to service dependencies #118

Closed

lguohan added a commit to lguohan/sonic-buildimage that referenced this pull request Feb 9, 2017

[swss]: Wait for redis server start before database clean

3b1d56e

picked this change from sonic-mgmt repo. sonic-net/sonic-mgmt#110

lguohan mentioned this pull request Feb 9, 2017

[swss]: Wait for redis server start before database clean sonic-net/sonic-buildimage#281

Merged

lguohan added a commit to sonic-net/sonic-buildimage that referenced this pull request Feb 9, 2017

[swss]: Wait for redis server start before database clean (#281)

e5616f2

picked this change from sonic-mgmt repo. sonic-net/sonic-mgmt#110

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes problems with database - swss - syncd synchronization. #110

Fixes problems with database - swss - syncd synchronization. #110

vitaliy-senchyshyn commented Feb 3, 2017 •

edited

Loading

lguohan Feb 3, 2017

vitaliy-senchyshyn Feb 3, 2017

marian-pritsak Feb 3, 2017

vitaliy-senchyshyn Feb 3, 2017

lguohan commented Feb 3, 2017

vitaliy-senchyshyn commented Feb 3, 2017

stcheng Feb 3, 2017

stcheng Feb 3, 2017

svc-acs Feb 3, 2017

stcheng Feb 3, 2017

Fixes problems with database - swss - syncd synchronization. #110

Fixes problems with database - swss - syncd synchronization. #110

Conversation

vitaliy-senchyshyn commented Feb 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Feb 3, 2017

vitaliy-senchyshyn commented Feb 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliy-senchyshyn commented Feb 3, 2017 •

edited

Loading