Make swssconfig status FATAL when it fails #1009

qiluo-msft · 2017-10-03T23:11:02Z

Make supervisor controled one-shot program autorestart 0 time, so the status will become FATAL instead of EXITED if fails. 'supervisorctl status' differentiates between normal exit and failure.

It will help diagnose in running switch.

Signed-off-by: Qi Luo qiluo-msft@users.noreply.github.com

- What I did
The supervisor controlled one-shot program status will become FATAL instead of EXITED if failure happens
- How I did it
Make supervisor controled one-shot program autorestart 0 time
- How to verify it
Test in lab switch
- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

…e status will become FATAL instead of EXITED if failure happens Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

stcheng · 2017-10-03T23:13:09Z

dockers/docker-orchagent/swssconfig.sh

@@ -46,7 +46,7 @@ elif [ "$HWSKU" == "Force10-S6000-Q32" ]; then
 elif [ "$HWSKU" == "Arista-7050-QX32" ]; then
    SWSSCONFIG_ARGS+="td2.32ports.buffers.json td2.32ports.qos.json "
 elif [[ "$HWSKU" == "ACS-MSN27"* ]]; then
-    sonic-cfggen -m /etc/sonic/minigraph.xml -t /usr/share/sonic/templates/msn27xx.32ports.buffers.json.j2 > /etc/swss/config.d/msn27xx.32ports.buffers.json
+    sonic-cfggen -m /etc/sonic/minigraph.xml -t /usr/share/sonic/templates/msn27xx.32ports.buffers.json.j2 > /etc/swss/config.d/msn27xx.32ports.buffers.json || exit 1


what about set -e for this script?

The swssconfig.sh could be improved in future. Currently there is no assumption that every command will success there, except the obvious line such as sonic-cfggen.

I didn't get why there's no assumption that every command will success? right now there're sonic-cfggen and swssconfig commands. I think both of them need to be successful after execution.

jleveque · 2017-10-03T23:28:40Z

What is the ultimate behavior change you're trying to achieve with this?

Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

stcheng

please also check with other reviewers

lguohan · 2017-10-03T23:44:16Z

does this change the syslog message?

qiluo-msft · 2017-10-03T23:47:21Z

It only change syslog this way

Oct  4 06:25:42.643077 sonic INFO supervisord: start.sh neighsyncd: started
Oct  4 06:25:42.801254 sonic INFO swss.sh[26769]: 2017-10-04 06:25:42,800 INFO spawned: 'swssconfig' with pid 53
Oct  4 06:25:43.402113 sonic INFO supervisord: swssconfig Traceback (most recent call last):
Oct  4 06:25:43.402279 sonic INFO supervisord: swssconfig   File "/usr/local/bin/sonic-cfggen", line 215, in <module>
Oct  4 06:25:43.402502 sonic INFO supervisord: swssconfig     main()
Oct  4 06:25:43.402593 sonic INFO supervisord: swssconfig   File "/usr/local/bin/sonic-cfggen", line 196, in main
Oct  4 06:25:43.402676 sonic INFO supervisord: swssconfig     print template.render(data)
Oct  4 06:25:43.402755 sonic INFO supervisord: swssconfig   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
Oct  4 06:25:43.403235 sonic INFO supervisord: swssconfig     return self.environment.handle_exception(exc_info, True)
Oct  4 06:25:43.403330 sonic INFO supervisord: swssconfig   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
Oct  4 06:25:43.403695 sonic INFO supervisord: swssconfig     reraise(exc_type, exc_value, tb)
Oct  4 06:25:43.403790 sonic INFO supervisord: swssconfig   File "/usr/share/sonic/templates/msn27xx.32ports.buffers.json.j2", line 209, in top-level template code
Oct  4 06:25:43.404032 sonic INFO supervisord: swssconfig     {%- set port_config = speed + '_' + cable -%}
Oct  4 06:25:43.404121 sonic INFO supervisord: swssconfig jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'speed'
Oct  4 06:25:43.423018 sonic INFO swss.sh[26769]: 2017-10-04 06:25:43,422 INFO exited: swssconfig (exit status 1; not expected)
Oct  4 06:25:44.424811 sonic INFO swss.sh[26769]: 2017-10-04 06:25:44,424 INFO gave up: swssconfig entered FATAL state, too many start retries too quickly
Oct  4 06:25:44.428831 sonic INFO supervisord: start.sh swssconfig: ERROR (spawn error)

Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

* msft_github/master: [DHCP Relay]: Support Multiple VLANs (Separate DHCP Relay Agents, One Per VLAN) (sonic-net#999) [build]: sonic-utilities package depends on swsssdk; build as wheel and add build dependency (sonic-net#1011) Make swssconfig status FATAL when it fails (sonic-net#1009) [swss]: Update swss-common/sairedis/swss submodules (sonic-net#1008) [config-engine]: Fix bug multiple ports connecting to same neighbor (sonic-net#1005)

Fix a typo in mellanox_buffer_migrator (sonic-net#1090) [CLI][PFCWD][Multi-ASIC] Added multi ASIC support to 'pfcwd' CLI command (sonic-net#1080) Add namespace of the process in the coredump filename. (sonic-net#1091) [setup.py] Add aliases.ini to sonic_installer package (sonic-net#1088) [pfcwd] Add single asic unit tests for show commands (sonic-net#1085) Enhance SONiC with kubernetes management commands (sonic-net#962) [counterpoll] add port buffer drop group (sonic-net#1009) [CLI][PFC] Add multi ASIC options for pfcstat and 'show pfc counters' (sonic-net#1057) Signed-off-by: Stephen Sun <stephens@nvidia.com>

5c173f7 [counterpoll] add port buffer drop group (sonic-net#1009) 62e44d9 [CLI][PFC] Add multi ASIC options for pfcstat and 'show pfc counters' (sonic-net#1057) Signed-off-by: Volodymyr Samotiy <volodymyrs@nvidia.com>

Signed-off-by: Mykola Faryma <mykolaf@mellanox.com> Co-authored-by: Danny Allen <daall@microsoft.com> Co-authored-by: Volodymyr Samotiy <volodymyrs@mellanox.com>

Make supervisor controlled one-shot program autorestart 0 time, so th…

733d723

…e status will become FATAL instead of EXITED if failure happens Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

qiluo-msft requested review from stcheng, jleveque and lguohan October 3, 2017 23:11

stcheng reviewed Oct 3, 2017

View reviewed changes

Make swssconfig.sh strictly exit on any failure

ffac483

Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

qiluo-msft changed the title ~~Make supervisor controled one-shot program autorestart 0 time, so the status will become FATAL instead of EXITED if failure happens~~ Make swssconfig status FATAL when it fails Oct 3, 2017

stcheng approved these changes Oct 3, 2017

View reviewed changes

Tune startretries, tested in supervisor 3.3.2-1

57ccac2

Signed-off-by: Qi Luo <qiluo-msft@users.noreply.github.com>

jleveque approved these changes Oct 4, 2017

View reviewed changes

lguohan approved these changes Oct 4, 2017

View reviewed changes

lguohan merged commit 554114c into sonic-net:master Oct 4, 2017

qiluo-msft deleted the qiluo/supervisorfatal branch October 4, 2017 17:31

volodymyrsamotiy mentioned this pull request Sep 8, 2020

[sonic-utilities] Update submodule pointer #5336

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make swssconfig status FATAL when it fails #1009

Make swssconfig status FATAL when it fails #1009

qiluo-msft commented Oct 3, 2017 •

edited

Loading

stcheng Oct 3, 2017

qiluo-msft Oct 3, 2017

stcheng Oct 3, 2017

jleveque commented Oct 3, 2017

stcheng left a comment

lguohan commented Oct 3, 2017

qiluo-msft commented Oct 3, 2017 •

edited

Loading

Make swssconfig status FATAL when it fails #1009

Make swssconfig status FATAL when it fails #1009

Conversation

qiluo-msft commented Oct 3, 2017 • edited Loading

stcheng Oct 3, 2017

Choose a reason for hiding this comment

qiluo-msft Oct 3, 2017

Choose a reason for hiding this comment

stcheng Oct 3, 2017

Choose a reason for hiding this comment

jleveque commented Oct 3, 2017

stcheng left a comment

Choose a reason for hiding this comment

lguohan commented Oct 3, 2017

qiluo-msft commented Oct 3, 2017 • edited Loading

qiluo-msft commented Oct 3, 2017 •

edited

Loading

qiluo-msft commented Oct 3, 2017 •

edited

Loading