[Multi Asic] support of swss.rec and sairedis.rec for multi asic #6310

arlakshm · 2020-12-28T23:10:18Z

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan arlakshm@microsoft.com

- Why I did it
This PR has the changes to support having different swss.rec and sairedis.rec for each asic.
The logrotate script is updated as well

- How I did it

Update the orchagent.sh script to use the logfile name options in these PRs([logfile]: Add option to specify swss rec file name sonic-swss#1546 and [logfile]: Add handling of Sairedis rec filename sonic-sairedis#747)
In multi asic platforms the record files will be different for each asic, with the format swss.asic{x}.rec and sairedis.asic{x}.rec
Update the logrotate script for multiasic platform .

- How to verify it
The following tests are done to verify
Verification logs multi asic

Check the record files are created

 admin@sonic:/var/log/swss$ ls -l
total 29532
-rw-r--r-- 1 root root 4497346 Dec 29 23:12 sairedis.asic0.rec
-rw-r--r-- 1 root root 3236040 Dec 29 23:12 sairedis.asic1.rec
-rw-r--r-- 1 root root 1938290 Dec 29 23:12 sairedis.asic2.rec
-rw-r--r-- 1 root root 2009720 Dec 29 23:12 sairedis.asic3.rec
-rw-r--r-- 1 root root 3089764 Dec 29 23:12 sairedis.asic4.rec
-rw-r--r-- 1 root root 3090256 Dec 29 23:12 sairedis.asic5.rec
-rw-r--r-- 1 root root 2746843 Dec 29 23:12 swss.asic0.rec
-rw-r--r-- 1 root root 1813896 Dec 29 23:12 swss.asic1.rec
-rw-r--r-- 1 root root 2110988 Dec 29 23:12 swss.asic2.rec
-rw-r--r-- 1 root root 2148698 Dec 29 23:12 swss.asic3.rec
-rw-r--r-- 1 root root 1766758 Dec 29 23:12 swss.asic4.rec
-rw-r--r-- 1 root root 1766644 Dec 29 23:12 swss.asic5.rec

logrotate manually
admin@sonic:/var/log/swss$ sudo /usr/sbin/logrotate /etc/logrotate.conf /dev/null 2>&1
Verify the files are rotated

admin@sonic:/var/log/swss$ ls -l
total 29544
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic0.rec
-rw-r--r-- 1 root root 4497346 Dec 29 23:12 sairedis.asic0.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic1.rec
-rw-r--r-- 1 root root 3236040 Dec 29 23:12 sairedis.asic1.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic2.rec
-rw-r--r-- 1 root root 1938290 Dec 29 23:12 sairedis.asic2.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic3.rec
-rw-r--r-- 1 root root 2009720 Dec 29 23:12 sairedis.asic3.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic4.rec
-rw-r--r-- 1 root root 3089764 Dec 29 23:12 sairedis.asic4.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 sairedis.asic5.rec
-rw-r--r-- 1 root root 3090256 Dec 29 23:12 sairedis.asic5.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic0.rec
-rw-r--r-- 1 root root 2746843 Dec 29 23:12 swss.asic0.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic1.rec
-rw-r--r-- 1 root root 1813896 Dec 29 23:12 swss.asic1.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic2.rec
-rw-r--r-- 1 root root 2110988 Dec 29 23:12 swss.asic2.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic3.rec
-rw-r--r-- 1 root root 2148698 Dec 29 23:12 swss.asic3.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic4.rec
-rw-r--r-- 1 root root 1766758 Dec 29 23:12 swss.asic4.rec.1
-rw-r--r-- 1 root root       0 Dec 29 23:13 swss.asic5.rec
-rw-r--r-- 1 root root 1766644 Dec 29 23:12 swss.asic5.rec.1

Check the syslogs

admin@sonic:/var/log/swss$ sudo grep -i "Sending SIGHUP to OA" /var/log/syslog
Dec 29 23:13:14.409308 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic0.rec
Dec 29 23:13:14.699731 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic1.rec
Dec 29 23:13:14.988207 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic2.rec
Dec 29 23:13:15.277998 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic3.rec
Dec 29 23:13:15.572765 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic4.rec
Dec 29 23:13:15.861483 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:sairedis.asic5.rec
Dec 29 23:13:16.149143 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic0.rec
Dec 29 23:13:16.438341 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic1.rec
Dec 29 23:13:16.729833 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic2.rec
Dec 29 23:13:17.019212 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic3.rec
Dec 29 23:13:17.317649 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic4.rec
Dec 29 23:13:17.616700 sonic INFO logrotate: Sending SIGHUP to OA log_file_name:swss.asic5.rec

- Which release branch to backport (provide reason below if selected)

201811
201911
202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

arlakshm · 2020-12-29T04:33:21Z

retest vsimage please

lguohan · 2020-12-29T19:46:02Z

files/image_config/logrotate/logrotate.d/rsyslog

-/var/log/swss/sairedis.rec
-/var/log/swss/swss.rec
+/var/log/swss/sairedis.*.rec
+/var/log/swss/swss.*.rec


will this work for single asic?

fixed in the latest commit

lguohan · 2020-12-29T19:48:50Z

files/image_config/logrotate/logrotate.d/rsyslog

-            pgrep -x orchagent | xargs /bin/kill -HUP 2>/dev/null || true
+            # for multi asic platforms, there are multiple orchagents
+            # send the SIGHUP only to the orchagent the which needs log file rotation
+            if [ $(pgrep -x orchagent | wc -l) -gt 0 ]; then


if somehow only one orchagent starts in the multi-asic platform, this logic could do wrong classification. why not use the num_asic variable?

If there is only on orchagent, the else block will take care of sending SIGHUP to the orchagent.

I tried using the using the num_asic by sourcing the asic.conf file source doesn't seem to work in this logrotate script.

updated to use the asic.conf in the latest commit

lguohan · 2020-12-29T19:52:43Z

files/image_config/logrotate/logrotate.d/rsyslog

+            if [ $(pgrep -x orchagent | wc -l) -gt 0 ]; then
+                log_file=$1
+                log_file_name=${log_file#/var/log/swss/}
+                pgrep -xa orchagent | grep $log_file_name | awk '{ print $1; }' | xargs /bin/kill -HUP 2>/dev/null || true


what does the pgrep -xa orchagent looks like on a multi-asic platform? can you provide output?

admin@sonic:/var/log/swss$ pgrep -xa orchagent 11710 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 03:00.0 -f swss.asic0.rec -j sairedis.asic0.rec -m 00:be:75:3a:ef:50 12159 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 04:00.0 -f swss.asic1.rec -j sairedis.asic1.rec -m 00:be:75:3a:ef:50 12168 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 08:00.0 -f swss.asic4.rec -j sairedis.asic4.rec -m 02:42:f0:7f:01:06 12169 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 09:00.0 -f swss.asic5.rec -j sairedis.asic5.rec -m 02:42:f0:7f:01:02 12205 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 05:00.0 -f swss.asic2.rec -j sairedis.asic2.rec -m 00:be:75:3a:ef:50 12236 /usr/bin/orchagent -d /var/log/swss -b 8192 -i 06:00.0 -f swss.asic3.rec -j sairedis.asic3.rec -m 00:be:75:3a:ef:50

arlakshm · 2020-12-29T23:27:41Z

Verification logs Single Asic

check record files are created

admin@vlab-01:/var/log/swss$ ls -l
total 8016
-rw-r--r-- 1 root root 4144152 Dec 29 22:59 sairedis.rec
-rw-r--r-- 1 root root 4061342 Dec 29 22:59 swss.rec

logrotate manually
admin@vlab-01:/var/log/swss$ /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1
check logfiles are rotated

admin@vlab-01:/var/log/swss$ ls -l
total 8020
-rw-r--r-- 1 root root       0 Dec 29 23:00 sairedis.rec
-rw-r--r-- 1 root root 4144152 Dec 29 22:59 sairedis.rec.1
-rw-r--r-- 1 root root    1328 Dec 29 23:00 swss.rec
-rw-r--r-- 1 root root 4061424 Dec 29 23:00 swss.rec.1

check syslog

  admin@vlab-01:/var/log/swss$ sudo grep -i "Sending SIGHUP to OA" /var/log/syslog
Dec 29 23:00:01.429946 vlab-01 INFO logrotate: Sending SIGHUP to OA
Dec 29 23:00:01.479333 vlab-01 INFO logrotate: Sending SIGHUP to OA
admin@vlab-01:/var/log/swss$

abdosi · 2020-12-30T19:00:07Z

files/image_config/logrotate/logrotate.d/rsyslog

+                logger -p syslog.info -t "logrotate" "Sending SIGHUP to OA log_file_name:$log_file_name"
+                pgrep -xa orchagent | grep $log_file_name | awk '{ print $1; }' | xargs /bin/kill -HUP 2>/dev/null || true
+            else
+                logger -p syslog.info -t "logrotate" "Sending SIGHUP to OA"


one minor comment i think for consistency if we can keep same syslog message format for both single and multi-asic.
Display file name for both.

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

arlakshm · 2021-01-20T22:40:29Z

retest

arlakshm · 2021-01-21T18:56:31Z

retest this please

arlakshm · 2021-01-22T02:10:15Z

/AzurePipelines run

azure-pipelines · 2021-01-22T02:10:25Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan arlakshm@microsoft.com - Why I did it This PR has the changes to support having different swss.rec and sairedis.rec for each asic. The logrotate script is updated as well - How I did it Update the orchagent.sh script to use the logfile name options in these PRs(sonic-net/sonic-swss#1546 and sonic-net/sonic-sairedis#747) In multi asic platforms the record files will be different for each asic, with the format swss.asic{x}.rec and sairedis.asic{x}.rec Update the logrotate script for multiasic platform .

arlakshm added 2 commits December 28, 2020 22:56

support of swss.rec for multi asic

a4ad85b

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

fix typo

356be38

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

arlakshm requested a review from lguohan December 29, 2020 04:32

arlakshm marked this pull request as ready for review December 29, 2020 04:32

arlakshm requested review from abdosi and jleveque December 29, 2020 04:32

lguohan reviewed Dec 29, 2020

View reviewed changes

address review comments

7ae95e4

arlakshm requested a review from lguohan December 29, 2020 23:32

abdosi reviewed Dec 30, 2020

View reviewed changes

Address review comment

d93c5f2

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

lguohan approved these changes Jan 6, 2021

View reviewed changes

arlakshm merged commit 0e12ca8 into sonic-net:master Jan 22, 2021

arlakshm added Multi-ASIC Request for 201911 Branch Request for 202012 Branch labels Jan 22, 2021

abdosi added the Included in 201911 Branch label Jan 28, 2021

lguohan mentioned this pull request Feb 6, 2021

Fix bind-mount of /var/log/swss directory in multi-asic setup #5990

Closed

yxieca added the Included in 202012 Branch label Feb 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Multi Asic] support of swss.rec and sairedis.rec for multi asic #6310

[Multi Asic] support of swss.rec and sairedis.rec for multi asic #6310

arlakshm commented Dec 28, 2020 •

edited

Loading

arlakshm commented Dec 29, 2020

lguohan Dec 29, 2020

arlakshm Dec 29, 2020

lguohan Dec 29, 2020

arlakshm Dec 29, 2020

arlakshm Dec 29, 2020

lguohan Dec 29, 2020

arlakshm Dec 29, 2020

arlakshm commented Dec 29, 2020

abdosi Dec 30, 2020

arlakshm commented Jan 20, 2021

arlakshm commented Jan 21, 2021

arlakshm commented Jan 22, 2021

azure-pipelines bot commented Jan 22, 2021

[Multi Asic] support of swss.rec and sairedis.rec for multi asic #6310

[Multi Asic] support of swss.rec and sairedis.rec for multi asic #6310

Conversation

arlakshm commented Dec 28, 2020 • edited Loading

arlakshm commented Dec 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arlakshm commented Dec 29, 2020

Choose a reason for hiding this comment

arlakshm commented Jan 20, 2021

arlakshm commented Jan 21, 2021

arlakshm commented Jan 22, 2021

azure-pipelines bot commented Jan 22, 2021

arlakshm commented Dec 28, 2020 •

edited

Loading