[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

jleveque · 2017-04-20T00:43:20Z

Implemented changes started by John Arnold last year
Fixed a few bugs
Increased frequency of log rotation from daily to once per minute

jleveque · 2017-04-20T00:46:27Z

files/image_config/rsyslog/rsyslog.d/00-sonic.conf

+                    "teamsyncd",
+                    "updategraph"]
+    then {
+    ?SONiCPerProcessLogfile,SONiCFileFormat


Per-process SONiC logging is not currently working. I don't think John finished this. I updated the list of programs, just in case we want it working, but I believe we'd just like everything to log to /var/log/syslog. If so, I will remove this. Please share your thoughts.

I decided to pull this for now, as it was incomplete. We can revisit it again in the future.

…logging to /var/log/messages

… to /var/log/syslog

…names to SONiC rules clause

…teamd log rule

… files

…dy-rotated logs (e.g., bgpd.log.1.1.1.1.1...)

lguohan · 2017-04-20T22:01:30Z

@pavel-shirshov, can you review?

pavel-shirshov · 2017-04-20T00:52:12Z

files/image_config/cron/crontab

+# 
+# m h  dom mon dow   command
+
+* * * * * /usr/sbin/logrotate -f /etc/logrotate.d/rsyslog


Better put this as a file into /etc/cron.d/ ?

The current directory structure of files/image_config is not set to mirror the structure of /etc on the SONiC image. If we want to do this for crontab, we should rearrange everything into the /etc/... hierarchy to make things clear across the board.

Also, with our Debian distribution, crontab lives directly in /etc, not in /etc/cron.d.

What I see in SONiC

$ ls /etc/cron*
/etc/crontab

/etc/cron.d:

/etc/cron.daily:
apt bsdmainutils dpkg logrotate ntp passwd

/etc/cron.hourly:

/etc/cron.monthly:

/etc/cron.weekly:

Can we put a file to /etc/cron.d/ directory?

Gotcha. Done.

pavel-shirshov · 2017-04-20T00:54:01Z

files/image_config/rsyslog/rsyslog.conf.j2

@@ -42,8 +42,8 @@ $UDPServerRun 514
 #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

 # Define a custom template
-$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
-$ActionFileDefaultTemplate ACSFileFormat
+$template SONiCFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"


Can we add milliseconds into syslog output?

Unfortunately, from my research, it appears rsyslog doesn't provide granularity of milliseconds. It only provides "subseconds" which are, in fact, microseconds. Do we want microsecond granularity? If so, I'll add this. The format would appear as follows:

Apr 20 23:30:01.827792 str-s6000-acs-12 INFO CRON[12536]: (root) CMD (/usr/sbin/logrotate -f /etc/logrotate.d/rsyslog) Apr 20 23:30:41.652233 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:41.651 * 10000 changes in 60 seconds. Saving... Apr 20 23:30:41.655740 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:41.654 * Background saving started by pid 84 Apr 20 23:30:41.975609 str-s6000-acs-12 INFO database.sh[1728]: 84:C 20 Apr 23:30:41.974 * DB saved on disk Apr 20 23:30:41.981183 str-s6000-acs-12 INFO database.sh[1728]: 84:C 20 Apr 23:30:41.980 * RDB: 1 MB of memory used by copy-on-write Apr 20 23:30:42.057539 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:42.056 * Background saving terminated with success Apr 20 23:30:48.0 str-s6000-acs-12 ALERT sensord: Sensor alarm: Chip ltc4215-i2c-11-42: in1: +11.33 V [ALARM] Apr 20 23:31:01.856022 str-s6000-acs-12 INFO CRON[12599]: (root) CMD (/usr/sbin/logrotate -f /etc/logrotate.d/rsyslog)

Yes. it will work for me too.

pavel-shirshov · 2017-04-20T23:39:16Z

files/image_config/systemd/journald.conf

+#MaxLevelSyslog=debug
+#MaxLevelKMsg=notice
+#MaxLevelConsole=info
+#MaxLevelWall=emerg


Why do we need to comment all entries out of this file?

This is how John originally added the file. He only intended to take advantage of SystemMaxUse and RuntimeMaxUse. We can define other variables if we'd like.

I'm good with this. Thanks

pavel-shirshov · 2017-04-20T23:40:33Z

files/image_config/rsyslog/rsyslog.d/99-default.conf

-kern.*                          -/var/log/kern.log
-kern.*                          -/var/persist/log/kern.log
-lpr.*                           -/var/log/lpr.log
+# Do not redirect cron, daemon, kernel or lpr logs to


Do we really need this?
Does SONiC emits all these types?

The *.* on line 5 catches all messages and logs them to /var/log/syslog. Then, all messages that match cron.*, daemon.* etc. are logged again to their respective files. I commented out the individual files so that all messages are only logged once, to /var/log/syslog.

pavel-shirshov · 2017-04-20T23:42:40Z

files/image_config/logrotate.d/rsyslog

@@ -0,0 +1,44 @@
+/var/log/syslog
+/var/log/quagga/*.log
+/var/log/sonic/*.log


/var/log/sonic? Are we using this?

Not currently. Please see my first comment on 00-sonic.conf up above.

… crontab file into /etc/cron.d

…again later

sairedis: * [SAI] Update SAI submodule to v1.5.1 (sonic-net#532) * Cleanup Makefile.am from BFN specific code (sonic-net#530) * [vs] Implement indices for debug counters in VS (sonic-net#531) * Enable FastReboot if we have key "FAST_REBOOT|system" in State db (sonic-net#529) * [flex_counter] Add sairedis support for drop counters (sonic-net#520) swss: * [orchagent] Add swss support for drop counters (sonic-net#1075) * [orchagent] warning fixes for 32bit arch compilation (sonic-net#1129) * [utilities] Create utility classes for interacting with flex counters (sonic-net#1093) * [portsorch] add support to set mac-address learning attribute on bridge-port (sonic-net#809) * Fix traceroute issue (sonic-net#1113) utilities: * Add CLI support for configurable drop counters (sonic-net#688) * Revert "SONiC Management Framework Release 1.0 (sonic-net#659)" (sonic-net#741) * SONiC Management Framework Release 1.0 (sonic-net#659) Signed-off-by: Danny Allen <daall@microsoft.com>

sairedis: * [SAI] Update SAI submodule to v1.5.1 (#532) * Cleanup Makefile.am from BFN specific code (#530) * [vs] Implement indices for debug counters in VS (#531) * Enable FastReboot if we have key "FAST_REBOOT|system" in State db (#529) * [flex_counter] Add sairedis support for drop counters (#520) swss: * [orchagent] Add swss support for drop counters (#1075) * [orchagent] warning fixes for 32bit arch compilation (#1129) * [utilities] Create utility classes for interacting with flex counters (#1093) * [portsorch] add support to set mac-address learning attribute on bridge-port (#809) * Fix traceroute issue (#1113) utilities: * Add CLI support for configurable drop counters (#688) * Revert "SONiC Management Framework Release 1.0 (#659)" (#741) * SONiC Management Framework Release 1.0 (#659) Signed-off-by: Danny Allen <daall@microsoft.com>

sairedis: * [SAI] Update SAI submodule to v1.5.1 (sonic-net#532) * Cleanup Makefile.am from BFN specific code (sonic-net#530) * [vs] Implement indices for debug counters in VS (sonic-net#531) * Enable FastReboot if we have key "FAST_REBOOT|system" in State db (sonic-net#529) * [flex_counter] Add sairedis support for drop counters (sonic-net#520) swss: * [orchagent] Add swss support for drop counters (sonic-net#1075) * [orchagent] warning fixes for 32bit arch compilation (sonic-net#1129) * [utilities] Create utility classes for interacting with flex counters (sonic-net#1093) * [portsorch] add support to set mac-address learning attribute on bridge-port (sonic-net#809) * Fix traceroute issue (sonic-net#1113) utilities: * Add CLI support for configurable drop counters (sonic-net#688) * Revert "SONiC Management Framework Release 1.0 (sonic-net#659)" (sonic-net#741) * SONiC Management Framework Release 1.0 (sonic-net#659) Signed-off-by: Danny Allen <daall@microsoft.com>

* Adds functions to syncd_flex_counter for configuring and polling port-level debug counters Signed-off-by: Danny Allen daall@microsoft.com * Remove uneccesary includes * Add stubs for querying sai capabilities * Add support for serializing debug counter attributes * Add support for switch level debug counters * Improve vslib support for debug counters * Remove interface query stubs * Add missing swss log * Clean-up virtual switch implementation * Simplify vs debug counters * Remove unnecessary includes * Fix formatting * Respond to review comments

52b7a47 (HEAD, origin/master, origin/HEAD) [schema]: MACsec statistics support (sonic-net#520) 48d7d8a [ci]: use native arm64 and armhf pool to build (sonic-net#552) Signed-off-by: Ze Gan <ganze718@gmail.com>

52b7a47 (HEAD, origin/master, origin/HEAD) [schema]: MACsec statistics support (#520) 48d7d8a [ci]: use native arm64 and armhf pool to build (#552) Signed-off-by: Ze Gan <ganze718@gmail.com>

…D automatically (#19629) #### Why I did it src/sonic-platform-daemons ``` * 5bbe6d6 - (HEAD -> 202311, origin/202311) [ycabled][active-active] Fix in gRPC channel callback logic by creating swsscommon table within the context (#509) (9 minutes ago) [vdahiya12] * 12aede7 - Initialize application specific fields as 'N/A' in TRANSCEIVER_INFO table (#511) (#520) (4 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

jleveque added 3 commits April 19, 2017 18:01

Rename 'ACSFileFormat' -> 'SONiCFileFormat'

27e610b

Rename '00-acs.conf' -> '00-sonic.conf'

364c945

Add logrotate.d and systemd-journald config files to image

c8e435d

jleveque commented Apr 20, 2017

View reviewed changes

jleveque added the Enhancement ➕ label Apr 20, 2017

jleveque added 8 commits April 20, 2017 00:53

Log all SONiC process messages to /var/log/syslog; prevent duplicate …

3817db5

…logging to /var/log/messages

Do not redirect cron and daemon logs to their own files, let them log…

21b08c2

… to /var/log/syslog

Log all teamd messages to /var/log/teamd.log; Add more SONiC program …

6304f9d

…names to SONiC rules clause

Remove duplicate code by condensing quagga programs into a list; Fix …

7b0dd39

…teamd log rule

Kernel and LPR messages no longer getting duplicated to their own log…

9bfaf4f

… files

Now calling logrotate every minute via cron job

88e58d6

Need full path to logrotate in cron job

c7e075c

Add '.log' suffix to wildcards, otherwise logrotate will rotate alrea…

7c02b86

…dy-rotated logs (e.g., bgpd.log.1.1.1.1.1...)

stcheng assigned jleveque Apr 20, 2017

pavel-shirshov suggested changes Apr 20, 2017

View reviewed changes

Add microsecond granularity to syslog messages

6ccd0d6

pavel-shirshov suggested changes Apr 20, 2017

View reviewed changes

pavel-shirshov approved these changes Apr 21, 2017

View reviewed changes

jleveque added 2 commits April 21, 2017 00:46

Don't overwrite system crontab, instead, install additional logrotate…

de4552b

… crontab file into /etc/cron.d

Removed incomplete concept of per-process SONiC logs. We can revisit …

ea8e998

…again later

lguohan merged commit 6c20291 into sonic-net:master Apr 21, 2017

jleveque deleted the syslog_improvements branch April 21, 2017 18:14

daall mentioned this pull request Nov 19, 2019

[submodule]: Update sairedis/swss/utilities #3786

Merged

madhanmellanox pushed a commit to madhanmellanox/sonic-buildimage that referenced this pull request Mar 23, 2020

SAI 1.3 header changes (sonic-net#520)

19f9108

Pterosaur mentioned this pull request Nov 8, 2021

[submodule]: update for swss-common #9192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

jleveque commented Apr 20, 2017

jleveque Apr 20, 2017 •

edited

Loading

jleveque Apr 21, 2017

lguohan commented Apr 20, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 20, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 21, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 20, 2017 •

edited

Loading

pavel-shirshov Apr 20, 2017

jleveque Apr 20, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 20, 2017

pavel-shirshov Apr 21, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 21, 2017

pavel-shirshov Apr 21, 2017

pavel-shirshov Apr 20, 2017

jleveque Apr 21, 2017

pavel-shirshov Apr 21, 2017

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

Conversation

jleveque commented Apr 20, 2017

jleveque Apr 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Apr 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jleveque Apr 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jleveque Apr 20, 2017 •

edited

Loading

jleveque Apr 20, 2017 •

edited

Loading