Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

Merged
merged 14 commits into from
Apr 21, 2017
Merged

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
27e610b
Rename 'ACSFileFormat' -> 'SONiCFileFormat'
jleveque Apr 19, 2017
364c945
Rename '00-acs.conf' -> '00-sonic.conf'
jleveque Apr 19, 2017
c8e435d
Add logrotate.d and systemd-journald config files to image
jleveque Apr 19, 2017
3817db5
Log all SONiC process messages to /var/log/syslog; prevent duplicate …
jleveque Apr 19, 2017
21b08c2
Do not redirect cron and daemon logs to their own files, let them log…
jleveque Apr 19, 2017
6304f9d
Log all teamd messages to /var/log/teamd.log; Add more SONiC program …
jleveque Apr 19, 2017
7b0dd39
Remove duplicate code by condensing quagga programs into a list; Fix …
jleveque Apr 19, 2017
9bfaf4f
Kernel and LPR messages no longer getting duplicated to their own log…
jleveque Apr 19, 2017
88e58d6
Now calling logrotate every minute via cron job
jleveque Apr 19, 2017
c7e075c
Need full path to logrotate in cron job
jleveque Apr 20, 2017
7c02b86
Add '.log' suffix to wildcards, otherwise logrotate will rotate alrea…
jleveque Apr 20, 2017
6ccd0d6
Add microsecond granularity to syslog messages
jleveque Apr 20, 2017
de4552b
Don't overwrite system crontab, instead, install additional logrotate…
jleveque Apr 21, 2017
ea8e998
Removed incomplete concept of per-process SONiC logs. We can revisit …
jleveque Apr 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions dockers/docker-base/rsyslog.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ $ModLoad imuxsock # provides support for local system logging
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down
9 changes: 9 additions & 0 deletions files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,9 @@ sudo cp -f $IMAGE_CONFIGS/bash/bash.bashrc $FILESYSTEM_ROOT/etc/
sudo dpkg --root=$FILESYSTEM_ROOT -i target/debs/sonic-device-data_*.deb || \
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f

# Copy crontab
sudo cp -f $IMAGE_CONFIGS/cron/crontab $FILESYSTEM_ROOT/etc/

# Copy NTP configuration files and templates
sudo cp $IMAGE_CONFIGS/ntp/ntp-config.service $FILESYSTEM_ROOT/etc/systemd/system/
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable ntp-config.service
Expand All @@ -101,6 +104,12 @@ sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog-config.sh $FILESYSTEM_ROOT/usr/bin/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.conf.j2 $FILESYSTEM_ROOT/usr/share/sonic/templates/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.d/* $FILESYSTEM_ROOT/etc/rsyslog.d/

# Copy logrotate.d configuration files
sudo cp -f $IMAGE_CONFIGS/logrotate.d/rsyslog $FILESYSTEM_ROOT/etc/logrotate.d/

# Copy systemd-journald configuration files
sudo cp -f $IMAGE_CONFIGS/systemd/journald.conf $FILESYSTEM_ROOT/etc/systemd/

# Copy interfaces configuration files and templates
sudo cp $IMAGE_CONFIGS/interfaces/interfaces-config.service $FILESYSTEM_ROOT/etc/systemd/system/
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable interfaces-config.service
Expand Down
25 changes: 25 additions & 0 deletions files/image_config/cron/crontab
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command

* * * * * /usr/sbin/logrotate -f /etc/logrotate.d/rsyslog
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better put this as a file into /etc/cron.d/ ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current directory structure of files/image_config is not set to mirror the structure of /etc on the SONiC image. If we want to do this for crontab, we should rearrange everything into the /etc/... hierarchy to make things clear across the board.

Also, with our Debian distribution, crontab lives directly in /etc, not in /etc/cron.d.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I see in SONiC

$ ls /etc/cron*
/etc/crontab

/etc/cron.d:

/etc/cron.daily:
apt bsdmainutils dpkg logrotate ntp passwd

/etc/cron.hourly:

/etc/cron.monthly:

/etc/cron.weekly:

Can we put a file to /etc/cron.d/ directory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Done.


44 changes: 44 additions & 0 deletions files/image_config/logrotate.d/rsyslog
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/var/log/syslog
/var/log/quagga/*.log
/var/log/sonic/*.log
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/var/log/sonic? Are we using this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not currently. Please see my first comment on 00-sonic.conf up above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it

/var/log/teamd.log
{
rotate 7
# Removed 'daily' interval, as we now call logrotate more frequently via cron
# and we want to check these logs every time
# daily
size 100M
missingok
notifempty
compress
delaycompress
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
size 100M
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}

4 changes: 2 additions & 2 deletions files/image_config/rsyslog/rsyslog.conf.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ $UDPServerRun 514
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add milliseconds into syslog output?

Copy link
Contributor Author

@jleveque jleveque Apr 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, from my research, it appears rsyslog doesn't provide granularity of milliseconds. It only provides "subseconds" which are, in fact, microseconds. Do we want microsecond granularity? If so, I'll add this. The format would appear as follows:

Apr 20 23:30:01.827792 str-s6000-acs-12 INFO CRON[12536]: (root) CMD (/usr/sbin/logrotate -f /etc/logrotate.d/rsyslog)
Apr 20 23:30:41.652233 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:41.651 * 10000 changes in 60 seconds. Saving...
Apr 20 23:30:41.655740 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:41.654 * Background saving started by pid 84
Apr 20 23:30:41.975609 str-s6000-acs-12 INFO database.sh[1728]: 84:C 20 Apr 23:30:41.974 * DB saved on disk
Apr 20 23:30:41.981183 str-s6000-acs-12 INFO database.sh[1728]: 84:C 20 Apr 23:30:41.980 * RDB: 1 MB of memory used by copy-on-write
Apr 20 23:30:42.057539 str-s6000-acs-12 INFO database.sh[1728]: 1:M 20 Apr 23:30:42.056 * Background saving terminated with success
Apr 20 23:30:48.0 str-s6000-acs-12 ALERT sensord: Sensor alarm: Chip ltc4215-i2c-11-42: in1: +11.33 V [ALARM]
Apr 20 23:31:01.856022 str-s6000-acs-12 INFO CRON[12599]: (root) CMD (/usr/sbin/logrotate -f /etc/logrotate.d/rsyslog)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. it will work for me too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down
46 changes: 0 additions & 46 deletions files/image_config/rsyslog/rsyslog.d/00-acs.conf
View file
Open in desktop

This file was deleted.

41 changes: 41 additions & 0 deletions files/image_config/rsyslog/rsyslog.d/00-sonic.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
## Quagga rules

if $programname == ["quagga",
"watchquagga",
"zebra"]
then {
/var/log/quagga/zebra.log
stop
}

if $programname == "bgpd" then {
/var/log/quagga/bgpd.log
stop
}

## Teamd rules

if $programname contains "teamd_" then {
/var/log/teamd.log
stop
}

## SONiC rules

if $programname == ["fpmsyncd",
"intfsyncd",
"neighsyncd",
"orchagent",
"portsyncd",
"saidump",
"saiplayer",
"syncd",
"swssconfig",
"swssloglevel",
"teamsyncd",
"updategraph"]
then {
?SONiCPerProcessLogfile,SONiCFileFormat
Copy link
Contributor Author

@jleveque jleveque Apr 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-process SONiC logging is not currently working. I don't think John finished this. I updated the list of programs, just in case we want it working, but I believe we'd just like everything to log to /var/log/syslog. If so, I will remove this. Please share your thoughts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to pull this for now, as it was incomplete. We can revisit it again in the future.

stop
}

23 changes: 14 additions & 9 deletions files/image_config/rsyslog/rsyslog.d/99-default.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
kern.* -/var/persist/log/kern.log
lpr.* -/var/log/lpr.log
# Do not redirect cron, daemon, kernel or lpr logs to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this?
Does SONiC emits all these types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The *.* on line 5 catches all messages and logs them to /var/log/syslog. Then, all messages that match cron.*, daemon.* etc. are logged again to their respective files. I commented out the individual files so that all messages are only logged once, to /var/log/syslog.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see

# their own files. Let them log to /var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
#kern.* -/var/log/kern.log
#kern.* -/var/persist/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

Expand All @@ -32,10 +34,13 @@ news.notice -/var/log/news/news.notice
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Removed as duplicates:
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
*.=crit;*.=alert;*.=emerg -/var/persist/log/alarms
#
# Emergencies are sent to everybody logged in.
Expand Down
36 changes: 36 additions & 0 deletions files/image_config/systemd/journald.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=50MB
#SystemKeepFree=
#SystemMaxFileSize=
RuntimeMaxUse=50MB
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to comment all entries out of this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how John originally added the file. He only intended to take advantage of SystemMaxUse and RuntimeMaxUse. We can define other variables if we'd like.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with this. Thanks


4 changes: 2 additions & 2 deletions platform/p4/docker-sonic-p4/rsyslog.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ $ModLoad imuxsock # provides support for local system logging
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down