[logrotate] error in logs "Failed to start Rotate log files"

[ppikh]

Description

Error in logs happen: "Failed to start Rotate log files"

Steps to reproduce the issue:

It reproduces only in case when time is 12:00AM and logrotation started

There are two triggers for logrotate - One is the default debian logrotate timer which runs daily at 12:00 AM. The second one is the logrotate by cron job which SONiC has created and runs every 10 mins.

The SONiC cron job has a logic to kill any logrotate that is running. #2942

# Attempt to rotate system logs once every 10 minutes.
# First kill any logrotate process(es) if they are still running, as they're most likely hung
*/10 * * * * root /usr/bin/pkill -9 logrotate > /dev/null 2>&1; /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1

The cron job in this case was aligned in such a way in runs at 12:00:01 which is 1 sec after the default timer triggers a logrotate. Since the cron finds a logrotate already running it kills it which results in the log

Oct  5 00:00:00.938784 r-tigon-21 INFO systemd[1]: Starting Rotate log files...
Oct  5 00:00:01.247788 r-tigon-21 WARNING systemd[1]: logrotate.service: Main process exited, code=killed, status=9/KILL
Oct  5 00:00:01.248858 r-tigon-21 WARNING systemd[1]: logrotate.service: Failed with result 'signal'.
Oct  5 00:00:01.249684 r-tigon-21 ERR systemd[1]: Failed to start Rotate log files.

From cron log:

Oct  5 00:00:01.226074 r-tigon-21 INFO CRON[541643]: (root) CMD (/usr/bin/pkill -9 logrotate > /dev/null 2>&1; /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1)

This is non functional since the logrotate from cron will eventually accomplish the task.

However given there is already a cron running in SONiC the default daily logrotate can be disabled

Describe the results you received:

Got error in logs

Oct  5 00:00:01.249684 r-tigon-21 ERR systemd[1]: Failed to start Rotate log files.

Describe the results you expected:

No error in logs

Output of show version:

SONiC.202205_rc.17-12870dde8_Internal

Output of show techsupport:

Can not attach sysdump due to max limit of file 25Mb,
If someone want - can ask me and I will send this file in private message.

Additional information you deem important (e.g. issue happens only occasionally):