Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[logrotate] Enhance robustness #2942

Merged
merged 2 commits into from
May 26, 2019
Merged

[logrotate] Enhance robustness #2942

merged 2 commits into from
May 26, 2019

Conversation

jleveque
Copy link
Contributor

@jleveque jleveque commented May 24, 2019

- What I did

  • Enhance the robustness of logrotate

- How I did it

  • Prevent multiple hung logrotate processes from accumulating and consuming CPU/memory
    • Decrease frequency of cron job which runs logrotate from every 1 minute to every 10 minutes
    • In cron job, before running logrotate, kill any lingering logrotate processes, as they are most likely hung
  • Add step to prerotate script to delete all *.1.gz files which, if they are present, means they were left over from a prior, incomplete run of logrotate. If any of these files exist, logrotate will fail to do its job for that log.
  • Rename prerotate script to firstaction, as prerotate is run after logrotate renames existing scripts, which is too late to delete all *.1.gz files, as the renaming will fail and prerotate will never be run. firstaction runs before everything, so this is the appropriate script for all of this cleanup.
  • Remove note about working around a bug in init-system-helpers < 1.47. Debian Stretch now has init-system-helpers version 1.48 installed, and while calling service rsyslog rotate no longer reports an error, it doesn't seem to cause rsyslogd to properly close the old file. Therefore, we will continue sending SIGHUP directly to the rsyslogd process and no longer consider it a workaround.

- How to verify it

1.) Simulate a left-behind *.1.gz file: touch /var/log/syslog.1.gz
2.) Fill syslog past the 1MB size threshold specified in the block
3.) Wait for 10 minutes (you can also tail /var/log/cron.log to watch for the next execution of logrotate), then check to ensure /var/log/syslog.1.gz is no longer present and that syslog has been rotated.
4.) Repeat with the other log files specified in this section

… init-system-helpers < 1.47 workaround

However, continue to send SIGHUP directly to rsyslogd process
because 'service rsyslog rotate' still doesn't work properly with
init-system-helpers version 1.48
@jleveque
Copy link
Contributor Author

Retest this please.

@lguohan lguohan merged commit 3ec3e20 into sonic-net:master May 26, 2019
@jleveque jleveque deleted the logrotate_robustness branch May 26, 2019 01:02
MichelMoriniaux pushed a commit to criteo-forks/sonic-buildimage that referenced this pull request May 28, 2019
* [logrotate] Decrease frequency to every 10 minutes; kill any lingering logrotate processes

* [logrotate] Delete all *.1.gz files as firstaction; Remove note about init-system-helpers < 1.47 workaround

However, continue to send SIGHUP directly to rsyslogd process
because 'service rsyslog rotate' still doesn't work properly with
init-system-helpers version 1.48
yxieca pushed a commit that referenced this pull request May 29, 2019
* [logrotate] Decrease frequency to every 10 minutes; kill any lingering logrotate processes

* [logrotate] Delete all *.1.gz files as firstaction; Remove note about init-system-helpers < 1.47 workaround

However, continue to send SIGHUP directly to rsyslogd process
because 'service rsyslog rotate' still doesn't work properly with
init-system-helpers version 1.48
mssonicbld added a commit that referenced this pull request Nov 14, 2023
…lly (#17142)

#### Why I did it
src/sonic-swss
```
* 644b227a - (HEAD -> master, origin/master, origin/HEAD) [portsorch]: Implement port PFC asym capability check (#2942) (3 days ago) [Nazarii Hnydyn]
```
#### How I did it
#### How to verify it
#### Description for the changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants