[Supervisord] Deduplicate the alerting messages of critical processes from Supervisord. #6849
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
jleveque
suggested changes
Feb 23, 2021
1.Fix the format of alerting message. 2.For each exited process, there are two fields: the time of last alert and number of dead minutes. Use a dict to hold these two fields instead of a list. 3.Use a formula to calculate how many minutes the process was in dead state instead of hard code. Signed-off-by: Yong Zhao <yozhao@microsoft.com>
jleveque
suggested changes
Feb 24, 2021
jleveque
approved these changes
Feb 25, 2021
@lguohan Can you please help me review this change? |
lguohan
approved these changes
Feb 25, 2021
yxieca
pushed a commit
that referenced
this pull request
Mar 4, 2021
… from Supervisord. (#6849) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it In the configuration of rsyslog, duplicate messages will be suppressed and reported in the format of message repeated n times. Due to this behavior, if a critical process in a container exited unexpectedly, the alerting message will be written into syslog once and not be written into syslog anymore until the second critical process exited. This PR aims to differentiate these alerting messages such that they will not be suppressed by rsyslogd and can appear in the syslog periodically. How I did it This PR adds a counter into the alerting message and shows how many minutes a critical process was not running. How to verify it I verified and test this implementation on a physical DUT.
carl-nokia
pushed a commit
to carl-nokia/sonic-buildimage
that referenced
this pull request
Aug 7, 2021
… from Supervisord. (sonic-net#6849) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it In the configuration of rsyslog, duplicate messages will be suppressed and reported in the format of message repeated n times. Due to this behavior, if a critical process in a container exited unexpectedly, the alerting message will be written into syslog once and not be written into syslog anymore until the second critical process exited. This PR aims to differentiate these alerting messages such that they will not be suppressed by rsyslogd and can appear in the syslog periodically. How I did it This PR adds a counter into the alerting message and shows how many minutes a critical process was not running. How to verify it I verified and test this implementation on a physical DUT.
lolyu
pushed a commit
to lolyu/sonic-buildimage
that referenced
this pull request
Sep 13, 2021
… from Supervisord. (sonic-net#6849) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it In the configuration of rsyslog, duplicate messages will be suppressed and reported in the format of message repeated n times. Due to this behavior, if a critical process in a container exited unexpectedly, the alerting message will be written into syslog once and not be written into syslog anymore until the second critical process exited. This PR aims to differentiate these alerting messages such that they will not be suppressed by rsyslogd and can appear in the syslog periodically. How I did it This PR adds a counter into the alerting message and shows how many minutes a critical process was not running. How to verify it I verified and test this implementation on a physical DUT.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yong Zhao yozhao@microsoft.com
Why I did it
In the configuration of
rsyslog, duplicate messages will be suppressed and reported in the format ofmessage repeated n times.Due to this behavior, if a critical process in a container exited unexpectedly, the alerting message will be written into syslog once
and not be written into syslog anymore until the second critical process exited. This PR aims to differentiate these alerting messages such that they will not be suppressed by
rsyslogdand can appear in the syslog periodically.How I did it
This PR adds a counter into the alerting message and shows how many minutes a critical process was not running.
How to verify it
I verified and test this implementation on a physical DUT.
Which release branch to backport (provide reason below if selected)
Description for the changelog
A picture of a cute animal (not mandatory but encouraged)