[supervisord] Monitoring the critical processes with supervisord. #6242

yozhao101 · 2020-12-17T21:30:09Z

- Why I did it
Initially, we used Monit to monitor critical processes in each container. If one of critical processes was not running
or crashed due to some reasons, then Monit will write an alerting message into syslog periodically. If we add a new process
in a container, the corresponding Monti configuration file will also need to update. It is a little hard for maintenance.

Currently we employed event listener of Supervisod to do this monitoring. Since processes in each container are managed by
Supervisord, we can only focus on the logic of monitoring.

- How I did it
We borrowed the event listener of Supervisord to monitor critical processes in containers. The event listener will take
following steps if it was notified one of critical processes exited unexpectedly:

The event listener will first check whether the auto-restart mechanism was enabled for this container or not. If auto-restart mechanism was enabled, event listener will kill the Supervisord process, which should cause the container to exit and subsequently get restarted.
If auto-restart mechanism was not enabled for this contianer, the event listener will enter a loop which will first sleep 1 minute and then check whether the process is running. If yes, the event listener exits. If no, an alerting message will be written into syslog.

- How to verify it
First, we need checked whether the auto-restart mechanism of a container was enabled or not by running the command show feature status. If enabled, one critical process should be selected and killed manually, then we need check whether the container will be restarted or not.

Second, we can disable the auto-restart mechanism if it was enabled at step 1 by running the commnad sudo config feature autorestart <container_name> disabled. Then one critical process should be selected and killed. After that, we will see the alerting message which will appear in the syslog every 1 minute.

- Which release branch to backport (provide reason below if selected)

201811
201911
[x ] 202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

of Monit. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2

files/scripts/supervisor-proc-exit-listener

lguohan · 2020-12-18T13:04:10Z

files/scripts/supervisor-proc-exit-listener

+    """
+    while True:
+        time.sleep(60)
+        if not is_process_running(process_name):


if this process is not running, then you are trapped in this loop, you never get it out.

If a critical process was not running, event listener need periodically (every 1 minute) write an alerting message into syslog like Monit, right?

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

files/scripts/supervisor-proc-exit-listener

lguohan

please use one event listener.

files/scripts/supervisor-proc-exit-listener

lguohan · 2021-01-13T19:14:19Z

any update?

'readline()'. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 · 2021-01-13T20:25:26Z

any update?

Updated. Please help me review this PR.

files/scripts/supervisor-proc-exit-listener

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

alerts. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

files/scripts/supervisor-proc-exit-listener

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

be set to "READY" state. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2021-01-19T06:23:29Z

now, code is much better readable now. please remove is_running_process() and subscribe proc running state, then I can sign-off this pr.

yozhao101 · 2021-01-19T07:19:34Z

now, code is much better readable now. please remove is_running_process() and subscribe proc running state, then I can sign-off this pr.

I think this suggestion is great and also clean. Will update the PR.

files/scripts/supervisor-proc-exit-listener

jleveque · 2021-01-19T17:57:48Z

files/scripts/supervisor-proc-exit-listener

+    if not namespace_prefix or not namespace_id:
+        syslog.syslog(syslog.LOG_ERR, "Process '{}' is not running in 'host'.".format(process_name))
+    else:
+        namespace = namespace_prefix + namespace_id
+        syslog.syslog(
+            syslog.LOG_ERR, "Process '{}' is not running in namespace '{}'.".format(process_name, namespace))


Suggest simplifying as follows:

if not namespace_prefix or not namespace_id: namespace = "host" else: namespace = namespace_prefix + namespace_id syslog.syslog(syslog.LOG_ERR, "Process '{}' is not running in namespace '{}'.".format(process_name, namespace))

"RUNNING", the event listener will receive "PROCESS_STATE_RUNNING" event and not alert anymore. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2021-01-20T08:09:15Z

please resolve the conflict.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2021-01-20T17:49:04Z

files/scripts/supervisor-proc-exit-listener

+                        process_under_alerting[process_name] = time.time()
+
+            # Handle the PROCESS_STATE_RUNNING event
+            if headers['eventname'] == 'PROCESS_STATE_RUNNING':


lguohan · 2021-01-20T17:51:01Z

i think several containers are missing. can you make sure you have changes all container file?

configuration files and fix if...elif... in event listener. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 · 2021-01-20T19:23:21Z

i think several containers are missing. can you make sure you have changes all container file?

Fixed.

lguohan · 2021-01-20T19:25:24Z

there are still containers missing? can you do a complete check??!!!

configuration file. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 · 2021-01-20T20:24:28Z

there are still containers missing? can you do a complete check??!!!

Can you help me review again please?

jleveque · 2021-01-20T20:42:11Z

there are still containers missing? can you do a complete check??!!!

Can you help me review again please?

It appears you have now covered all containers which are currently configured to use supervisor-proc-exit-listener. However, there are a handful of containers which are not using supervisor-proc-exit-listener:

dockers/docker-iccpd
dockers/docker-macsec
dockers/docker-sonic-mgmt-framework
platform/barefoot/docker-saiserver-bfn
platform/broadcom/docker-saiserver-brcm
platform/cavium/docker-saiserver-cavm
platform/innovium/docker-syncd-invm
platform/mellanox/docker-saiserver-mlnx

@lguohan: Should we add supervisor-proc-exit-listener to these containers along with (empty?) critical_processes files as part of this PR, or leave them for now and ask the appropriate owners to add it?

lguohan · 2021-01-21T05:42:36Z

retest vs please

yozhao101 · 2021-01-21T16:45:40Z

Retest vs please

yozhao101 · 2021-01-21T16:46:09Z

Retest vsimage please

yozhao101 · 2021-01-21T16:46:32Z

Retest broadcom please

yozhao101 · 2021-01-21T16:46:51Z

Retest mellanox please

lguohan · 2021-01-21T20:01:10Z

@jleveque can you check to see if you approve this?

…ord. (sonic-net#6242)" This reverts commit be3c036.

) - Why I did it Initially, we used Monit to monitor critical processes in each container. If one of critical processes was not running or crashed due to some reasons, then Monit will write an alerting message into syslog periodically. If we add a new process in a container, the corresponding Monti configuration file will also need to update. It is a little hard for maintenance. Currently we employed event listener of Supervisod to do this monitoring. Since processes in each container are managed by Supervisord, we can only focus on the logic of monitoring. - How I did it We borrowed the event listener of Supervisord to monitor critical processes in containers. The event listener will take following steps if it was notified one of critical processes exited unexpectedly: The event listener will first check whether the auto-restart mechanism was enabled for this container or not. If auto-restart mechanism was enabled, event listener will kill the Supervisord process, which should cause the container to exit and subsequently get restarted. If auto-restart mechanism was not enabled for this contianer, the event listener will enter a loop which will first sleep 1 minute and then check whether the process is running. If yes, the event listener exits. If no, an alerting message will be written into syslog. - How to verify it First, we need checked whether the auto-restart mechanism of a container was enabled or not by running the command show feature status. If enabled, one critical process should be selected and killed manually, then we need check whether the container will be restarted or not. Second, we can disable the auto-restart mechanism if it was enabled at step 1 by running the commnad sudo config feature autorestart <container_name> disabled. Then one critical process should be selected and killed. After that, we will see the alerting message which will appear in the syslog every 1 minute. - Which release branch to backport (provide reason below if selected) 201811 201911 [x ] 202006

yozhao101 added 4 commits December 17, 2020 12:55

[supervisord] Monitoring the critical processes with supervisord instead

abb9c3e

of Monit. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Delete an unuseful variable.

f42f034

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Delete an extra empty line.

97101bb

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[supervisord] Use python3 instead of python.

83e7545

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan reviewed Dec 18, 2020

View reviewed changes

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2 Outdated Show resolved Hide resolved

lguohan reviewed Dec 18, 2020

View reviewed changes

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

lguohan reviewed Dec 18, 2020

View reviewed changes

[supervisord] Delete extra space and period.

e8823ac

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan reviewed Dec 19, 2020

View reviewed changes

files/scripts/supervisor-proc-exit-listener Show resolved Hide resolved

lguohan requested changes Dec 19, 2020

View reviewed changes

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare December 25, 2020 18:34

yozhao101 marked this pull request as ready for review December 25, 2020 21:12

yozhao101 requested a review from jleveque December 28, 2020 21:17

jleveque suggested changes Dec 28, 2020

View reviewed changes

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

yozhao101 added 4 commits January 13, 2021 11:55

[Supervisord] Use non-blocking method 'select(...)' instead of

c7fc86f

'readline()'. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Fix the existing typo.

8b42bb2

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Use python3.

e999c62

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Combine the excepts together.

d51dfd6

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes Jan 13, 2021

View reviewed changes

yozhao101 added 3 commits January 13, 2021 16:47

[Supervisord] Remove trailing space and fix the alignment.

52ae7f2

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Change the variable name.

88257dd

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Reorganize the logic to process the event and log the

7483821

alerts. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes Jan 14, 2021

View reviewed changes

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

yozhao101 added 3 commits January 13, 2021 18:27

[Supervisord] Reorganize the comments.

fb0c79c

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Fix the comment about the status transition.

630d60f

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Use a flag to indicate whether the event listener should

1f9bb83

be set to "READY" state. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque added Enhancement ➕ Request for 202012 Branch labels Jan 14, 2021

lguohan reviewed Jan 19, 2021

View reviewed changes

files/scripts/supervisor-proc-exit-listener Outdated Show resolved Hide resolved

jleveque suggested changes Jan 19, 2021

View reviewed changes

[Supervisord] If the status of a process was changed from "EXITED" to

ba7f5ab

"RUNNING", the event listener will receive "PROCESS_STATE_RUNNING" event and not alert anymore. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 added 2 commits January 20, 2021 09:23

Merge branch 'master' into monitoring_critical_processes

23358dd

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Remove the outlier parenthesis at line 136.

9bbb629

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan reviewed Jan 20, 2021

View reviewed changes

[Supervisord] Add the event "PROCESS_STATE_RUNNING" in Superviord

f54bf9c

configuration files and fix if...elif... in event listener. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Supervisord] Add the "PROCESS_STATE_RUNNING" in dhcp-relay

3c4484a

configuration file. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan approved these changes Jan 21, 2021

View reviewed changes

jleveque approved these changes Jan 21, 2021

View reviewed changes

yozhao101 merged commit be3c036 into sonic-net:master Jan 21, 2021

stepanblyschak added a commit to stepanblyschak/sonic-buildimage that referenced this pull request Jan 22, 2021

Revert "[supervisord] Monitoring the critical processes with supervis…

7da879a

…ord. (sonic-net#6242)" This reverts commit be3c036.

daall added the Included in 202012 Branch label Jan 29, 2021

yozhao101 mentioned this pull request Feb 8, 2021

[pytest] Test the feature of monitoring critical processes by Supervisord. sonic-net/sonic-mgmt#2955

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[supervisord] Monitoring the critical processes with supervisord. #6242

[supervisord] Monitoring the critical processes with supervisord. #6242

yozhao101 commented Dec 17, 2020 •

edited

Loading

lguohan Dec 18, 2020

yozhao101 Dec 18, 2020 •

edited

Loading

lguohan left a comment

lguohan commented Jan 13, 2021

yozhao101 commented Jan 13, 2021

lguohan commented Jan 19, 2021

yozhao101 commented Jan 19, 2021

jleveque Jan 19, 2021

yozhao101 Jan 19, 2021

lguohan commented Jan 20, 2021

lguohan Jan 20, 2021

yozhao101 Jan 20, 2021

lguohan commented Jan 20, 2021

yozhao101 commented Jan 20, 2021

lguohan commented Jan 20, 2021

yozhao101 commented Jan 20, 2021

jleveque commented Jan 20, 2021

lguohan commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

lguohan commented Jan 21, 2021

[supervisord] Monitoring the critical processes with supervisord. #6242

[supervisord] Monitoring the critical processes with supervisord. #6242

Conversation

yozhao101 commented Dec 17, 2020 • edited Loading

lguohan Dec 18, 2020

Choose a reason for hiding this comment

yozhao101 Dec 18, 2020 • edited Loading

Choose a reason for hiding this comment

lguohan left a comment

Choose a reason for hiding this comment

lguohan commented Jan 13, 2021

yozhao101 commented Jan 13, 2021

lguohan commented Jan 19, 2021

yozhao101 commented Jan 19, 2021

jleveque Jan 19, 2021

Choose a reason for hiding this comment

yozhao101 Jan 19, 2021

Choose a reason for hiding this comment

lguohan commented Jan 20, 2021

lguohan Jan 20, 2021

Choose a reason for hiding this comment

yozhao101 Jan 20, 2021

Choose a reason for hiding this comment

lguohan commented Jan 20, 2021

yozhao101 commented Jan 20, 2021

lguohan commented Jan 20, 2021

yozhao101 commented Jan 20, 2021

jleveque commented Jan 20, 2021

lguohan commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

yozhao101 commented Jan 21, 2021

lguohan commented Jan 21, 2021

yozhao101 commented Dec 17, 2020 •

edited

Loading

yozhao101 Dec 18, 2020 •

edited

Loading