[pytest] Test the feature of monitoring critical processes by Supervisord. #2955

yozhao101 · 2021-02-08T07:54:35Z

Signed-off-by: Yong Zhao yozhao@microsoft.com

Description of PR

Summary:
This PR aims to test the feature of monitoring the critical processes by Supervisord.

Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
[ x] Test case(new/improvement)

Approach

What is the motivation for this PR?

The motivation for this PR is to test the feature of monitoring the critical processes by Supervisord. This monitoring feature required Supervisord should write an alerting message into syslog if a critical process exited unexpectedly and the autorestart of container hosting this process was disabled.

How did you do it?

The logic of this PR is as follows:
Step 1: Records the autorestart status of all containers.
Step 2: Gets the number of ASICs on DUT.
Step 3: Gets all the containers which were enabled on DUT and at the same time for each enabled container, retrieve a list of namespaces which this container should be running.
Step 4: Disables the autorestart of all enabled containers.
Step 5: Manually stops all the critical process in all enabled containers on DUT.
Step 6: Waits for 70 seconds for the alerting messages to appear.
Step 7: Checks whether the expected alerting messages appeared in syslog or not.
Step 8: Executes the config_reload(...) to restart containers.
Step 9: Checks whether all critical processes in enabled containers are running.
Step 10: Restores the autorestart status of all containers.
Step 11: Does the post-check.

How did you verify/test it?

I tested this PR on a physical testbed with single ASIC.

Any platform specific information?

N/A

Supported testbed topology if it's a new test case?

N/A

Documentation

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-08T08:22:23Z

This pull request introduces 1 alert when merging 9af1b25 into 2b00e37 - view on LGTM.com

new alerts:

1 for Syntax error

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2021-02-10T22:26:08Z

tests/monitoring_processes/test_monitoring_critical_processes.py

+    """
+    logger.info("Checking the alerting messages from syslog...")
+    command_output = duthost.shell("sudo cat /var/log/syslog | grep '.*ERR.*supervisor-proc-exit-listener'",
+                                   module_ignore_errors=True)


why not use log analyzer to achieve this?

Will fix. Another pytest PR has the same issue and will also fix.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

tests/monitoring_processes/test_monitoring_critical_processes.py

1.Use `num_asics()` to get number of asics. 2.Use `sonic-db-cli CONFIG_DB keys 'FEATURE|*' to get the container list. 3.Move two functions to `tests/common/helpers/dut_utils.py`. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-18T01:40:29Z

This pull request introduces 1 alert when merging 418ebed into e893f73 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

tests/monitoring_processes/test_monitoring_critical_processes.py

the DUTs installed with 201911 or old image version. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-18T18:36:56Z

This pull request introduces 1 alert when merging f0c549c into 856c106 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-19T07:33:40Z

This pull request introduces 1 alert when merging b0ae37a into cb81ec8 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-19T08:49:38Z

This pull request introduces 1 alert when merging 9cfca51 into a1a713e - view on LGTM.com

new alerts:

1 for Variable defined multiple times

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-19T18:16:16Z

This pull request introduces 1 alert when merging f0af91f into a1a713e - view on LGTM.com

new alerts:

1 for Variable defined multiple times

tests/monitoring_processes/test_monitoring_critical_processes.py

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lgtm-com · 2021-02-24T08:01:46Z

This pull request introduces 1 alert when merging 2bf80f0 into 795d794 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

lguohan · 2021-02-25T23:10:48Z

check the lgtm alerts?

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yxieca · 2021-02-18T17:11:47Z

tests/monitoring_processes/test_monitoring_critical_processes.py

+    return is_enabled, has_global_scope, has_per_asic_scope
+
+
+def parse_feature_table(duthost, num_asics, skip_containers):


why not using https://github.com/Azure/sonic-mgmt/blob/master/tests/common/devices.py#L968 and https://github.com/Azure/sonic-mgmt/blob/master/tests/common/devices.py#L949?

The function parse_feature_table(...) was used to get running containers list and which namespaces currently they are running. Currently this function was already removed and replaced by the function get_containers_namespace_ids(...) in which I used the get_feature_status(...) (https://github.com/Azure/sonic-mgmt/blob/67d5be22c740636c132047340d41fce163b9cee1/tests/common/devices.py#L972) to get the running container list.

jleveque · 2021-02-27T01:35:13Z

tests/kvmtest.sh

@@ -154,7 +155,8 @@ test_t1_lag() {
    lldp/test_lldp.py \
    route/test_default_route.py \
    platform_tests/test_cpu_memory_usage.py \
-    bgp/test_bgpmon.py"
+    bgp/test_bgpmon.py \
+    monitoring_processes/test_monitoring_critical_processes.py"


Suggest renaming the directory to process_monitoring and the file to test_critical_process_monitoring.py

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

critical process monitoring in kvmtest.sh. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Test the feature of monitoring critical process by Supervisord.

9af1b25

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 changed the title ~~[pytest] Test the feature of monitoring critical process by Supervisord.~~ [pytest] Test the feature of monitoring critical processes by Supervisord. Feb 8, 2021

yozhao101 added 5 commits February 8, 2021 23:02

[pytest] Fix the syntax error and remove trailing spaces.

5aeeeb4

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Skip DUTs which were installed with 201911 image.

76342c1

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Fix a logic error.

85973df

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Remove the extra imports.

2737c77

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Reorganize the comments.

48179bf

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 marked this pull request as ready for review February 9, 2021 23:14

yozhao101 requested a review from a team as a code owner February 9, 2021 23:14

yozhao101 added 3 commits February 9, 2021 15:26

[pytest] Reorganize the comments.

ae9d1b3

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Fix the typos in comments.

394ba40

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Fix a typo in the log message.

7848cb7

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 requested review from yxieca, jleveque and lguohan February 10, 2021 19:09

lguohan reviewed Feb 10, 2021

View reviewed changes

yozhao101 added 3 commits February 11, 2021 14:38

[pytest] Fix a typo.

6f1a456

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Add missing the Macro.

09e3625

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Use loganalyzer to analyzer the alerting messages from syslog.

f266144

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 16, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

[pytest] Made the following changes:

418ebed

1.Use `num_asics()` to get number of asics. 2.Use `sonic-db-cli CONFIG_DB keys 'FEATURE|*' to get the container list. 3.Move two functions to `tests/common/helpers/dut_utils.py`. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yxieca reviewed Feb 18, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yxieca reviewed Feb 18, 2021

View reviewed changes

tests/monitoring_processes/test_monitoring_critical_processes.py Outdated Show resolved Hide resolved

yozhao101 added 3 commits February 18, 2021 09:57

[pytest] Use pytest_require(...) instead of pytest.skip(...) to skip

554b17d

the DUTs installed with 201911 or old image version. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Import the pytest_require(...).

aa6fdbe

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Import the logging.

f0c549c

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 added 2 commits February 18, 2021 22:48

[pytest] Add a function get_namespace_ids(...) into devices.py.

23cd3e3

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Remove extra logging statement and fix typos.

b0ae37a

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Fix a bug.

9cfca51

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Reorganize the comments.

f0af91f

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes Feb 24, 2021

View reviewed changes

[pytest] Rename the functions and reorganize their comments.

2bf80f0

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 added 2 commits February 25, 2021 17:22

[pytest] Add this test case into kvm test.

37b9f43

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Skip the gbsyncd container.

8cb60fd

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yxieca approved these changes Feb 26, 2021

View reviewed changes

jleveque suggested changes Feb 27, 2021

View reviewed changes

yozhao101 added 2 commits February 26, 2021 17:48

[pytest] Rename the directory and test file names.

485e176

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[pytest] Update the directory and test file names related to testing

cbeb62c

critical process monitoring in kvmtest.sh. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque approved these changes Feb 27, 2021

View reviewed changes

yozhao101 merged commit eccce7d into sonic-net:master Feb 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pytest] Test the feature of monitoring critical processes by Supervisord. #2955

[pytest] Test the feature of monitoring critical processes by Supervisord. #2955

yozhao101 commented Feb 8, 2021 •

edited

Loading

lgtm-com bot commented Feb 8, 2021

lguohan Feb 10, 2021

yozhao101 Feb 10, 2021 •

edited

Loading

lgtm-com bot commented Feb 18, 2021

lgtm-com bot commented Feb 18, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 24, 2021

lguohan commented Feb 25, 2021

yxieca Feb 18, 2021

yozhao101 Feb 26, 2021

jleveque Feb 27, 2021

yozhao101 Feb 27, 2021

		return is_enabled, has_global_scope, has_per_asic_scope


		def parse_feature_table(duthost, num_asics, skip_containers):

[pytest] Test the feature of monitoring critical processes by Supervisord. #2955

[pytest] Test the feature of monitoring critical processes by Supervisord. #2955

Conversation

yozhao101 commented Feb 8, 2021 • edited Loading

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

lgtm-com bot commented Feb 8, 2021

lguohan Feb 10, 2021

Choose a reason for hiding this comment

yozhao101 Feb 10, 2021 • edited Loading

Choose a reason for hiding this comment

lgtm-com bot commented Feb 18, 2021

lgtm-com bot commented Feb 18, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 19, 2021

lgtm-com bot commented Feb 24, 2021

lguohan commented Feb 25, 2021

yxieca Feb 18, 2021

Choose a reason for hiding this comment

yozhao101 Feb 26, 2021

Choose a reason for hiding this comment

jleveque Feb 27, 2021

Choose a reason for hiding this comment

yozhao101 Feb 27, 2021

Choose a reason for hiding this comment

yozhao101 commented Feb 8, 2021 •

edited

Loading

yozhao101 Feb 10, 2021 •

edited

Loading