[202012] test_pmon_ledd_kill_and_start_status is consistently failing

[bingwang-ms]

Description

test_pmon_ledd_kill_and_start_status was consistently failing with following errors

        post_daemon_status, post_daemon_pid = duthost.get_pmon_daemon_status(daemon_name)
        pytest_assert(post_daemon_status == expected_running_status,
>                             "Pcied expected restarted status is {} but is {}".format(expected_running_status, post_daemon_status))
E       Failed: Pcied expected restarted status is RUNNING but is EXITED

Steps to reproduce the issue:

1. Run test case platform_tests\daemon\test_ledd.py::test_pmon_ledd_kill_and_start_status

Describe the results you received:

Test failed with assertion error.

Describe the results you expected:

Test should pass.

Output of show version:

SONiC Software Version: SONiC.20201231.09
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: 6bac72d7fc
Build date: Mon Jul 19 21:34:00 UTC 2021
Built by: AzDevOps@sonic-int-build-workers-000281

Platform: x86_64-arista_7050cx3_32s
HwSKU: Arista-7050CX3-32S-C32
ASIC: broadcom
ASIC Count: 1
Serial Number: JPE21041273
Uptime: 02:36:18 up  5:48,  1 user,  load average: 1.22, 1.27, 1.27

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-mux                 20201231.09         72396d902fea        453MB
docker-mux                 latest              72396d902fea        453MB
docker-acms                20201231.09         818e4dfe9b32        197MB
docker-acms                latest              818e4dfe9b32        197MB
docker-orchagent           20201231.09         a40bf40365b8        430MB
docker-orchagent           latest              a40bf40365b8        430MB
docker-teamd               20201231.09         82085c9f5f19        411MB
docker-teamd               latest              82085c9f5f19        411MB
docker-fpm-frr             20201231.09         83856739bffc        430MB
docker-fpm-frr             latest              83856739bffc        430MB
docker-sonic-telemetry     20201231.09         fc8b671a4b4c        491MB
docker-sonic-telemetry     latest              fc8b671a4b4c        491MB
docker-dhcp-relay          20201231.09         648202a28eda        408MB
docker-dhcp-relay          latest              648202a28eda        408MB
docker-syncd-brcm          20201231.09         eb90310a44ef        693MB
docker-syncd-brcm          latest              eb90310a44ef        693MB
docker-snmp                20201231.09         04c196eb2983        442MB
docker-snmp                latest              04c196eb2983        442MB
docker-router-advertiser   20201231.09         06d58e4f1ad4        401MB
docker-router-advertiser   latest              06d58e4f1ad4        401MB
docker-platform-monitor    20201231.09         9935ce4a8c7e        609MB
docker-platform-monitor    latest              9935ce4a8c7e        609MB
docker-lldp                20201231.09         70ea0aeb1177        441MB
docker-lldp                latest              70ea0aeb1177        441MB
docker-database            20201231.09         cb137c1be7b5        401MB
docker-database            latest              cb137c1be7b5        401MB
k8s.gcr.io/pause           3.2                 80d28bedfe5d        683kB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[lguohan]

@sujinmkang, why the error message says pcied exited, but the test is ledd?

[sujinmkang]

@lguohan I will fix the message which is confusing. I will check the issue with Arista.

[sujinmkang]

@bingwang-ms 2 PRs are available to resolve this issue. One at sonic-mgmt with log message clean up and the other one to enable auto-restart for all PMON daemons.

[sujinmkang]

The last PR to fix all pmon regression test needs to be merged. sonic-net/sonic-mgmt#3922