monit errors in the logs "ERR /memory_checker: Failed to retrieve the running container list from docker daemon" #11472
Assignees
Labels
Comments
yozhao101
pushed a commit
that referenced
this issue
Jul 27, 2022
…emon is not running (#11476) Fix in Monit memory_checker plugin. Skip fetching running containers if docker engine is down (can happen in deinit). This PR fixes issue #11472. Signed-off-by: liora liora@nvidia.com Why I did it In the case where Monit runs during deinit flow, memory_checker plugin is fetching the running containers without checking if Docker service is still running. I added this check. How I did it Use systemctl is-active to check if Docker engine is still running. How to verify it Use systemctl to stop docker engine and reload Monit, no errors in log and relevant print appears in log. Which release branch to backport (provide reason below if selected) The fix is required in 202205 and 202012 since the PR that introduced the issue was cherry picked to those branches (#11129).
qiluo-msft
pushed a commit
that referenced
this issue
Jul 27, 2022
…emon is not running (#11476) Fix in Monit memory_checker plugin. Skip fetching running containers if docker engine is down (can happen in deinit). This PR fixes issue #11472. Signed-off-by: liora liora@nvidia.com Why I did it In the case where Monit runs during deinit flow, memory_checker plugin is fetching the running containers without checking if Docker service is still running. I added this check. How I did it Use systemctl is-active to check if Docker engine is still running. How to verify it Use systemctl to stop docker engine and reload Monit, no errors in log and relevant print appears in log. Which release branch to backport (provide reason below if selected) The fix is required in 202205 and 202012 since the PR that introduced the issue was cherry picked to those branches (#11129).
yxieca
pushed a commit
that referenced
this issue
Jul 28, 2022
…emon is not running (#11476) Fix in Monit memory_checker plugin. Skip fetching running containers if docker engine is down (can happen in deinit). This PR fixes issue #11472. Signed-off-by: liora liora@nvidia.com Why I did it In the case where Monit runs during deinit flow, memory_checker plugin is fetching the running containers without checking if Docker service is still running. I added this check. How I did it Use systemctl is-active to check if Docker engine is still running. How to verify it Use systemctl to stop docker engine and reload Monit, no errors in log and relevant print appears in log. Which release branch to backport (provide reason below if selected) The fix is required in 202205 and 202012 since the PR that introduced the issue was cherry picked to those branches (#11129).
skbarista
pushed a commit
to skbarista/sonic-buildimage
that referenced
this issue
Aug 17, 2022
…emon is not running (sonic-net#11476) Fix in Monit memory_checker plugin. Skip fetching running containers if docker engine is down (can happen in deinit). This PR fixes issue sonic-net#11472. Signed-off-by: liora liora@nvidia.com Why I did it In the case where Monit runs during deinit flow, memory_checker plugin is fetching the running containers without checking if Docker service is still running. I added this check. How I did it Use systemctl is-active to check if Docker engine is still running. How to verify it Use systemctl to stop docker engine and reload Monit, no errors in log and relevant print appears in log. Which release branch to backport (provide reason below if selected) The fix is required in 202205 and 202012 since the PR that introduced the issue was cherry picked to those branches (sonic-net#11129).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The following error appear in log during deinit flow:
ERR /memory_checker: Failed to retrieve the running container list from docker daemon! Error message is: 'Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))'
Steps to reproduce the issue:
This is issue is not deterministic. It happens when Monit plugin memory_checker that run periodically is running after reboot was perfomed. The plugin is trying to fetch the running containers but Docker engine was already stopped by systemd.
Describe the results you received:
The following error appear in log:
ERR /memory_checker: Failed to retrieve the running container list from docker daemon! Error message is: 'Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))'
Describe the results you expected:
If Docker engine is not running, do not try to fetch running containers and return empty list.
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: