monit errors in the logs "ERR memory_checker: [memory_checker] Failed to execute the command" #10088
Assignees
Comments
nazariig
changed the title
|
What is the severity of this issue? Does it have any impact other than syslog ERR message? |
1 task
yozhao101
added a commit
that referenced
this issue
Jun 17, 2022
…t created (#11129) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it This PR aims to fix an issue (#10088) by enhancing the script memory_checker. Specifically, if container is not created successfully during device is booted/rebooted, then memory_checker do not need check its memory usage. How I did it In the script memory_checker, a function is added to get names of running containers. If the specified container name is not in current running container list, then this script will exit without checking its memory usage. How to verify it I tested on a lab device by following the steps: Stops telemetry container with command sudo systemctl stop telemetry.service Removes telemetry container with command docker rm telemetry Checks whether the script memory_checker ran by Monit will generate the syslog message saying it will exit without checking memory usage of telemetry.
yxieca
pushed a commit
that referenced
this issue
Jun 19, 2022
…t created (#11129) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it This PR aims to fix an issue (#10088) by enhancing the script memory_checker. Specifically, if container is not created successfully during device is booted/rebooted, then memory_checker do not need check its memory usage. How I did it In the script memory_checker, a function is added to get names of running containers. If the specified container name is not in current running container list, then this script will exit without checking its memory usage. How to verify it I tested on a lab device by following the steps: Stops telemetry container with command sudo systemctl stop telemetry.service Removes telemetry container with command docker rm telemetry Checks whether the script memory_checker ran by Monit will generate the syslog message saying it will exit without checking memory usage of telemetry.
qiluo-msft
pushed a commit
that referenced
this issue
Jul 5, 2022
…t created (#11129) Signed-off-by: Yong Zhao yozhao@microsoft.com Why I did it This PR aims to fix an issue (#10088) by enhancing the script memory_checker. Specifically, if container is not created successfully during device is booted/rebooted, then memory_checker do not need check its memory usage. How I did it In the script memory_checker, a function is added to get names of running containers. If the specified container name is not in current running container list, then this script will exit without checking its memory usage. How to verify it I tested on a lab device by following the steps: Stops telemetry container with command sudo systemctl stop telemetry.service Removes telemetry container with command docker rm telemetry Checks whether the script memory_checker ran by Monit will generate the syslog message saying it will exit without checking memory usage of telemetry.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The issue is caused by a missing
telemetrycontainer.The situation may take place on a system start when
dockercontainer is not created yet.The root cause is
monitasync service health state check.Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
No error messages are expected when docker doesn't exist
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
Monit summary:
Monit configuration:
Monit memory checker:
root@r-boxer-sw01:/home/admin# cat /usr/bin/memory_checker def get_command_result(command): """Executes the command and return the resulting output. Args: command: A string contains the command to be executed. Returns: A string which contains the output of command. """ command_stdout = "" try: proc_instance = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True) command_stdout, command_stderr = proc_instance.communicate() if proc_instance.returncode != 0: syslog.syslog(syslog.LOG_ERR, "[memory_checker] Failed to execute the command '{}'. Return code: '{}'" .format(command, proc_instance.returncode)) sys.exit(1) except (OSError, ValueError) as err: syslog.syslog(syslog.LOG_ERR, "[memory_checker] Failed to execute the command '{}'. Error: '{}'" .format(command, err)) sys.exit(2) return command_stdout.strip()The text was updated successfully, but these errors were encountered: