[memory_monitoring] Enhance monitoring the memory usage of containers #19179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kperumalbfn
reviewed
Jun 3, 2024
zbud-msft
reviewed
Jun 5, 2024
5 tasks
…ic-buildimage into memory_checker
…ic-buildimage into memory_checker
…ic-buildimage into memory_checker
5 tasks
maipbui
reviewed
Jun 12, 2024
|
/azpw ms_conflict -f |
1 similar comment
|
/azpw ms_conflict -f |
|
/azpw ms_checker -f |
|
/azpw ms_checker |
|
/azpw ms_checker |
maipbui
approved these changes
Sep 5, 2024
qiluo-msft
approved these changes
Sep 5, 2024
|
/azpw ms_checker |
|
/azpw ms_conflict |
FengPan-Frank
added a commit
to FengPan-Frank/sonic-buildimage
that referenced
this pull request
Sep 9, 2024
…sonic-net#19179) ### Why I did it We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit. ### How I did it Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats. Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage. #### How to verify it Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry" <img width="799" alt="image" src="https://github.com/sonic-net/sonic-buildimage/assets/97083744/3807fc7f-cfc2-4e2f-a078-eaf08b68f803"> Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below: <img width="1121" alt="image" src="https://github.com/user-attachments/assets/2c7ce241-7967-41ee-a2e9-4bdb2e43f8c2">
FengPan-Frank
added a commit
to FengPan-Frank/sonic-buildimage
that referenced
this pull request
Sep 9, 2024
…sonic-net#19179) We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit. Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats. Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage. Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry" <img width="799" alt="image" src="https://github.com/sonic-net/sonic-buildimage/assets/97083744/3807fc7f-cfc2-4e2f-a078-eaf08b68f803"> Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below: <img width="1121" alt="image" src="https://github.com/user-attachments/assets/2c7ce241-7967-41ee-a2e9-4bdb2e43f8c2">
11 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Sep 11, 2024
…sonic-net#19179) ### Why I did it We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit. ### How I did it Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats. Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage. #### How to verify it Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry" <img width="799" alt="image" src="https://github.com/sonic-net/sonic-buildimage/assets/97083744/3807fc7f-cfc2-4e2f-a078-eaf08b68f803"> Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below: <img width="1121" alt="image" src="https://github.com/user-attachments/assets/2c7ce241-7967-41ee-a2e9-4bdb2e43f8c2">
|
Cherry-pick PR to 202311: #20234 |
mssonicbld
pushed a commit
that referenced
this pull request
Sep 12, 2024
…#19179) ### Why I did it We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit. ### How I did it Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats. Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage. #### How to verify it Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry" <img width="799" alt="image" src="https://github.com/sonic-net/sonic-buildimage/assets/97083744/3807fc7f-cfc2-4e2f-a078-eaf08b68f803"> Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below: <img width="1121" alt="image" src="https://github.com/user-attachments/assets/2c7ce241-7967-41ee-a2e9-4bdb2e43f8c2">
mssonicbld
added
Included in 202311 Branch
and removed
Created PR to 202311 Branch
labels
vvolam
pushed a commit
to vvolam/sonic-buildimage
that referenced
this pull request
Sep 12, 2024
…sonic-net#19179) ### Why I did it We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit. ### How I did it Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats. Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage. #### How to verify it Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry" <img width="799" alt="image" src="https://github.com/sonic-net/sonic-buildimage/assets/97083744/3807fc7f-cfc2-4e2f-a078-eaf08b68f803"> Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below: <img width="1121" alt="image" src="https://github.com/user-attachments/assets/2c7ce241-7967-41ee-a2e9-4bdb2e43f8c2">
|
Hi @FengPan-Frank, publish_events is now deleted in this PR, which is not expected. |
11 tasks
@zbud-msft sorry seems this publish_events was missed previously, may I ask which test case failed? so that I can further check it if update this part code later. |
qiluo-msft
pushed a commit
that referenced
this pull request
Sep 23, 2024
### Why I did it #19179 removed call to publish_events when memory usage container exceeds threshold, causing test_events to fail. ### How I did it Add back call to publish_events #### How to verify it Manual test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
We need to restrict memory usage of container specifically, and the reliable option is to read cgroup subsystem files instead of using "docker stats" commands, since the commands will be no response if containers hits hard limit.
Work item tracking
How I did it
Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats.
Refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage.
How to verify it
Local verified, since it's just internal enhancement for getting memory usage of container, below are comparison between new memory_check and previous implementation based on "docker stats --no-stream --format {{.MemUsage}} telemetry"
Added Unit test code, since there's no build time UT available in this repo currently so verified manually as below:
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)