Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flex counter] Flex counter threads consume too much CPU resources #2031

Merged
merged 2 commits into from
Nov 18, 2021

Conversation

StormLiangMS
Copy link
Contributor

@StormLiangMS StormLiangMS commented Nov 15, 2021

What I did
Watermark flex counter will consume many CPU resources on some platforms.
By increasing the default interval for those counters from 10 seconds to 60 seconds to
workaround this issue. The performance issue of flex counter need to be addressed
separately, this is just to reduce CPU consumption by adjusting the granularity of watermark related counters .
If smaller granularity is needed, the interval could be adjusted through CLI under SONIC shell as below.
counterpoll watermark interval <interval_val_in_ms>

Fixes sonic-net/sonic-buildimage#9202

Why I did it

How I verified it

Details if related

1. water thread flex counter will consume many cpu resouces on some platforms.
change the default interval for those counters from 10 seconds to 60 seconds to
workaround this issue. The performance issue of flex counter read need to be addressed
separately, this is just a workaround.
if it needs smaller granulatiry, the interval could be adjusted through CLI under SONIC shell.
counterpoll
Copy link
Collaborator

@prsunny prsunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this required to be backported?

@StormLiangMS
Copy link
Contributor Author

is this required to be backported?

Yes, this one should be back ported to 201911 and 202012.

@prsunny
Copy link
Collaborator

prsunny commented Nov 19, 2021

@StormLiangMS , for future PRs, please create a fork of this repo and raise PR from there

@prsunny prsunny deleted the cpu_utilization_storm branch November 19, 2021 02:45
@StormLiangMS StormLiangMS changed the title [flex counter] Flex counter threads consume too much CPU resources #9202 [flex counter] Flex counter threads consume too much CPU resources Nov 23, 2021
qiluo-msft pushed a commit that referenced this pull request Nov 23, 2021
…202 (#2031)

* [flex counter] Flex counter threads consume too much CPU resources #9202

1. water thread flex counter will consume many cpu resouces on some platforms.
change the default interval for those counters from 10 seconds to 60 seconds to
workaround this issue. The performance issue of flex counter read need to be addressed
separately, this is just a workaround.
if it needs smaller granulatiry, the interval could be adjusted through CLI under SONIC shell.
counterpoll

* Increase the buffer pool watermark interval from 10 seconds to 60 seconds
neethajohn added a commit to sonic-net/sonic-utilities that referenced this pull request Feb 28, 2022
… counters (#2082)

Signed-off-by: Neetha John <nejo@microsoft.com>

sonic-net/sonic-swss#2031 updated the flex counter polling interval to 60s for watermark counters but the show command wasn't updated to reflect the correct default value

What I did
Display the correct poll interval for watermark related counters in the 'counterpoll show' command

How I did it
Update the default interval to be the same as the one updated by the Orchs

How to verify it
Issue "counterpoll show", the queue, pg and buffer pool watermark should show default 60s
Updated the counterpoll unit tests to reflect the same
judyjoseph pushed a commit to sonic-net/sonic-utilities that referenced this pull request Mar 7, 2022
… counters (#2082)

Signed-off-by: Neetha John <nejo@microsoft.com>

sonic-net/sonic-swss#2031 updated the flex counter polling interval to 60s for watermark counters but the show command wasn't updated to reflect the correct default value

What I did
Display the correct poll interval for watermark related counters in the 'counterpoll show' command

How I did it
Update the default interval to be the same as the one updated by the Orchs

How to verify it
Issue "counterpoll show", the queue, pg and buffer pool watermark should show default 60s
Updated the counterpoll unit tests to reflect the same
malletvapid23 added a commit to malletvapid23/Sonic-Utility that referenced this pull request Aug 3, 2023
… counters (#2082)

Signed-off-by: Neetha John <nejo@microsoft.com>

sonic-net/sonic-swss#2031 updated the flex counter polling interval to 60s for watermark counters but the show command wasn't updated to reflect the correct default value

What I did
Display the correct poll interval for watermark related counters in the 'counterpoll show' command

How I did it
Update the default interval to be the same as the one updated by the Orchs

How to verify it
Issue "counterpoll show", the queue, pg and buffer pool watermark should show default 60s
Updated the counterpoll unit tests to reflect the same
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[flex counter] Flex counter threads consume too much CPU resources.
4 participants