Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

orchagent continuously prints out "set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS" #5570

Closed
daall opened this issue Oct 9, 2020 · 1 comment · Fixed by #5666, sonic-net/sonic-sairedis#680 or sonic-net/sonic-swss#1478
Closed

orchagent continuously prints out "set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS" #5570

daall opened this issue Oct 9, 2020 · 1 comment · Fixed by #5666, sonic-net/sonic-sairedis#680 or sonic-net/sonic-swss#1478
Assignees
@theasianpianist
Labels
Bug 🐛 Master Branch Quality P0 Priority of the issue

Comments

@daall
Copy link
Contributor

daall commented Oct 9, 2020

Description
Orchagent continually sprays this message to the syslog:

set: setting attribute 0x10000004 status: SAI_STATUS_SUCCESS

Which causes this to occur:

Oct  9 00:33:00.374116 sonic INFO swss#rsyslogd: imuxsock[pid: 42, name: /usr/bin/orchagent] from <sonic:orchagent>: begin to drop messages due to rate-limiting
Oct  9 00:36:57.412514 sonic INFO swss#rsyslogd: imuxsock[pid: 42, name: /usr/bin/orchagent]: 32511 messages lost due to rate-limiting

Which masks legitimate syslog messages from orchagent (e.g. "create ACL table foo").

Steps to reproduce the issue:

  1. Start the DUT.
  2. Check the syslog.

Describe the results you received:

Oct  9 00:33:00.374116 sonic INFO swss#rsyslogd: imuxsock[pid: 42, name: /usr/bin/orchagent] from <sonic:orchagent>: begin to drop messages due to rate-limiting
Oct  9 00:36:57.412514 sonic INFO swss#rsyslogd: imuxsock[pid: 42, name: /usr/bin/orchagent]: 32511 messages lost due to rate-limiting

Describe the results you expected:
There shouldn't be any throttling/need to throttle while orchagent is idle and no updates are being received from neighbors/bgp/etc.
@daall daall mentioned this issue Oct 9, 2020
Closed
@daall daall assigned theasianpianist and unassigned lguohan Oct 14, 2020
@theasianpianist
Copy link
Contributor

From what I can tell, the SAI_STATUS_SUCCESS message does not seem to be causing the rate limiting. Will continue investigating.

@daall next time you notice this, can you grep for "begin to drop messages" in your syslogs?

@theasianpianist theasianpianist mentioned this issue Oct 19, 2020
Merged
3 tasks
@daall daall linked a pull request Oct 19, 2020 that will close this issue
[docker-base]: Rate limit priority INFO and lower in syslog #5666
Merged
3 tasks
This was referenced Oct 23, 2020
Merged
Merged
qiluo-msft added a commit to sonic-net/sonic-swss that referenced this issue Nov 5, 2020
@qiluo-msft
Revert: swss: flush g_asicState after each event is done #570 (#1478)
b9084a7 
**What I did**
Revert #570
We should only `flush` the orchagent/syncd communication channel when there is no active tasks in orchagent. This will not influence the end-to-end performance in long run but introduce SELECT_TIMEOUT (1 s) latency if there is remaining data left inside the orchagent/syncd communication channel after previous `flush`, which is not a big deal.

Fix sonic-net/sonic-buildimage#5570
daall pushed a commit to daall/sonic-swss that referenced this issue Dec 7, 2020
@qiluo-msft @daall
Revert: swss: flush g_asicState after each event is done sonic-net#570 (
eea7b38 
sonic-net#1478)

**What I did**
Revert sonic-net#570
We should only `flush` the orchagent/syncd communication channel when there is no active tasks in orchagent. This will not influence the end-to-end performance in long run but introduce SELECT_TIMEOUT (1 s) latency if there is remaining data left inside the orchagent/syncd communication channel after previous `flush`, which is not a big deal.

Fix sonic-net/sonic-buildimage#5570
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@theasianpianist theasianpianist

Labels
Bug 🐛 Master Branch Quality P0 Priority of the issue
Projects
None yet
Milestone
No milestone
3 participants
@lguohan @theasianpianist @daall