config reload doesn't remove all the lags in a scaled configuration #19310
Assignees
Labels
Issue for 202205
Issue for 202305
Issue for 202311
Issue for 202405
Triaged
this issue has been triaged
Comments
|
@vivekrnv is this seen only on 202205? How about 202305 or master? |
|
@yxieca could you please have someone help with this issue? |
|
yinxi@yinxi-vm0:~/src/sonic-202311$ git hist 58391e3..442fe3e
|
|
@saiarcot895 the change list between the named hashes doesn't have apparent answer. Can you take a look? |
We saw it also on 202305. |
|
Issue is seen in 202311 as well |
vivekrnv
changed the title
|
@saiarcot895 @yxieca Any update on this bug? |
|
@dgsudharsan @vivekrnv Can you attach a show techsupport after the issue has happened? This seems like the config reload hasn't really happened. I tried reproing this issue at a smaller scale (24+4 LAGs) and couldn't repro the issue. |
|
Hi @saiarcot895, easiest way to repro this would be to run ecmp/inner_hashing/test_inner_hashing_lag.py on a topology with lot of ports/lag's. I'll figure out a way to send the techsupport. I might not be able to upload it here |
|
This is a timing issue. The bug is caused by interrupted/invalid clean up flow. That results in unconfigured netdev kernel leftovers. Platform: Platform: x86_64-mlnx_msn4600c-r0
HwSKU: ACS-MSN4600C
ASIC: mellanox
ASIC Count: 1Steps to reproduce:
root@sonic:/home/admin# config reload -y -f
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
root@sonic:/home/admin# show int po
Traceback (most recent call last):
File "/usr/local/bin/show", line 8, in <module>
sys.exit(cli())
^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/show/interfaces/portchannel.py", line 169, in portchannel
team.get_teams_info()
File "/usr/local/lib/python3.11/dist-packages/utilities_common/multi_asic.py", line 157, in wrapped_run_on_all_asics
func(self, *args, **kwargs)
File "/usr/local/lib/python3.11/dist-packages/show/interfaces/portchannel.py", line 49, in get_teams_info
self.get_teamshow_result()
File "/usr/local/lib/python3.11/dist-packages/show/interfaces/portchannel.py", line 115, in get_teamshow_result
info['protocol'] += "(A)" if state['runner.active'] == "true" else '(I)'
~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'runner.active'Details: Regular config reload does full LAG clean up: 2024 Aug 13 13:03:39.982010 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Waiting for port channel PortChannel44 pid 435 to stop...
2024 Aug 13 13:03:39.983914 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Waiting for port channel PortChannel22 pid 241 to stop...
2024 Aug 13 13:03:39.985737 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Waiting for port channel PortChannel41 pid 411 to stop...
2024 Aug 13 13:03:39.987608 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Waiting for port channel PortChannel108 pid 129 to stop...
2024 Aug 13 13:03:39.989530 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Waiting for port channel PortChannel43 pid 427 to stop...
2024 Aug 13 13:03:39.991344 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: LAGs cleanup is done
2024 Aug 13 13:03:39.991344 sonic NOTICE teamd#teammgrd: :- main: ExitingThe issue happens when LAG clean up was interrupted by config reload: 2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 739 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 747 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 755 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 763 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 779 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 795 (exit status 0)
2024 Aug 13 13:08:50.024035 sonic INFO teamd#supervisord 2024-08-13 13:08:50,023 INFO reaped unknown pid 811 (exit status 0)
2024 Aug 13 13:08:52.029302 sonic INFO teamd#supervisord 2024-08-13 13:08:52,026 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
2024 Aug 13 13:08:55.029996 sonic INFO teamd#supervisord 2024-08-13 13:08:55,029 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
2024 Aug 13 13:08:58.033336 sonic INFO teamd#supervisord 2024-08-13 13:08:58,032 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
2024 Aug 13 13:09:01.037406 sonic INFO teamd#supervisord 2024-08-13 13:09:01,036 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
2024 Aug 13 13:09:04.041424 sonic INFO teamd#supervisord 2024-08-13 13:09:04,040 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
2024 Aug 13 13:09:07.044124 sonic INFO teamd#supervisord 2024-08-13 13:09:07,043 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to dieWhen system init is done a next errors are observed: # teammgrd
2024 Aug 13 13:10:04.413021 sonic INFO teamd#supervisord: teammgrd Using team device "PortChannel96".
2024 Aug 13 13:10:04.413404 sonic INFO teamd#supervisord: teammgrd
2024 Aug 13 13:10:04.413492 sonic INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel96.pid"
2024 Aug 13 13:10:04.413737 sonic INFO teamd#supervisord: teammgrd
2024 Aug 13 13:10:04.413785 sonic INFO teamd#supervisord: teammgrd This program is not intended to be run as root.
2024 Aug 13 13:10:04.414080 sonic INFO teamd#supervisord: teammgrd
2024 Aug 13 13:10:04.508688 sonic INFO teamd#supervisord: teammgrd Daemon process failed.
2024 Aug 13 13:10:04.508701 sonic INFO teamd#supervisord: teammgrd Failed: Cannot allocate memory
2024 Aug 13 13:10:04.511147 sonic INFO teamd#supervisord: teammgrd cat: /var/run/teamd/PortChannel96.pid: No such file or directory
2024 Aug 13 13:10:04.511370 sonic NOTICE teamd#teammgrd: :- removeLag: Failed to remove non-existent port channel PortChannel96 pid...
2024 Aug 13 13:10:04.511529 sonic INFO teamd#supervisord 2024-08-13 13:10:04,511 INFO reaped unknown pid 891 (exit status 1)
2024 Aug 13 13:10:04.517207 sonic INFO teamd#supervisord: teammgrd Using team device "PortChannel97".
2024 Aug 13 13:10:04.517207 sonic INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel97.pid"
2024 Aug 13 13:10:04.517207 sonic INFO teamd#supervisord: teammgrd This program is not intended to be run as root.
2024 Aug 13 13:10:04.616723 sonic INFO teamd#supervisord: teammgrd Daemon process failed.
2024 Aug 13 13:10:04.616966 sonic INFO teamd#supervisord: teammgrd Failed: Cannot allocate memory
2024 Aug 13 13:10:04.619616 sonic INFO teamd#supervisord: teammgrd cat: /var/run/teamd/PortChannel97.pid: No such file or directory
2024 Aug 13 13:10:04.619968 sonic NOTICE teamd#teammgrd: :- removeLag: Failed to remove non-existent port channel PortChannel97 pid...
2024 Aug 13 13:10:04.620040 sonic INFO teamd#supervisord 2024-08-13 13:10:04,619 INFO reaped unknown pid 897 (exit status 1)
2024 Aug 13 13:10:04.626054 sonic INFO teamd#supervisord: teammgrd Using team device "PortChannel98".
2024 Aug 13 13:10:04.626054 sonic INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel98.pid"
2024 Aug 13 13:10:04.626054 sonic INFO teamd#supervisord: teammgrd This program is not intended to be run as root.
2024 Aug 13 13:10:04.728823 sonic INFO teamd#supervisord: teammgrd Daemon process failed.
2024 Aug 13 13:10:04.728823 sonic INFO teamd#supervisord: teammgrd Failed: Cannot allocate memory
2024 Aug 13 13:10:04.731290 sonic INFO teamd#supervisord: teammgrd cat: /var/run/teamd/PortChannel98.pid: No such file or directory
2024 Aug 13 13:10:04.731398 sonic NOTICE teamd#teammgrd: :- removeLag: Failed to remove non-existent port channel PortChannel98 pid...
2024 Aug 13 13:10:04.731704 sonic INFO teamd#supervisord 2024-08-13 13:10:04,731 INFO reaped unknown pid 903 (exit status 1)
# teamsyncd
2024 Aug 13 13:10:12.056246 sonic NOTICE teamd#tlm_teamd: :- try_add_lag: The LAG 'PortChannel96' has been added.
2024 Aug 13 13:10:12.057047 sonic NOTICE teamd#tlm_teamd: :- try_add_lag: The LAG 'PortChannel99' has been added.
2024 Aug 13 13:10:12.057764 sonic NOTICE teamd#tlm_teamd: :- try_add_lag: The LAG 'PortChannel98' has been added.
2024 Aug 13 13:10:12.058502 sonic NOTICE teamd#tlm_teamd: :- try_add_lag: The LAG 'PortChannel97' has been added.
2024 Aug 13 13:10:13.032500 sonic ERR teamd#teamsyncd: :- poll_descriptors: readData error: Unable to initialize team socket: Cannot assign requested address
2024 Aug 13 13:10:13.032500 sonic INFO teamd#supervisord: teamsyncd libteam: port_list_init: Failed to get port list.
2024 Aug 13 13:10:13.032500 sonic INFO teamd#supervisord: teamsyncd libteam: team_init: Failed to init port list.
2024 Aug 13 13:10:13.034781 sonic WARNING teamd#teamsyncd: :- TeamPortSync: Failed to initialize team handler. LAG=PortChannel96 error=99:Unable to initialize team socket: Cannot assign requested address, attempt=1
2024 Aug 13 13:10:13.034781 sonic INFO teamd#supervisord: teamsyncd libteam: port_list_init: Failed to get port list.
2024 Aug 13 13:10:13.034836 sonic INFO teamd#supervisord: teamsyncd libteam: team_init: Failed to init port list.
2024 Aug 13 13:10:14.037428 sonic WARNING teamd#teamsyncd: :- TeamPortSync: Failed to initialize team handler. LAG=PortChannel96 error=99:Unable to initialize team socket: Cannot assign requested address, attempt=2CLI traceback is caused by missing LAG info: [10] > /usr/local/lib/python3.11/dist-packages/show/interfaces/portchannel.py(114)get_teamshow_result()
-> info['protocol'] = "LACP"
(Pdb++) ll
97 def get_teamshow_result(self):
98 """
99 Get teamshow results by parsing the output of teamdctl and combining port channel status.
100 """
101 for team in self.teams:
102 info = {}
103 team_id = self.get_team_id(team)
104 if team_id not in self.teamsraw:
105 info['protocol'] = 'N/A'
106 self.summary[team_id] = info
107 self.summary[team_id]['ports'] = ''
108 continue
109 state = self.teamsraw[team_id]
110
111 if 'runner.active' not in state:
112 import pdb; pdb.set_trace()
113
114 -> info['protocol'] = "LACP"
115 info['protocol'] += "(A)" if state['runner.active'] == "true" else '(I)'
116
117 portchannel_status = self.get_portchannel_status(team)
118 if portchannel_status is None:
119 info['protocol'] += '(N/A)'
120 elif portchannel_status.lower() == 'up':
121 info['protocol'] += '(Up)'
122 elif portchannel_status.lower() == 'down':
123 info['protocol'] += '(Dw)'
124 else:
125 info['protocol'] += '(N/A)'
126
127 info['ports'] = ""
128 member_keys = self.db.keys(self.db.STATE_DB, PORT_CHANNEL_MEMBER_STATE_TABLE_PREFIX+team+'|*')
129 if member_keys is None:
130 info['ports'] = 'N/A'
131 else:
132 ports = [key[len(PORT_CHANNEL_MEMBER_STATE_TABLE_PREFIX+team+'|'):] for key in member_keys]
133 for port in ports:
134 status = self.get_portchannel_member_status(team, port)
135 pstate = self.db.get_all(self.db.STATE_DB, PORT_CHANNEL_MEMBER_STATE_TABLE_PREFIX+team+'|'+port)
136 selected = True if pstate['runner.aggregator.selected'] == "true" else False
137 if clicommon.get_interface_naming_mode() == "alias":
138 alias = clicommon.InterfaceAliasConverter().name_to_alias(port)
139 info["ports"] += alias + "("
140 else:
141 info["ports"] += port + "("
142 info["ports"] += "S" if selected else "D"
143 if status is None or (status == "enabled" and not selected) or (status == "disabled" and selected):
144 info["ports"] += "*"
145 ...
(Pdb++) p state
{'admin_status': 'up', 'mtu': '9100', 'oper_status': 'down', 'state': 'ok'}
(Pdb++) import pprint
(Pdb++) pprint.pprint(self.teamsraw)
{'0': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '40',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2045'},
'1': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '48',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2046'},
'10': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '56',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2047'},
'100': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '64',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2048'},
'101': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '72',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2049'},
'102': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '80',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2050'},
'103': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '88',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2051'},
'104': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '96',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2052'},
'105': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '104',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2053'},
'106': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '112',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2054'},
'107': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '120',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2055'},
'108': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '128',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2056'},
'109': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '136',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2057'},
'11': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '144',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2058'},
'12': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '152',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2059'},
'13': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '160',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2060'},
'14': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '168',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2061'},
'15': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '176',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2062'},
'16': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '184',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2063'},
'17': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '192',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2064'},
'18': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '200',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2065'},
'19': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '208',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2066'},
'2': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '216',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2067'},
'20': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '224',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2068'},
'21': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '232',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2069'},
'22': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '240',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2070'},
'23': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '248',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2071'},
'24': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '256',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2072'},
'25': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '264',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2073'},
'26': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '272',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2074'},
'27': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '280',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2075'},
'28': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '290',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2076'},
'29': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '298',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2077'},
'3': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '306',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2078'},
'30': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '314',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2079'},
'31': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '322',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2080'},
'32': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '330',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2081'},
'33': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '338',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2082'},
'34': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '346',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2083'},
'35': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '354',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2084'},
'36': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '362',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2085'},
'37': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '370',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2086'},
'38': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '378',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2087'},
'39': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '386',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2088'},
'4': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '394',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2089'},
'40': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '402',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2090'},
'41': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '410',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2091'},
'42': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '418',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2092'},
'43': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '426',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2093'},
'44': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '434',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2094'},
'45': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '442',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2095'},
'46': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '450',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2096'},
'47': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '458',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2097'},
'48': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '466',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2098'},
'49': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '474',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2099'},
'5': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '482',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2100'},
'50': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '490',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2101'},
'51': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '498',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2102'},
'52': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '506',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2103'},
'53': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '514',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2104'},
'54': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '522',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2105'},
'55': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'runner.active': 'true',
'runner.fallback': 'false',
'runner.fast_rate': 'false',
'setup.kernel_team_mode_name': 'loadbalance',
'setup.pid': '530',
'state': 'ok',
'team_device.ifinfo.dev_addr': '1c:34:da:2c:b3:00',
'team_device.ifinfo.ifindex': '2106'},
'56': {'admin_status': 'up',
'mtu': '9100',
'oper_status': 'down',
'state': 'ok'}} |
|
@saiarcot895 Do we have an ETA for fix? |
|
Fix is in progress, estimated ETA next week. |
|
Current set of changes in my local tree somewhat help with this issue, but don't fully fix it. I'm looking at if it can be fully fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
On a system with scaled configuration of ports are portchannels, and ports, config reload doesn't remove all the portchannel netdevs and ASIC objects
Steps to reproduce the issue:
Issue is seen in hash: 442fe3e
Not reproduced in the following hash: 58391e3
Describe the results you received:
After config reload, PortChannel101-104 are written from CONFIG_DB, rest are netdevs creted by teamsyncd
Logs like this are seen in syslog
tlm_teamd errors can also be seen after config reload
Describe the results you expected:
No lag's without config must be present
The text was updated successfully, but these errors were encountered: