xcvrd SfpStateUpdateTask takes long time to shutdown than before

[Junchao-Mellanox]

Description

The issue is introduced by commit sonic-net/sonic-platform-daemons@153ea47. In this commit, it changed SfpStateUpdateTask from Process to Thread. In order to make shutdown flow fast, I saw the code raises an exception in the shutdown flow. However, raising an exception does not make shutdown flow fast due my test on Nvidia platform.

Steps to reproduce the issue:

1. Based on your change, change line from

status, port_dict, error_dict = _wrapper_get_transceiver_change_event(timeout)

to

status, port_dict, error_dict = _wrapper_get_transceiver_change_event(60000)

2. Stop xcvrd by command “supervisorctl stop xcvrd”

Describe the results you received:

I saw xcvrd took 10 seconds to shutdown which means that is was terminated by SIG_KILL.

root@r-leopard-41:/# time supervisorctl stop xcvrd                                                                                                                                                                                          
xcvrd: stopped                                                                                                                                                                                                                               
                                                                                                                                                                                                                                             
real    0m10.292s                                                                                                                                                                                                                            
user    0m0.205s                                                                                                                                                                                                                             
sys     0m0.013s 

Describe the results you expected:

From code I see that there is chance select timeout will be set to 60000.

elif next_state == STATE_NORMAL:
                timeout = STATE_MACHINE_UPDATE_PERIOD_MSECS

xcvrd should shutdown as fast as before

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[Junchao-Mellanox]

Hi @mihirpat1, please check.

[prgeor]

Issues is seen currently only on mellanox platform.

[Junchao-Mellanox]

xcvrd is a common sonic component shared by all vendors. Changes on it should work on all platforms.

BTW, I checked Arista platform API implementation. Arista platform did not run into the issue because the timeout value will be no more than 1 second in Arista platform. Please check this code from platform/barefoot/sonic-platform-modules-arista/arista/utils/sonic_platform/event.py:

   def wait(self, timeout=0):
      res = self.get_empty_results()
      block = (timeout == 0)
      detected = False

      while not detected and (timeout > 0 or block):
         begin = time.time()
         interval = self.pollInterval if block else min(timeout, self.pollInterval)

See the last line, the epoll timeout value will never be larger than self.pollInterval. And pollInterval is default 1sec:

class EventWatcher(object):
   def __init__(self, preserve=False, pollInterval=1000.):

So, I would suggest you run another test on Arista: change pollInterval from 1000 to 60000.

[prgeor]

@Junchao-Mellanox as discussed can you break down the sleep in your platform code to polling every < 1sec? That should fix this issue.

[Junchao-Mellanox]

PR created: #13611