CRM available counters does not decrease with route added

[Xichen96]

Description

There are 2 counters that keep track of CRM resources, one of them counts used and the other counts available. The expectation is that once a route is added, the used counter will increase by one and availble decrease by one. Now we are observing changes in used counter but not availble counter. When a route is added, SWSS will call SAI create_route_entry, once that is successful, used counter will be increased. SAI get_switch_attribute will also be called to retrieve the available counter and synced to redis database. Currently we are seeing logs that indicates success of create_route_entry, but calls to get_switch_attribute remains the same, which is not expected.

Steps to reproduce the issue:

On the dut:

1. sonic-db-cli COUNTERS_DB HMGET CRM:STATS crm_stats_ipv6_route_used crm_stats_ipv6_route_available to retrieve the 2 counters
2. ip -6 route add 2001::/126 via fc00::72 to add route. Use ip -6 route show to verify that it's been added. /var/log/syslog should also contain logs that indicate route has been added.
3. execute sonic-db-cli COUNTERS_DB HMGET CRM:STATS crm_stats_ipv6_route_used crm_stats_ipv6_route_available again and see that used counter has been increased but available counter remains unchanged.
4. ip -6 route del 2001::/126 via fc00::72 to remove route.

Describe the results you received:

Available counter remains unchanged before and after adding route and deleting route.

Describe the results you expected:

Available counter decreases by 1 as a route is added.

Output of show version:

Platform: x86_64-cel_e1031-r0
HwSku: Celestica-E1031-T48S4
ASIC: broadcom
SONIC.20201231.39

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[prsunny]

Are you observing this issue while adding only 1 entry or is it the same available count even if you add ~10-15 entries? It is possible that h/w may use some shared resource and available count may not change for just 1 entry. So this is more like an approximate available count rather than exact value. Please check the following:

1. Add few more routes, say 10 to 15 and check if available count is changed
2. Please make sure to wait for the polling interval.

[Xichen96]

@prsunny Hi, I added 10 routes, and still only used counter changed. Also I noticed 2 changes. 1. sonic-db-cli took a long time after reboot to function correctly, like more than 5 minutes. 2. When I posted this issue, the used counter was refreshed almost immediately. Now it changes after about 2 minutes.