[warm-upgrade] db_migrator.py takes more time to run due to numerous entries to migrate in APPL_DB ROUTE_TABLE

[stepanblyschak]

Description

db_migrator.py takes more then 4 sec more to execute when upgrading from 202205 to 202305 comparing to an upgrade from 202205 to 202211.
db_migrator.py must run before any other service, so it is contributing to the total downtime.
This is adding 4 sec to control plane restoration time.

Steps to reproduce the issue:

1. Warm upgrade from 202205 to 202305

Describe the results you received:

In 202205->202211 upgrade:

In 202205->202305 upgrade:

The profile (timings differnet due to taking the profile when the system is at idle):

         557687 function calls (554907 primitive calls) in 3.766 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     70/1    0.001    0.000    3.766    3.766 {built-in method builtins.exec}
        1    0.001    0.001    3.766    3.766 db_migrator.py:3(<module>)
        1    0.000    0.000    3.229    3.229 db_migrator.py:1100(main)
        1    0.000    0.000    3.064    3.064 db_migrator.py:1089(migrate)
        1    0.022    0.022    3.063    3.063 db_migrator.py:1052(common_migration_ops)
        9    0.053    0.006    2.713    0.301 swsscommon.py:2517(get_table)
        1    0.016    0.016    2.689    2.689 db_migrator.py:617(migrate_route_table)
        9    0.000    0.000    1.728    0.192 swsscommon.py:2302(get_table)
        9    1.728    0.192    1.728    0.192 {built-in method swsscommon._swsscommon.ConfigDBConnector_Native_get_table}
    13062    0.183    0.000    0.770    0.000 swsscommon.py:2407(raw_to_typed)

Most of the time spent in migrate_route_table. This call does not exist in 202211 db_migrator.

Describe the results you expected:

Migrating many APPL_DB entries can cause performance problems. How can we solve this issue?

Output of show version:

SONiC Software Version: SONiC.202305_RC.35-4e4396e96_Internal
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: 4e4396e96
Build date: Thu Nov 23 05:33:59 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci02-241

Platform: x86_64-mlnx_msn4600-r0
HwSKU: ACS-MSN4600
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2126X19873
Model Number: MSN4600-VS2FO_QP1
Hardware Revision: A1
Uptime: 14:04:39 up 2 min,  3 users,  load average: 1.76, 0.83, 0.32
Date: Tue 28 Nov 2023 14:04:39

Docker images:
REPOSITORY                                         TAG                               IMAGE ID       SIZE
docker-orchagent                                   202305_RC.35-4e4396e96_Internal   cc1ccf75c8df   330MB
docker-orchagent                                   latest                            cc1ccf75c8df   330MB
docker-fpm-frr                                     202305_RC.35-4e4396e96_Internal   59eadfd1fafa   350MB
docker-fpm-frr                                     latest                            59eadfd1fafa   350MB
docker-nat                                         202305_RC.35-4e4396e96_Internal   e97e6d163b7d   321MB
docker-nat                                         latest                            e97e6d163b7d   321MB
docker-sflow                                       202305_RC.35-4e4396e96_Internal   eb5c5c7386b5   319MB
docker-sflow                                       latest                            eb5c5c7386b5   319MB
docker-teamd                                       202305_RC.35-4e4396e96_Internal   0afc86803d48   318MB
docker-teamd                                       latest                            0afc86803d48   318MB
docker-macsec                                      latest                            0f60786bc180   320MB
docker-syncd-mlnx                                  202305_RC.35-4e4396e96_Internal   d4d24d9f0240   837MB
docker-syncd-mlnx                                  latest                            d4d24d9f0240   837MB
docker-platform-monitor                            202305_RC.35-4e4396e96_Internal   e4013c208790   829MB
docker-platform-monitor                            latest                            e4013c208790   829MB
docker-dhcp-relay                                  latest                            f35a99570254   308MB
docker-eventd                                      202305_RC.35-4e4396e96_Internal   eeb3da85fabf   300MB
docker-eventd                                      latest                            eeb3da85fabf   300MB
docker-sonic-telemetry                             202305_RC.35-4e4396e96_Internal   890ab6de3317   387MB
docker-sonic-telemetry                             latest                            890ab6de3317   387MB
docker-snmp                                        202305_RC.35-4e4396e96_Internal   5aecfb2a5b59   339MB
docker-snmp                                        latest                            5aecfb2a5b59   339MB
docker-lldp                                        202305_RC.35-4e4396e96_Internal   4aa78da9ff56   343MB
docker-lldp                                        latest                            4aa78da9ff56   343MB
docker-router-advertiser                           202305_RC.35-4e4396e96_Internal   a9f2e91777c9   300MB
docker-router-advertiser                           latest                            a9f2e91777c9   300MB
docker-database                                    202305_RC.35-4e4396e96_Internal   b9732ca92d6b   300MB
docker-database                                    latest                            b9732ca92d6b   300MB
docker-mux                                         202305_RC.35-4e4396e96_Internal   d73ec6e4f231   349MB
docker-mux                                         latest                            d73ec6e4f231   349MB
docker-sonic-mgmt-framework                        202305_RC.35-4e4396e96_Internal   5057035cb2b0   415MB
docker-sonic-mgmt-framework                        latest                            5057035cb2b0   415MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):