DB migration does not account mandatory entries in new init_cfg.json/FEATURE table in particular

[stepanblyschak]

sonic_dump_arc-switch1004_20210203_163415.tar.gz

Description

DB migration does not account mandatory entries in new init_cfg.json/FEATURE table in particular

Steps to reproduce the issue:

1. Install SONiC 201811 image and run it
2. Install SONiC 201911 image and perform warm upgrade 201811->201911
3. Observe that there are no FEATURE table config and services fail with errors:

Jan 28 17:15:14.132832 ptr-sonic-n1-t2 ERR dhcp_relay#supervisor-proc-exit-listener: Unable to retrieve feature 'dhcp_relay'. Exiting...
Jan 28 17:15:14.233687 ptr-sonic-n1-t2 ERR teamd#supervisor-proc-exit-listener: Unable to retrieve feature 'teamd'. Exiting...
Jan 28 17:15:14.882583 ptr-sonic-n1-t2 ERR swss#supervisor-proc-exit-listener: Unable to retrieve feature 'swss'. Exiting...
Jan 28 17:15:15.331937 ptr-sonic-n1-t2 ERR bgp#supervisor-proc-exit-listener: Unable to retrieve feature 'bgp'. Exiting...
Jan 28 17:15:15.507529 ptr-sonic-n1-t2 ERR snmp#supervisor-proc-exit-listener: Unable to retrieve feature 'snmp'. Exiting...
Jan 28 17:15:15.522642 ptr-sonic-n1-t2 ERR lldp#supervisor-proc-exit-listener: Unable to retrieve feature 'lldp'. Exiting...
Jan 28 17:15:15.613130 ptr-sonic-n1-t2 ERR syncd#supervisor-proc-exit-listener: Unable to retrieve feature 'syncd'. Exiting...
Jan 28 17:15:15.906551 ptr-sonic-n1-t2 ERR radv#supervisor-proc-exit-listener: Unable to retrieve feature 'radv'. Exiting...

Describe the results you received:
Non functional system after warm upgrade from 201811 to 201911

Describe the results you expected:
Smooth upgrade from 201811 to 201911

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

SONiC Software Version: SONiC.20191130.55
Distribution: Debian 9.13
Kernel: 4.9.0-11-2-amd64
Build commit: 5c4b6f8bb
Build date: Fri Nov 27 07:23:50 UTC 2020
Built by: sonicbld@jenkins-slave-phx-2

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700-D48C8
ASIC: mellanox
Serial Number: MT1932X27081
Uptime: 17:16:35 up  2:31,  1 user,  load average: 4.38, 3.69, 3.38

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-mlnx          20191130.55         2212cf4f2189        397MB
docker-syncd-mlnx          latest              2212cf4f2189        397MB
docker-sonic-telemetry     20191130.55         8a7f7f045b28        353MB
docker-sonic-telemetry     latest              8a7f7f045b28        353MB
docker-dhcp-relay          20191130.55         5c120e9a9bc6        300MB
docker-dhcp-relay          latest              5c120e9a9bc6        300MB
docker-router-advertiser   20191130.55         6363ddadc796        290MB
docker-router-advertiser   latest              6363ddadc796        290MB
docker-platform-monitor    20191130.55         2d9ad9939479        664MB
docker-platform-monitor    latest              2d9ad9939479        664MB
docker-fpm-frr             20191130.55         1b44f88efdfe        325MB
docker-fpm-frr             latest              1b44f88efdfe        325MB
docker-lldp-sv2            20191130.55         6febc5506feb        312MB
docker-lldp-sv2            latest              6febc5506feb        312MB
docker-sonic-restapi       20191130.55         d831a978388d        302MB
docker-sonic-restapi       latest              d831a978388d        302MB
docker-database            20191130.55         ea9a8769c1b3        289MB
docker-database            latest              ea9a8769c1b3        289MB
docker-acms                20191130.55         e61822123d39        182MB
docker-acms                latest              e61822123d39        182MB
docker-teamd               20191130.55         8ee2637b452b        315MB
docker-teamd               latest              8ee2637b452b        315MB
docker-snmp-sv2            20191130.55         eaa52e5f541c        348MB
docker-snmp-sv2            latest              eaa52e5f541c        348MB
docker-orchagent           20191130.55         1eaeb5a11c8b        333MB
docker-orchagent           latest              1eaeb5a11c8b        333MB
k8s.gcr.io/pause           3.2                 80d28bedfe5d        683kB

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[anshuv-mfst]

@jleveque - could you please provide input on this issue, thanks.

[lguohan]

@shi-su , can you check this one? it seems it affect 201811 to master as well.

[anshuv-mfst]

@stepanblyschak - could you please confirm if the issue is seen when upgrading to master, thanks.

[shi-su]

It seems that the FEATURE table does not show up in init_cfg.json, and the format of the file does not seem to come from the template file. I checked the files in 20191130.58 image and master image, both look normal. Is it possible that this file has ever been replaced?

[liat-grozovik]

the file is not replaced.
it is an upgrade via warmboot from 201811 to 201911

[shi-su]

Just noticed that the dump file seems to come from an upgrade from 201811.170-701a304 to 20181130.87. Wondering if it is possible to share the dump file you got in 201811->201911 upgrade?

[stepanblyschak]

Dump with an upgrade:
sonic_dump_ptr-sonic-n1-t2_20210128_171634(1).tar.gz

[daall]

@shi-su can you check the dump Stepan provided?

[shi-su]

I did some investigation into this issue. It appears that the warm boot logic does not read or apply init_cfg.json. I need to check with the owner of FEATURE table and DB migrator for a fix.

[shi-su]

This should have been fixed by sonic-net/sonic-utilities#1209. Need to include the commit in 201911.

[liat-grozovik]

Can it be cherry pick or need to create new PR against 201911? I don't see the label added to the fixed PR Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Shi Su <notifications@github.com> Sent: Thursday, March 4, 2021 1:59:02 AM To: Azure/sonic-buildimage <sonic-buildimage@noreply.github.com> Cc: Liat Grozovik <liatg@nvidia.com>; Comment <comment@noreply.github.com> Subject: Re: [Azure/sonic-buildimage] DB migration does not account mandatory entries in new init_cfg.json/FEATURE table in particular (#6659) This should have been fixed by sonic-net/sonic-utilities#1209<sonic-net/sonic-utilities#1209>. Need to include the commit in 201911. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#6659 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKTABAYFMYXFSUZBZ6HUPEDTB3EMNANCNFSM4XAZEQKQ>.

[liat-grozovik]

issue was fixed and is now also in 201911.