[YANG] Fix issue: Non compliant leaf list in config_db schema #10291

Junchao-Mellanox · 2022-03-21T01:47:04Z

Why I did it

Fix issue: Non compliant leaf list in config_db schema: #9801

How I did it

The basic flow of DPB is like:

Transfer config db json value to YANG json value, name it “yangIn”
Validate “yangIn” by libyang
Generate a YANG json value to represent the target configuration, name it “yangTarget”
Do diff between “yangIn” and “yangTarget”
Apply the diff to CONFIG DB json and save it back to DB

The fix:
• For step #1, If value of a leaf-list field string type, transfer it to a list by splitting it with “,” the purpose here is to make step#2 happy. We also need to save <table_name>..<field_name> to a set named “leaf_list_with_string_value_set”.
• For step#5, loop “leaf_list_with_string_value_set” and change those fields back to a string.

How to verify it

1.Manual test
2. Changed sample config DB and unit test passed

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Junchao-Mellanox · 2022-03-22T02:50:37Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-03-22T02:50:39Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-03-22T02:50:50Z

Azure Pipelines successfully started running 1 pipeline(s).

Junchao-Mellanox · 2022-03-24T02:26:12Z

Hi @qiluo-msft and @praveen-li , could you please review?

zhangyanzhao · 2022-03-24T17:14:41Z

@praveen-li would you please help to review this PR?

liat-grozovik · 2022-03-29T08:22:23Z

Hi @qiluo-msft and @praveen-li kindly reminder to review. As this is also needed for 202111 we need to urge you on that. sorry.

zhangyanzhao · 2022-03-31T17:14:09Z

@praveen-li would you please help to review and approve this PR? Thanks.

liat-grozovik · 2022-04-04T08:18:49Z

@praveen-li kindly reminder. this fix is must for 202111

praveen-li · 2022-04-04T16:15:56Z

Hi Liat I will surely review it today, Thanks, Regards Praveen

…

________________________________ From: liat-grozovik ***@***.***> Sent: Monday, April 4, 2022 1:18 AM To: Azure/sonic-buildimage ***@***.***> Cc: Praveen Chaudhary ***@***.***>; Mention ***@***.***> Subject: Re: [Azure/sonic-buildimage] [YANG] Fix issue: Non compliant leaf list in config_db schema (PR #10291) @praveen-li<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpraveen-li&data=04%7C01%7Cpchaudhary%40linkedin.com%7C89352bffb1ca4be2f4ad08da1613c821%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637846571482492274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=W4vTMlYHFZCbsL5exBZ%2BmXVpZWG4mnCyCKGzuFi05nw%3D&reserved=0> kindly reminder. this fix is must for 202111 — Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fsonic-buildimage%2Fpull%2F10291%23issuecomment-1087256055&data=04%7C01%7Cpchaudhary%40linkedin.com%7C89352bffb1ca4be2f4ad08da1613c821%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637846571482492274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=87AW5ds%2Fi5M3dbRfzWFJmvd2DUladThMp8CNCojcp5o%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJHZHPJCOWHZ6U6MCAPQZS3VDKQXHANCNFSM5RGLAFDQ&data=04%7C01%7Cpchaudhary%40linkedin.com%7C89352bffb1ca4be2f4ad08da1613c821%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637846571482492274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XdQNxUVLQlQRp8QhtrIIwt8al5y%2FKbwtWhhOF1CYlU4%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

src/sonic-yang-mgmt/sonic_yang_ext.py

Junchao-Mellanox · 2022-04-08T06:41:03Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-04-08T06:41:05Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-04-08T06:41:18Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2022-04-15T01:54:48Z

Azure Pipelines successfully started running 1 pipeline(s).

wen587 · 2022-04-15T10:08:35Z

Hi I saw testcase fail on GCU generic_config_updater/test_bgp_prefix.py
It seems the changes doesn't pass yang validation with below configuration.

admin@vlab-01:~/hijack/test$ cat add1.load
{
    "BGP_ALLOWED_PREFIXES": {
        "DEPLOYMENT_ID|0": {
            "prefixes_v4": [
                "10.20.0.0/16"
            ],
            "prefixes_v6": [
                "fc02:20::/64"
            ]
        }
    }
}

admin@vlab-01:~/hijack/test$ sudo config load -y add1.load
Running command: /usr/local/bin/sonic-cfggen -j add1.load --write-to-db

admin@vlab-01:~/hijack/test$ cd ~
admin@vlab-01:~$ cat verify_configDB_yang.py
import subprocess
import sonic_yang
import json
#import pytest
def get_config_db_json():
    cmd = "show runningconfiguration all"
    result = subprocess.Popen(cmd, shell=True, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    text, err = result.communicate()
    return json.loads(text)
configDbJson = get_config_db_json()
sy = sonic_yang.SonicYang("/usr/local/yang-models")
sy.loadYangModel()
sy.loadData(configDbJson )
sy.validate_data_tree()

admin@vlab-01:~$ python verify_configDB_yang.py
sonic_yang(6):Note: Below table(s) have no YANG models: CONSOLE_SWITCH, DEVICE_NEIGHBOR_METADATA, DHCP_SERVER, KDUMP, RESTAPI, SNMP, SNMP_COMMUNITY, TELEMETRY
sonic_yang(3):All Keys are not parsed in BGP_ALLOWED_PREFIXES
dict_keys(['DEPLOYMENT_ID|0'])
sonic_yang(3):exceptionList:['Value not found for deployment id community in DEPLOYMENT_ID|0', "'list' object has no attribute 'split'"]
sonic_yang(3):Data Loading Failed:All Keys are not parsed in BGP_ALLOWED_PREFIXES
dict_keys(['DEPLOYMENT_ID|0'])
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/sonic_yang_ext.py", line 1107, in loadData
    self._xlateConfigDB(xlateFile=xlateFile)
  File "/usr/local/lib/python3.9/dist-packages/sonic_yang_ext.py", line 727, in _xlateConfigDB
    self._xlateConfigDBtoYang(jIn, yangJ)
  File "/usr/local/lib/python3.9/dist-packages/sonic_yang_ext.py", line 713, in _xlateConfigDBtoYang
    self._xlateContainer(cmap['container'], yangJ[key][subkey], \
  File "/usr/local/lib/python3.9/dist-packages/sonic_yang_ext.py", line 692, in _xlateContainer
    raise(Exception("All Keys are not parsed in {}\n{}".format(table, \
Exception: All Keys are not parsed in BGP_ALLOWED_PREFIXES
dict_keys(['DEPLOYMENT_ID|0'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/admin/verify_configDB_yang.py", line 13, in <module>
    sy.loadData(configDbJson )
  File "/usr/local/lib/python3.9/dist-packages/sonic_yang_ext.py", line 1117, in loadData
    raise SonicYangException("Data Loading Failed\n{}".format(str(e)))
sonic_yang_ext.SonicYangException: Data Loading Failed
All Keys are not parsed in BGP_ALLOWED_PREFIXES
dict_keys(['DEPLOYMENT_ID|0'])

liat-grozovik · 2022-04-18T12:34:34Z

@praveen-li could you please help to review following comments provided?

liat-grozovik · 2022-04-22T08:10:42Z

@praveen-li kindly reminder. this must get in 2021111 so appreciate if we can close the loop. thanks.

praveen-li

Looks over all good to me.

praveen-li · 2022-04-27T23:04:15Z

src/sonic-yang-mgmt/sonic_yang_ext.py

@@ -407,6 +420,8 @@ def _yangConvert(val):
        # if it is a leaf-list do it for each element
        if leafDict[key]['__isleafList']:
            vValue = list()
+            if isinstance(value, str) and (self.elementPath[0], self.elementPath[-1]) in LEAF_LIST_WITH_STRING_VALUE_DICT:
+                value = (x.strip() for x in value.split(LEAF_LIST_WITH_STRING_VALUE_DICT[(self.elementPath[0], self.elementPath[-1])]))


plz explain with example this step. This has to break down the values from string to list. Please explain how it is working in this case. Thanks

zhangyanzhao · 2022-04-28T17:20:17Z

Once Praveen's comment is addressed, he will approve and merge.

Junchao-Mellanox · 2022-05-05T01:17:30Z

Hi @praveen-li , I have fixed the comments, could you please review and sign-off?

Junchao-Mellanox · 2022-05-05T01:17:54Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-05T01:17:56Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-05T01:18:08Z

Azure Pipelines successfully started running 1 pipeline(s).

praveen-li

LGTM, thanks for fixing it

zhangyanzhao · 2022-05-05T17:07:14Z

Qi will merge

Junchao-Mellanox · 2022-05-06T01:13:48Z

no clean cherry-pick, i will create separate PR for 202111.

…net#10291) Fix issue: Non compliant leaf list in config_db schema: sonic-net#9801 The basic flow of DPB is like: 1. Transfer config db json value to YANG json value, name it “yangIn” 2. Validate “yangIn” by libyang 3. Generate a YANG json value to represent the target configuration, name it “yangTarget” 4. Do diff between “yangIn” and “yangTarget” 5. Apply the diff to CONFIG DB json and save it back to DB The fix: • For step #1, If value of a leaf-list field string type, transfer it to a list by splitting it with “,” the purpose here is to make step#2 happy. We also need to save <table_name>.<key>.<field_name> to a set named “leaf_list_with_string_value_set”. • For step#5, loop “leaf_list_with_string_value_set” and change those fields back to a string. 1. Manual test 2. Changed sample config DB and unit test passed Conflicts: src/sonic-yang-mgmt/sonic_yang_ext.py

#10768) Fix issue: Non compliant leaf list in config_db schema: #9801 The basic flow of DPB is like: 1. Transfer config db json value to YANG json value, name it “yangIn” 2. Validate “yangIn” by libyang 3. Generate a YANG json value to represent the target configuration, name it “yangTarget” 4. Do diff between “yangIn” and “yangTarget” 5. Apply the diff to CONFIG DB json and save it back to DB The fix: • For step #1, If value of a leaf-list field string type, transfer it to a list by splitting it with “,” the purpose here is to make step#2 happy. We also need to save <table_name>.<key>.<field_name> to a set named “leaf_list_with_string_value_set”. • For step#5, loop “leaf_list_with_string_value_set” and change those fields back to a string. 1. Manual test 2. Changed sample config DB and unit test passed Conflicts: src/sonic-yang-mgmt/sonic_yang_ext.py

…net#10291) #### Why I did it Fix issue: Non compliant leaf list in config_db schema: sonic-net#9801 #### How I did it The basic flow of DPB is like: 1. Transfer config db json value to YANG json value, name it “yangIn” 2. Validate “yangIn” by libyang 3. Generate a YANG json value to represent the target configuration, name it “yangTarget” 4. Do diff between “yangIn” and “yangTarget” 5. Apply the diff to CONFIG DB json and save it back to DB The fix: • For step #1, If value of a leaf-list field string type, transfer it to a list by splitting it with “,” the purpose here is to make step#2 happy. We also need to save <table_name>.<key>.<field_name> to a set named “leaf_list_with_string_value_set”. • For step#5, loop “leaf_list_with_string_value_set” and change those fields back to a string. #### How to verify it 1. Manual test 2. Changed sample config DB and unit test passed

Junchao-Mellanox added 3 commits March 14, 2022 18:06

[YANG] Fix issue: Non compliant leaf list in config_db schema

9c91636

Fix issue found in test

aa7bca4

Fix review comment

59800d1

Junchao-Mellanox requested review from qiluo-msft and lguohan as code owners March 21, 2022 01:47

Junchao-Mellanox added Bug 🐛 Request for 202111 Branch For PRs being requested for 202111 branch labels Mar 21, 2022

Junchao-Mellanox requested a review from dgsudharsan March 21, 2022 02:24

dgsudharsan previously approved these changes Mar 21, 2022

View reviewed changes

dgsudharsan requested a review from praveen-li March 21, 2022 04:20

Junchao-Mellanox mentioned this pull request Mar 24, 2022

Rectify buffer_port_profile(ing/egr) yang files & related tests #10334

Closed

6 tasks

zhangyanzhao added the YANG YANG model related changes label Mar 24, 2022

praveen-li suggested changes Apr 5, 2022

View reviewed changes

Merge branch 'master' into fix-yang-leaf-list

9d89ccc

Junchao-Mellanox dismissed dgsudharsan’s stale review via 9d89ccc April 8, 2022 02:17

Junchao-Mellanox requested a review from praveen-li April 8, 2022 02:24

Fix vs test failure

f1c8883

praveen-li reviewed Apr 28, 2022

View reviewed changes

Fix review comment

40f3bf6

praveen-li approved these changes May 5, 2022

View reviewed changes

dgsudharsan approved these changes May 5, 2022

View reviewed changes

qiluo-msft approved these changes May 5, 2022

View reviewed changes

qiluo-msft merged commit f850122 into sonic-net:master May 5, 2022

Junchao-Mellanox deleted the fix-yang-leaf-list branch May 6, 2022 01:06

keboliu removed the Request for 202111 Branch For PRs being requested for 202111 branch label May 6, 2022

Junchao-Mellanox mentioned this pull request May 6, 2022

[202111] [YANG] Fix issue: Non compliant leaf list in config_db schema #10768

Merged

6 tasks

ghooo mentioned this pull request May 17, 2022

[GCU] Handle non-compliant leaf-list with string values sonic-net/sonic-utilities#2170

Closed

zhangyanzhao mentioned this pull request Sep 22, 2022

[YANG]sonic-dot1p-tc-map.yang would cause failed deployment via mgmt-framework #10386

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YANG] Fix issue: Non compliant leaf list in config_db schema #10291

[YANG] Fix issue: Non compliant leaf list in config_db schema #10291

Junchao-Mellanox commented Mar 21, 2022 •

edited

Loading

Junchao-Mellanox commented Mar 22, 2022

mssonicbld commented Mar 22, 2022

azure-pipelines bot commented Mar 22, 2022

Junchao-Mellanox commented Mar 24, 2022

zhangyanzhao commented Mar 24, 2022

liat-grozovik commented Mar 29, 2022

zhangyanzhao commented Mar 31, 2022

liat-grozovik commented Apr 4, 2022

praveen-li commented Apr 4, 2022 via email

Junchao-Mellanox commented Apr 8, 2022

mssonicbld commented Apr 8, 2022

azure-pipelines bot commented Apr 8, 2022

azure-pipelines bot commented Apr 15, 2022

wen587 commented Apr 15, 2022

liat-grozovik commented Apr 18, 2022

liat-grozovik commented Apr 22, 2022

praveen-li left a comment

praveen-li Apr 27, 2022

zhangyanzhao commented Apr 28, 2022

Junchao-Mellanox commented May 5, 2022

Junchao-Mellanox commented May 5, 2022

mssonicbld commented May 5, 2022

azure-pipelines bot commented May 5, 2022

praveen-li left a comment

zhangyanzhao commented May 5, 2022

Junchao-Mellanox commented May 6, 2022

[YANG] Fix issue: Non compliant leaf list in config_db schema #10291

[YANG] Fix issue: Non compliant leaf list in config_db schema #10291

Conversation

Junchao-Mellanox commented Mar 21, 2022 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Junchao-Mellanox commented Mar 22, 2022

mssonicbld commented Mar 22, 2022

azure-pipelines bot commented Mar 22, 2022

Junchao-Mellanox commented Mar 24, 2022

zhangyanzhao commented Mar 24, 2022

liat-grozovik commented Mar 29, 2022

zhangyanzhao commented Mar 31, 2022

liat-grozovik commented Apr 4, 2022

praveen-li commented Apr 4, 2022 via email

Junchao-Mellanox commented Apr 8, 2022

mssonicbld commented Apr 8, 2022

azure-pipelines bot commented Apr 8, 2022

azure-pipelines bot commented Apr 15, 2022

wen587 commented Apr 15, 2022

liat-grozovik commented Apr 18, 2022

liat-grozovik commented Apr 22, 2022

praveen-li left a comment

Choose a reason for hiding this comment

praveen-li Apr 27, 2022

Choose a reason for hiding this comment

zhangyanzhao commented Apr 28, 2022

Junchao-Mellanox commented May 5, 2022

Junchao-Mellanox commented May 5, 2022

mssonicbld commented May 5, 2022

azure-pipelines bot commented May 5, 2022

praveen-li left a comment

Choose a reason for hiding this comment

zhangyanzhao commented May 5, 2022

Junchao-Mellanox commented May 6, 2022

Junchao-Mellanox commented Mar 21, 2022 •

edited

Loading