[debug dump util] Module implementation Logic and Port Module #1667

vivekrnv · 2021-06-11T18:55:20Z

Signed-off-by: Vivek Reddy Karri vkarri@nvidia.com

DEBUG DUMP UTIL PR SEQ: 2

NOTE: PLEASE MERGE #1666 BEFORE MERGING THIS

What I did

HLD for Dump Utility: HLD.
For More Info on adding new modules, Check this section in the HLD:
MatchInfra

How I did it

How to verify it

tests/dump_tests/module_tests/port_test.py::TestPortModule::test_all_args PASSED [ 86%]
tests/dump_tests/module_tests/port_test.py::TestPortModule::test_missing_asic_hostif PASSED [ 86%]
tests/dump_tests/module_tests/port_test.py::TestPortModule::test_missing_asic_port PASSED [ 86%]
tests/dump_tests/module_tests/port_test.py::TestPortModule::test_missing_state_and_appl PASSED [ 86%]
tests/dump_tests/module_tests/port_test.py::TestPortModule::test_no_port PASSED [ 86%]
tests/dump_tests/module_tests/port_test.py::TestPortModule::test_working_state PASSED [ 86%]

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

admin@sonic:~$ dump state port Ethernet0
{
    "Ethernet0": {
        "CONFIG_DB": {
            "keys": [
                {
                    "PORT|Ethernet0": {
                        "admin_status": "up",
                        "alias": "etp1",
                        "index": "1",
                        "lanes": "0",
                        "speed": "25000"
                    }
                }
            ],
            "tables_not_found": []
        },
        "APPL_DB": {
            "keys": [
                {
                    "PORT_TABLE:Ethernet0": {
                        "admin_status": "up",
                        "alias": "etp1",
                        "description": "",
                        "index": "1",
                        "lanes": "0",
                        "mtu": "9100",
                        "oper_status": "up",
                        "speed": "25000"
                    }
                }
            ],
            "tables_not_found": []
        },
        "ASIC_DB": {
            "keys": [
                {
                    "ASIC_STATE:SAI_OBJECT_TYPE_HOSTIF:oid:0xd000000000a21": {
                        "SAI_HOSTIF_ATTR_NAME": "Ethernet0",
                        "SAI_HOSTIF_ATTR_OBJ_ID": "oid:0x10000000008d4",
                        "SAI_HOSTIF_ATTR_OPER_STATUS": "true",
                        "SAI_HOSTIF_ATTR_TYPE": "SAI_HOSTIF_TYPE_NETDEV"
                    }
                },
                {
                    "ASIC_STATE:SAI_OBJECT_TYPE_PORT:oid:0x10000000008d4": {
                        "NULL": "NULL",
                        "SAI_PORT_ATTR_ADMIN_STATE": "true",
                        "SAI_PORT_ATTR_MTU": "9122",
                        "SAI_PORT_ATTR_SPEED": "25000"
                    }
                }
            ],
            "tables_not_found": [],
            "vidtorid": {
                "oid:0xd000000000a21": "oid:0xd",
                "oid:0x10000000008d4": "oid:0x1690000000001"
            }
        },
        "STATE_DB": {
            "keys": [
                {
                    "PORT_TABLE|Ethernet0": {
                        "netdev_oper_status": "up",
                        "state": "ok"
                    }
                }
            ],
            "tables_not_found": []
        }
    }
}

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

into module_imp

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

lgtm-com · 2021-06-11T19:05:13Z

This pull request introduces 1 alert when merging 6d76514 into a425ca2 - view on LGTM.com

new alerts:

1 for Unused import

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…o module_imp

liat-grozovik · 2021-06-27T13:55:03Z

/azp run

azure-pipelines · 2021-06-27T13:55:13Z

Azure Pipelines successfully started running 1 pipeline(s).

…o module_imp

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2021-07-16T17:51:23Z

/azp run

azure-pipelines · 2021-07-16T17:51:28Z

Commenter does not have sufficient privileges for PR 1667 in repo Azure/sonic-utilities

…o module_imp

liat-grozovik · 2021-07-16T18:09:12Z

/app run

vivekrnv · 2021-07-20T18:52:37Z

Request for 202106

dump/plugins/port.py

…o module_imp

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

arlakshm

As comments

dump/plugins/__init__.py

dump/plugins/executor.py

dump/plugins/port.py

tests/dump_input/port/appl_db.json

tests/dump_input/port/asic_db.json

tests/dump_input/port/state_db.json

arlakshm · 2021-08-13T22:14:53Z

tests/dump_tests/module_tests/mock_sonicv2connector.py

+from utilities_common.constants import DEFAULT_NAMESPACE
+
+class MockSonicV2Connector():
+    def __init__(self, dedicated_dbs, **kwargs):


Why we need to introduce a new class for mocking Sonicv2connector. Can we add new functionality to dbconnector.py

Testing Use case for these modules is to read from a separate json files as opposed to other tests which read from the default json files inside mock_tables directory.

I did see there is a dedicated_db list used inside one of the dbconnector methods

global dedicated_dbs if dedicated_dbs and dedicated_dbs.get(db_name): self.dbintf.redis_kwargs['db_name'] = dedicated_dbs[db_name] else: self.dbintf.redis_kwargs['db_name'] = db_name

The only test which uses this is db_migrator_test.py. However in that test, it was initializing the SonicV2Connector class directly inside the test, whereas in dump_module_tests it is being initialized in the methods defined in the actual classes.

And probably because of the complex redirection used in the dbconnector.py to mock the SonicV2Connector, i was not able to get the dump_module_tests tests running on custom json files. I've also worked with stephan who introduced that change but couldn't get it running.

Hence, decided to add a new mock class.

I found a couple of examples of using the dedicated_dbs recently added recently

https://github.com/Azure/sonic-utilities/blob/master/tests/config_int_ip_test.py
https://github.com/Azure/sonic-utilities/blob/5002745beb89a99a1cdf420f34a495c2c0cb31bd/tests/mpls_test.py#L60-L67

Can you check if we can follow a similar approach here?
I feel we should try and avoid duplicate mock classes if possible

Thanks for the info. I was able to remove the extra mock and use the default mock. I couldn't get it to directly read the custom json files, but i've handled it nevertheless.

Regarding the connection pooling optimization, i've recently observed when operating at scale, (i.e. route, had around 10k routes to parse for the dump utility) i've observed a significant amount of time spent in connect calls. With this optimization, the number of connect calls can be effectively reduced from (number_of_routes*number_of_calls_per_route) to 4 (number of db's the module is looking into) and thus offering a significant improvement in performance.
This design change infact helped me to use the default mock as well.

@SuvarnaMeenakshi and @arlakshm , please provide your comments on the changes.

tests/dump_tests/module_tests/mock_sonicv2connector.py

tests/dump_tests/module_tests/port_test.py

SuvarnaMeenakshi

As comments.

dump/plugins/port.py

tests/dump_input/port/appl_db.json

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

lgtm-com · 2021-08-19T11:23:45Z

This pull request introduces 3 alerts when merging 0082da2 into 29f4a16 - view on LGTM.com

new alerts:

2 for First parameter of a method is not named 'self'
1 for Unused import

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

SuvarnaMeenakshi

lgtm

dgsudharsan · 2021-08-25T17:57:47Z

@arlakshm Can you please sign off if the changes are good?

dgsudharsan · 2021-08-30T22:15:42Z

@SuvarnaMeenakshi Can we merge this?

What I did Implemented vlan and vlan_member modules for debug dump utility. How I did it Used infrastructure and followed examples in #1666 #1667 #1668 #1669 #1670 How to verify it On switch: dump state vlan <vlan_name> dump state vlan_member '<vlan_name|<member_name>' Unit test: pytest-3 dump_tests/module_tests/vlan_test.py (same test file covers both vlan and vlan_member)

vivekrnv added 12 commits May 14, 2021 05:10

Port Module and UT Added

cb06a3b

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Multi-Asic Changes Added

d236a7d

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Added TODO to mock

f5ffd27

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/vivekreddynv/sonic-utilities

cef2433

into module_imp

redis_match name updated

bfe107e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

copp_cfg is not required in this PR

e6607da

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Final Changes before review made

ee22710

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Comments Addressed

6d094f4

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Minor changes addressed

ffe6158

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Comments addressed

b750eba

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Changes made

b92290e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Moved mock files to diff dir

6d76514

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv mentioned this pull request Jun 11, 2021

[debug dump util] Base Skeleton and Click Class added #1668

Merged

LGTM issue addressed

244ff65

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv mentioned this pull request Jun 14, 2021

Debug dump utility HLD sonic-net/SONiC#789

Merged

anshuv-mfst requested a review from SuvarnaMeenakshi June 22, 2021 15:50

vivekrnv added 2 commits June 23, 2021 01:09

Multi-Asic Related Changes made

483cc52

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

826ded0

…o module_imp

SuvarnaMeenakshi requested a review from arlakshm July 13, 2021 19:22

vivekrnv added 2 commits July 14, 2021 20:11

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

0d42722

…o module_imp

Minor Fixes

63f9aa4

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

06b8297

…o module_imp

liat-grozovik added the Request for 202106 Branch label Jul 25, 2021

nazariig reviewed Jul 27, 2021

View reviewed changes

dump/plugins/port.py Outdated Show resolved Hide resolved

vivekrnv added 3 commits July 27, 2021 18:33

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

b8d2421

…o module_imp

Refactored and Added Comments

94eedcc

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Refactored the port_test

0db4c12

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

arlakshm reviewed Aug 13, 2021

View reviewed changes

SuvarnaMeenakshi reviewed Aug 13, 2021

View reviewed changes

dump/plugins/port.py Show resolved Hide resolved

tests/dump_input/port/appl_db.json Outdated Show resolved Hide resolved

vivekrnv added 5 commits August 14, 2021 10:01

pep-8 issues handled

8dac306

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Remaining pep8 fixes

dc43339

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

pep-8 fixes and introduced connection pool optimization

1eb3be8

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Minor changes owing to connection pool addition

663f46b

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Removed the entra mock and added a match_engine fixture

0082da2

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

LGTM issues handled

2ea3725

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

SuvarnaMeenakshi approved these changes Aug 23, 2021

View reviewed changes

nazariig approved these changes Aug 25, 2021

View reviewed changes

arlakshm approved these changes Aug 30, 2021

View reviewed changes

SuvarnaMeenakshi merged commit 720b650 into sonic-net:master Aug 30, 2021

liat-grozovik removed the Request for 202106 Branch label Sep 2, 2021

raphaelt-nvidia mentioned this pull request Oct 13, 2021

[debug dump util] Vlan Module Added #1877

Merged

SuvarnaMeenakshi mentioned this pull request Feb 28, 2022

[debug dump] dump interface module added #2070

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[debug dump util] Module implementation Logic and Port Module #1667

[debug dump util] Module implementation Logic and Port Module #1667

vivekrnv commented Jun 11, 2021 •

edited

Loading

lgtm-com bot commented Jun 11, 2021

liat-grozovik commented Jun 27, 2021

azure-pipelines bot commented Jun 27, 2021

vivekrnv commented Jul 16, 2021

azure-pipelines bot commented Jul 16, 2021

liat-grozovik commented Jul 16, 2021

vivekrnv commented Jul 20, 2021

arlakshm left a comment

arlakshm Aug 13, 2021

vivekrnv Aug 13, 2021 •

edited

Loading

arlakshm Aug 17, 2021

vivekrnv Aug 19, 2021

SuvarnaMeenakshi left a comment

lgtm-com bot commented Aug 19, 2021

SuvarnaMeenakshi left a comment

dgsudharsan commented Aug 25, 2021

dgsudharsan commented Aug 30, 2021

[debug dump util] Module implementation Logic and Port Module #1667

[debug dump util] Module implementation Logic and Port Module #1667

Conversation

vivekrnv commented Jun 11, 2021 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

lgtm-com bot commented Jun 11, 2021

liat-grozovik commented Jun 27, 2021

azure-pipelines bot commented Jun 27, 2021

vivekrnv commented Jul 16, 2021

azure-pipelines bot commented Jul 16, 2021

liat-grozovik commented Jul 16, 2021

vivekrnv commented Jul 20, 2021

arlakshm left a comment

Choose a reason for hiding this comment

arlakshm Aug 13, 2021

Choose a reason for hiding this comment

vivekrnv Aug 13, 2021 • edited Loading

Choose a reason for hiding this comment

arlakshm Aug 17, 2021

Choose a reason for hiding this comment

vivekrnv Aug 19, 2021

Choose a reason for hiding this comment

SuvarnaMeenakshi left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Aug 19, 2021

SuvarnaMeenakshi left a comment

Choose a reason for hiding this comment

dgsudharsan commented Aug 25, 2021

dgsudharsan commented Aug 30, 2021

vivekrnv commented Jun 11, 2021 •

edited

Loading

vivekrnv Aug 13, 2021 •

edited

Loading