Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bgp.test_bgp_suppress_fib.test_bgp_route_with_suppress failed with syncd errors #10272

Closed
siqbal1986 opened this issue Oct 9, 2023 · 6 comments
Closed

bgp.test_bgp_suppress_fib.test_bgp_route_with_suppress failed with syncd errors #10272

siqbal1986 opened this issue Oct 9, 2023 · 6 comments
Assignees
@roy-sror
Labels
20220531 issue

Comments

@siqbal1986
Copy link
Contributor

Description
The test bgp.test_bgp_suppress_fib][test_bgp_route_with_suppress is itself passing but this is causing some synd errors resulting in syslog messages on mlnx 4700 and 2700 platforms.

These failures are happening on 4700 and 2700 platforms on 20230531 builds.

Steps to reproduce the issue:

  1. Run this test with log analyzer enabled.

Describe the results you received:
The test keeps complaining that its not able to find some prefix.
Describe the results you expected:

Additional information you deem important:

**Output of `show version`:**

```
     "SONiC Software Version: SONiC.20230531.03",
        "SONiC OS Version: 11",
        "Distribution: Debian 11.7",
        "Kernel: 5.10.0-18-2-amd64",
        "Build commit: cedd805bac",
        "Build date: Mon Sep 11 14:22:53 UTC 2023",
        "Built by: cloudtest@fbe34d43c000000",
        "",
        "Platform: x86_64-mlnx_msn2700-r0",
        "HwSKU: Mellanox-SN2700",
        "ASIC: mellanox",
        "ASIC Count: 1",
        "Serial Number: MT1826X04469",
        "Model Number: MSN2700-CS2ROS",
        "Hardware Revision: A1",
        "Uptime: 02:27:14 up 57 min,  0 users,  load average: 3.51, 3.55, 4.22",
        "Date: Wed 13 Sep 2023 02:27:14",
        "",
        "Docker images:",
        "REPOSITORY                 TAG           IMAGE ID       SIZE",
        "docker-orchagent           20230531.03   1d924f58e89d   328MB",
        "docker-orchagent           latest        1d924f58e89d   328MB",
        "docker-fpm-frr             20230531.03   bfe6c97ab321   341MB",
        "docker-fpm-frr             latest        bfe6c97ab321   341MB",
        "docker-teamd               20230531.03   9819f90e6c54   316MB",
        "docker-teamd               latest        9819f90e6c54   316MB",
        "docker-macsec              latest        4ba204852071   319MB",
        "docker-syncd-mlnx          20230531.03   91881260b8c3   830MB",
        "docker-syncd-mlnx          latest        91881260b8c3   830MB",
        "docker-platform-monitor    20230531.03   fad7128b3f7c   820MB",
        "docker-platform-monitor    latest        fad7128b3f7c   820MB",
        "docker-dhcp-relay          latest        638db1e6c39f   308MB",
        "docker-snmp                20230531.03   0ca7e32ebf59   338MB",
        "docker-snmp                latest        0ca7e32ebf59   338MB",
        "docker-eventd              20230531.03   2e043392f0b8   299MB",
        "docker-eventd              latest        2e043392f0b8   299MB",
        "docker-router-advertiser   20230531.03   295aae2adfb7   299MB",
        "docker-router-advertiser   latest        295aae2adfb7   299MB",
        "docker-mux                 20230531.03   157d620ec986   347MB",
        "docker-mux                 latest        157d620ec986   347MB",
        "docker-lldp                20230531.03   e7fa39b0de85   341MB",
        "docker-lldp                latest        e7fa39b0de85   341MB",
        "docker-sonic-gnmi          20230531.03   2fba802b6d6b   599MB",
        "docker-sonic-gnmi          latest        2fba802b6d6b   599MB",
        "docker-database            20230531.03   201733d7b0f0   299MB",
        "docker-database            latest        201733d7b0f0   299MB",
        "docker-acms                20230531.03   2f409bac0e73   346MB",
        "docker-acms                latest        2f409bac0e73   346MB",
        "docker-vnet-monitor        20230531.03   19bbf941d696   322MB",
        "docker-vnet-monitor        latest        19bbf941d696   322MB",
        "docker-sonic-telemetry     20230531.03   d39af90003c7   599MB",
        "docker-sonic-telemetry     latest        d39af90003c7   599MB",

bgp/test_bgp_suppress_fib.py::test_bgp_route_with_suppress[default]
-------------------------------- live log call ---------------------------------
06:46:44 utilities.wait_until L0135 ERROR | Exception caught while checking check_route_status:Traceback (most recent call last):
File "/azp/_work/22/s/tests/common/utilities.py", line 131, in wait_until
check_result = condition(*args, **kwargs)
File "/azp/_work/22/s/tests/bgp/bgp_helpers.py", line 640, in check_route_status
check_field_status = out[route][0].get(check_field, None)
KeyError: '91.0.1.0/24'
, error:'91.0.1.0/24'
06:46:48 utilities.wait_until L0135 ERROR | Exception caught while checking check_route_status:Traceback (most recent call last):
File "/azp/_work/22/s/tests/common/utilities.py", line 131, in wait_until
check_result = condition(*args, **kwargs)
File "/azp/_work/22/s/tests/bgp/bgp_helpers.py", line 640, in check_route_status
check_field_status = out[route][0].get(check_field, None)
KeyError: '91.0.1.0/24'
, error:'91.0.1.0/24'
06:46:51 utilities.wait_until L0135 ERROR | Exception caught while checking check_route_status:Traceback (most recent call last):
File "/azp/_work/22/s/tests/common/utilities.py", line 131, in wait_until
check_result = condition(*args, **kwargs)
File "/azp/_work/22/s/tests/bgp/bgp_helpers.py", line 640, in check_route_status
check_field_status = out[route][0].get(check_field, None)
KeyError: '91.0.1.0/24'
, error:'91.0.1.0/24'
PASSED [ 20%]
bgp/test_bgp_suppress_fib.py::test_bgp_route_with_suppress[default] ERROR [ 20%]

2023-09-12T07:10:08.7718889Z E Failed: Processes "['analyze_logs--']" failed with exit code "1"
2023-09-12T07:10:08.7719994Z E Exception:
2023-09-12T07:10:08.7720804Z E match: 18
2023-09-12T07:10:08.7721640Z E expected_match: 0
2023-09-12T07:10:08.7722526Z E expected_missing_match: 0
2023-09-12T07:10:08.7723379Z E
2023-09-12T07:10:08.7724189Z E Match Messages:
2023-09-12T07:10:08.7725800Z E Sep 12 06:44:36.333228 str2-msn2700-spy-2 ERR syncd#SDK: [SX_API_INTERNAL.ERR] Failed command read at communication channel: Connection reset by peer
2023-09-12T07:10:08.7727143Z E
2023-09-12T07:10:08.7728608Z E Sep 12 06:44:47.370633 str2-msn2700-spy-2 ERR sonic-db-cli: :- getDbInfo: Failed to find CHASSIS_APP_DB database in namespace
2023-09-12T07:10:08.7730052Z E
2023-09-12T07:10:08.7731828Z E Sep 12 06:44:57.773009 str2-msn2700-spy-2 ERR eventd#eventd: :- deserialize: deserialize Failed: input stream errorstr[0:64]:(#1) data type: std::__cxx11::basic_string<char, std::char_traits, std::allocator >&
2023-09-12T07:10:08.7733422Z E
2023-09-12T07:10:08.7734932Z E Sep 12 06:45:34.905102 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_BRIDGE.ERR] ./src/mlnx_sai_bridge.c[1928]- mlnx_bridge_port_lag_or_port_get: Invalid port type - 2
2023-09-12T07:10:08.7736234Z E
2023-09-12T07:10:08.7737979Z E Sep 12 06:45:34.905260 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2013]- get_dispatch_attribs_handler: Failed Get #0, PORT_ID, key:BRIDGE_PORT [OID:0x2040000003A] [bridge_ports_db[516]]
2023-09-12T07:10:08.7739470Z E
2023-09-12T07:10:08.7740940Z E Sep 12 06:45:34.905385 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2190]- sai_get_attributes: Failed to get attribute
2023-09-12T07:10:08.7743122Z E
2023-09-12T07:10:08.7744898Z E Sep 12 06:45:34.905535 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_BRIDGE.ERR] ./src/mlnx_sai_bridge.c[2687]- mlnx_bridge_port_isolation_group_get: Isolation group is only supported for bridge port type port
2023-09-12T07:10:08.7746372Z E
2023-09-12T07:10:08.7748182Z E Sep 12 06:45:34.905651 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2013]- get_dispatch_attribs_handler: Failed Get #0, ISOLATION_GROUP, key:BRIDGE_PORT [OID:0x2040000003A] [bridge_ports_db[516]]
2023-09-12T07:10:08.7749748Z E
2023-09-12T07:10:08.7751286Z E Sep 12 06:45:34.905763 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2190]- sai_get_attributes: Failed to get attribute
2023-09-12T07:10:08.7752620Z E
2023-09-12T07:10:08.7754288Z E Sep 12 06:45:34.905893 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_BRIDGE.ERR] ./src/mlnx_sai_bridge.c[1479]- mlnx_bridge_1d_oid_to_data: Unexpected bridge type 0 is not 1D
2023-09-12T07:10:08.7755636Z E
2023-09-12T07:10:08.7757479Z E Sep 12 06:45:34.906011 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2013]- get_dispatch_attribs_handler: Failed Get #0, UNKNOWN_UNICAST_FLOOD_GROUP, key:BRIDGE [OID:0x10010039] [Type:.1Q, ID:0, sx_bridge_id:4097]
2023-09-12T07:10:08.7759045Z E
2023-09-12T07:10:08.7760495Z E Sep 12 06:45:34.906125 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2190]- sai_get_attributes: Failed to get attribute
2023-09-12T07:10:08.7761782Z E
2023-09-12T07:10:08.7763324Z E Sep 12 06:45:34.906253 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_BRIDGE.ERR] ./src/mlnx_sai_bridge.c[1479]- mlnx_bridge_1d_oid_to_data: Unexpected bridge type 0 is not 1D
2023-09-12T07:10:08.7764665Z E
2023-09-12T07:10:08.7766574Z E Sep 12 06:45:34.906366 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2013]- get_dispatch_attribs_handler: Failed Get #0, UNKNOWN_MULTICAST_FLOOD_GROUP, key:BRIDGE [OID:0x10010039] [Type:.1Q, ID:0, sx_bridge_id:4097]
2023-09-12T07:10:08.7768189Z E
2023-09-12T07:10:08.7769661Z E Sep 12 06:45:34.906476 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2190]- sai_get_attributes: Failed to get attribute
2023-09-12T07:10:08.7770939Z E
2023-09-12T07:10:08.7772498Z E Sep 12 06:45:34.906602 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_BRIDGE.ERR] ./src/mlnx_sai_bridge.c[1479]- mlnx_bridge_1d_oid_to_data: Unexpected bridge type 0 is not 1D
2023-09-12T07:10:08.7773831Z E
2023-09-12T07:10:08.7775657Z E Sep 12 06:45:34.906714 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2013]- get_dispatch_attribs_handler: Failed Get #0, BROADCAST_FLOOD_GROUP, key:BRIDGE [OID:0x10010039] [Type:.1Q, ID:0, sx_bridge_id:4097]
2023-09-12T07:10:08.7777356Z E
2023-09-12T07:10:08.7778800Z E Sep 12 06:45:34.906822 str2-msn2700-spy-2 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2190]- sai_get_attributes: Failed to get attribute
2023-09-12T07:10:08.7780061Z E
2023-09-12T07:10:08.7780863Z E Traceback:
2023-09-12T07:10:08.7782522Z E Traceback (most recent call last):
2023-09-12T07:10:08.7783691Z E File "/azp/_work/22/s/tests/common/helpers/parallel.py", line 35, in run
2023-09-12T07:10:08.7784761Z E Process.run(self)
2023-09-12T07:10:08.7785821Z E File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
2023-09-12T07:10:08.7786971Z E self._target(*self._args, **self._kwargs)
2023-09-12T07:10:08.7788182Z E File "/azp/_work/22/s/tests/common/helpers/parallel.py", line 245, in wrapper
2023-09-12T07:10:08.7789286Z E target(*args, **kwargs)
2023-09-12T07:10:08.7790551Z E File "/azp/_work/22/s/tests/common/plugins/loganalyzer/init.py", line 39, in analyze_logs
2023-09-12T07:10:08.7791757Z E dut_analyzer.analyze(markers[node.hostname])
2023-09-12T07:10:08.7793117Z E File "/azp/_work/22/s/tests/common/plugins/loganalyzer/loganalyzer.py", line 386, in analyze
2023-09-12T07:10:08.7794252Z E self._verify_log(analyzer_summary)
2023-09-12T07:10:08.7795447Z E File "/azp/_work/22/s/tests/common/plugins/loganalyzer/loganalyzer.py", line 139, in _verify_log
2023-09-12T07:10:08.7796598Z E raise LogAnalyzerError(result_str)
2023-09-12T07:10:08.7797737Z E tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzerError: match: 18
2023-09-12T07:10:08.7798789Z E expected_match: 0
2023-09-12T07:10:08.7799655Z E expected_missing_match: 0
```
@bingwang-ms
Copy link
Collaborator

@roy-sror Can you pelase help take a look?

@roy-sror
Copy link
Contributor

Yes, will do.

@lipxu
Copy link
Contributor

lipxu commented Oct 16, 2023

Hi, @roy-sror , May you please share the latest update and ETA, thank you very much!

@roy-sror
Copy link
Contributor

Hi folks, this is a duplicate of sonic-net/sonic-buildimage#7895

Do you see any failures other than the error logs?

If no, then I will ensure that the errors will be skipped and this ticket will be closed.

@lipxu
Copy link
Contributor

lipxu commented Oct 19, 2023

Hi, @roy-sror , thanks for looking at this issue, the case failed on different platform devices,

below is the failure log on Arista platform, the case passed but failed with the below error message.

May you please help to take a look at it
whether it is the same issue, Do we need to ignore these error messages, thanks a lot.

bgp/test_bgp_suppress_fib.py::test_bgp_route_with_suppress[default] PASSED [ 20%]
bgp/test_bgp_suppress_fib.py::test_bgp_route_with_suppress[default] ERROR [ 20%]

ERR eventd#eventd: :- deserialize: deserialize Failed: input stream errorstr[0:64]:(#1) data type: std::__cxx11::basic_string<char, std::char_traits, std::allocator >&

ERR swss#portsyncd: :- readData: netlink reports an error=-33 on reading a netlink socket

ERR syncd#syncd: [none] SAI_API_SWITCH:sai_query_attribute_capability:486 Switch Attr capabilities failed with error -2.

ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:441 Invalid sai_api_t xx passed to sai_api_query

ERR syncd#syncd: [none] SAI_API_QUEUE:_brcm_sai_cosq_stat_get:1661 Bulk read not supported for this counter.

ERR syncd#syncd: [none] SAI_API_SWITCH:sai_bulk_object_get_stats:666 get bulk queue stats failed with error -2.

@roy-sror roy-sror mentioned this issue Oct 19, 2023
Merged
7 tasks
@roy-sror
Copy link
Contributor

@lipxu - I've added skip for the issues I am aware of. The remaining are new to me, and I believe that they should be reported either as community sonic-buildimage bugs or reported to the relevant vendor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roy-sror roy-sror

Labels
20220531 issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@siqbal1986 @bingwang-ms @roy-sror @lipxu