[snmp_interfaces] DUT return nothing when using snmp_facts to get IF-MIB information

[okanchou9]

Description
DUT return nothing when using snmp_facts to get IF-MIB information.

Steps to reproduce the issue:

1. Loading a minigraph.xml on DUT, I'm using T1 topology(sample)

2. Run sump_interfaces script by CLI: ansible-playbook test_sonic.yml -i inventory --limit switch2 --become --tags snmp_interfaces -vvvvvv

3. Get nothing form DUT when using snmp_facts to get IF-MIB information

Describe the results you received:
TASK [test : Gathering basic snmp facts about the device] **********************
task path: /home/tester/SONiC/Azure/ansible/roles/test/tasks/snmp/interfaces.yml:2
Monday 20 November 2017 13:37:33 +0800 (0:00:00.202) 0:00:31.259 *******
ESTABLISH LOCAL CONNECTION FOR USER: tester
172.20.192.94 EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1511156253.51-280868843475900 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1511156253.51-280868843475900 )" )
172.20.192.94 PUT /tmp/tmpGurYa3 TO /home/tester/.ansible/tmp/ansible-tmp-1511156253.51-280868843475900/snmp_facts
172.20.192.94 EXEC /bin/sh -c 'sudo -H -S -p "[sudo via ansible, key=hjolswgfzuchclflcuhpozpbfjriuwkm] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-hjolswgfzuchclflcuhpozpbfjriuwkm; LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python /home/tester/.ansible/tmp/ansible-tmp-1511156253.51-280868843475900/snmp_facts; rm -rf "/home/tester/.ansible/tmp/ansible-tmp-1511156253.51-280868843475900/" > /dev/null 2>&1'"'"''
ok: [switch2] => {"ansible_facts": {"ansible_ChTotalFreeMemery": 6920820, "ansible_ChTotalMemery": 8165340, "ansible_all_ipv4_addresses": [], "ansible_syscontact": "Azure Cloud Switch vteam linuxnetdev@microsoft.com", "ansible_sysdescr": "SONiC Software Version: SONiC.SONiC-Ingrasys-v1.3.0-437419c - HwSku: INGRASYS-S9100-C32 - Distribution: Debian 8.9 - Kernel: 3.16.0-4-amd64", "ansible_syslocation": "public", "ansible_sysname": "switch2", "ansible_sysobjectid": "1.3.6.1.4.1.8072.3.2.10", "ansible_sysuptime": "959876"}, "changed": false, "invocation": {"module_args": {"authkey": null, "community": "public", "host": "172.20.192.94", "integrity": null, "is_dell": false, "is_ingrasys": false, "level": null, "privacy": null, "privkey": null, "removeplaceholder": null, "username": null, "version": "v2c"}, "module_name": "snmp_facts"}}

TASK [test : set_fact] *********************************************************
task path: /home/tester/SONiC/Azure/ansible/roles/test/tasks/snmp/interfaces.yml:6
Monday 20 November 2017 13:37:34 +0800 (0:00:00.856) 0:00:32.116 *******
ok: [switch2] => {"ansible_facts": {"mg_intf": [], "snmp_intf": []}, "changed": false, "invocation": {"module_args": {"mg_intf": [], "snmp_intf": []}, "module_name": "set_fact"}}

TASK [test : Create snmp interfaces list] **************************************
task path: /home/tester/SONiC/Azure/ansible/roles/test/tasks/snmp/interfaces.yml:10
Monday 20 November 2017 13:37:34 +0800 (0:00:00.106) 0:00:32.222 *******
fatal: [switch2]: FAILED! => {"failed": true, "msg": "ERROR! 'snmp_interfaces' is undefined"}

Describe the results you expected:
DUT should return correct IF-MIB information like sample

Additional information you deem important (e.g. issue happens only occasionally):

**Output of 'show version':**

root@switch2:/home/admin# show version
SONiC Software Version: SONiC.SONiC-Ingrasys-v1.3.0-437419c
Distribution: Debian 8.9
Kernel: 3.16.0-4-amd64
Build commit: 437419c
Build date: Fri Oct 27 09:02:25 UTC 2017
Built by: sonic@sonic

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm SONiC-Ingrasys-v1.3.0-437419c cdcee25d83a4 318.3 MB
docker-syncd-brcm latest cdcee25d83a4 318.3 MB
docker-orchagent-brcm SONiC-Ingrasys-v1.3.0-437419c fcb06af1a06d 259.1 MB
docker-orchagent-brcm latest fcb06af1a06d 259.1 MB
docker-lldp-sv2 SONiC-Ingrasys-v1.3.0-437419c 71c44d8b4d46 256.5 MB
docker-lldp-sv2 latest 71c44d8b4d46 256.5 MB
docker-dhcp-relay SONiC-Ingrasys-v1.3.0-437419c c5ea6ae37ceb 253.5 MB
docker-dhcp-relay latest c5ea6ae37ceb 253.5 MB
docker-database SONiC-Ingrasys-v1.3.0-437419c e38cab48359e 251.7 MB
docker-database latest e38cab48359e 251.7 MB
docker-snmp-sv2 SONiC-Ingrasys-v1.3.0-437419c 97e0ae330326 291.3 MB
docker-snmp-sv2 latest 97e0ae330326 291.3 MB
docker-teamd SONiC-Ingrasys-v1.3.0-437419c 44ef14751dc7 256.1 MB
docker-teamd latest 44ef14751dc7 256.1 MB
docker-platform-monitor SONiC-Ingrasys-v1.3.0-437419c af75bbc58bf1 271 MB
docker-platform-monitor latest af75bbc58bf1 271 MB
docker-fpm-quagga SONiC-Ingrasys-v1.3.0-437419c 3b724be6ed5f 262.7 MB
docker-fpm-quagga latest 3b724be6ed5f 262.7 MB

[okanchou9]

@maggiemsft

Would like to know did you face the same issue?

[kaiyu22]

I fount that the root cause seems to be the parameters of snmp process.
While snmp service running, it will block some OIDs as well (include IF-MIB).
Please refer the config file /etc/supervisor/conf.d/supervisord.conf in docker snmp

    [program:snmpd]
    command=/usr/sbin/snmpd -f -LS4d -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf,ifTable,ifXTable,inetCidrRouteTable,ipCidrRouteTable,ip,disk_hw -p /run/snmpd.pid
    priority=3
    autostart=false
    autorestart=false
    stdout_logfile=syslog
    stderr_logfile=syslog

Thus, this test won't be passed because the snmp won't retun data when user querying IF-MIB (has been blocked).
This situation is in the latest code of snmp in master.

HI @stcheng, @maggiemsft,
Do you have any comment?

[maggiemsft]

@kaiyu22 Thanks, will take a look

[qiluo-msft]

@kaiyu22 The root cause is not true. We purposely disable the implementation of several MIBs inside snmpd because of some issues with interface index, however we provide alternative (maybe partial) implementation in sonic-snmpagent. If you find IF-MIB is not working, the root cause may be outside the snmp docker.

[maggiemsft]

@kaiyu22
2nd what @qiluo-msft comments.
I checked my environment, didn't find this issue, it might be some other issues cause this failure.

[kaiyu22]

Hi @maggiemsft , @qiluo-msft
Thanks for your reply.

I have found that the key point is the PR #1094, it has fixed the snmp-subagent issue.
Before fixing, the error message will show in syslog and the error is due to "COUNTERS_DB".

Nov 29 08:10:14.687472 sonic INFO supervisord: snmp-subagent Traceback (most recent call last):
Nov 29 08:10:14.687472 sonic INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Nov 29 08:10:14.687653 sonic INFO supervisord: snmp-subagent     "__main__", mod_spec)
Nov 29 08:10:14.687653 sonic INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
Nov 29 08:10:14.687687 sonic INFO supervisord: snmp-subagent     exec(code, run_globals)
Nov 29 08:10:14.687808 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/__main__.py", line 74, in <module>
Nov 29 08:10:14.687808 sonic INFO supervisord: snmp-subagent     from .main import main
Nov 29 08:10:14.687842 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/main.py", line 14, in <module>
Nov 29 08:10:14.688048 sonic INFO supervisord: snmp-subagent     from .mibs.ietf import rfc1213, rfc2863, rfc4292, rfc4363
Nov 29 08:10:14.688183 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 337, in <module>
Nov 29 08:10:14.688183 sonic INFO supervisord: snmp-subagent     class InterfacesMIB(metaclass=MIBMeta, prefix='.1.3.6.1.2.1.2'):
Nov 29 08:10:14.688183 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 342, in InterfacesMIB
Nov 29 08:10:14.688208 sonic INFO supervisord: snmp-subagent     if_updater = InterfacesUpdater()
Nov 29 08:10:14.688232 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 157, in __init__
Nov 29 08:10:14.688439 sonic INFO supervisord: snmp-subagent     self.update_data()
Nov 29 08:10:14.688439 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 176, in update_data
Nov 29 08:10:14.688439 sonic INFO supervisord: snmp-subagent     for sai_id in self.if_id_map}
Nov 29 08:10:14.688439 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 176, in <dictcomp>
Nov 29 08:10:14.688469 sonic INFO supervisord: snmp-subagent     for sai_id in self.if_id_map}
Nov 29 08:10:14.688505 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 38, in wrapped
Nov 29 08:10:14.688505 sonic INFO supervisord: snmp-subagent     ret_data = f(inst, db_name, *args, **kwargs)
Nov 29 08:10:14.688691 sonic INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 303, in get_all
Nov 29 08:10:14.688691 sonic INFO supervisord: snmp-subagent     raise UnavailableDataError(message, _hash)
Nov 29 08:10:14.688733 sonic INFO supervisord: snmp-subagent swsssdk.exceptions.UnavailableDataError: Key 'b'COUNTERS:1000000000003'' unavailable in database 'COUNTERS_DB'

My sonic version is older than PR #1094, so the query of IF-MIB is failed.
After patching this PR, the issue is gone.

Thanks.

[okanchou9]

Close this issue since it had been fixed by PR #1094.