SNMP discovery failing for new devices with different string

Mike Z · ‎12-15-2016

We have multiple devices in our environment (Cisco networking and Cisco UCS), which have two different SNMPv2c strings. The networking devices have an include list of 10.*.*.* 172.*.*.* and 192.168.*.*, while the UCS group is host IP-based (e.g. 10.50.20.60, 10.55.21.20, etc.). Further, I put the UCS IPs in the networking exclude list, just to rule out overlap. See screenshots attached.

But even with the more specific UCS SNMP at the top, I still get a "SNMP Unreachable or Incorrect SNMP Credentials" error during discovery. My Linux servers are able to poll all devices without issue, so I know SNMP is configured and working properly. We have no firewalls and no whitelisted SNMP host restrictions. My devices for testing are actually in the same subnet.

Why is discovery failing and how can I perform discovery on devices with different SNMP strings?

Dennis Mink · ‎12-15-2016

Can you debug snmp on the cisco device end and see if your cisco network components gets hit by snmp requests at all.

also have you got an source restriction on you network devices that can restrict incoming SNMP request from certain sources only?

your polling, is that icmp polling?

Please remember to rate useful posts, by clicking on the stars below.

CC Cisco · ‎12-16-2016

Are you able to successfully snmpwalk the devices from the collector's command line?

This can be accomplished from the adminshell (using admin user), by running the below command:

# sudo snmpwalk

In addition, if you enabled the collectorlogin user, it is much simpler.

$ snmpwalk

Regards,

Cesar

Mike Z · ‎12-19-2016

Just tried this and it timed out, but when I did a packet capture, it shows that it did work but just stopped after the first response:

admin# sudo snmpwalk -v 2c -c "<removed>" 10.55.21.20
Timeout: No Response from 10.55.21.20

admin# sudo tcpdump port 161
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:50:27.227424 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:27.261001 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
09:50:28.228728 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:28.262096 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
09:50:29.229930 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:29.263450 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
09:50:30.231134 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:30.264987 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
09:50:31.232333 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:31.266165 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
09:50:32.233553 IP totalcare01.domain.local.44887 > ucs01.domain.local.snmp: C=ro_snmp_string GetNextRequest(25)
09:50:32.266979 IP ucs01-b.domain.local.snmp > totalcare01.domain.local.44887: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"

12 packets captured
12 packets received by filter
0 packets dropped by kernel

CC Cisco · ‎12-19-2016

Hi Mike,

From the time stamps in your capture, it looks like even those couple requests took about 5 seconds. Which is kind of slow.

However, this might indicate that we can increase the timeout on the collector to wait long enough.

Can you please retry the snmpwalk, preferably from "collectorlogin's" user, then adding the -t flag. The -t flag allows you to specify a timeout.

$ snmpwalk -v 2c -c "<removed>" 10.55.21.20 -t xx

If this is successful, then modify the collector's discovery timeouts via: Settings -> Discovery Settings.

There are 3 timeout values to increase:

SNMP Timeout (in sec):

Max Discovery Time (in sec):

Max Discovery Time (in sec):

Regards,

Cesar

Mike Z · ‎12-19-2016

I don't think it's a timeout issue, the gap is in milliseconds... for example, the difference from 09:50:27.227424 to 09:50:27.261001 is only 0.033577 seconds = 33ms. But I tried changing timeout to 10 seconds, and same result (output truncated since message repeats multiple times):

[collectorlogin@nyptotalcare01 ~]$ snmpwalk -v 2c -c "<removed>" -t 10 10.55.21.20
Timeout: No Response from 10.55.21.20
[collectorlogin@nyptotalcare01 ~]$ ping 10.55.21.20
PING 10.55.21.20 (10.55.21.20) 56(84) bytes of data.
64 bytes from 10.55.21.20: icmp_seq=1 ttl=64 time=0.182 ms
64 bytes from 10.55.21.20: icmp_seq=2 ttl=64 time=0.172 ms
64 bytes from 10.55.21.20: icmp_seq=3 ttl=64 time=0.205 ms
64 bytes from 10.55.21.20: icmp_seq=4 ttl=64 time=0.187 ms
64 bytes from 10.55.21.20: icmp_seq=5 ttl=64 time=0.217 ms
64 bytes from 10.55.21.20: icmp_seq=6 ttl=64 time=0.180 ms
^C
--- 10.55.21.20 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5492ms
rtt min/avg/max/mdev = 0.172/0.190/0.217/0.020 ms

10:23:40.310018 IP totalcare01.domain.local.48967 > 10.55.21.20.snmp: C=ro_snmp_string GetNextRequest(25)
10:23:40.310623 IP 10.55.21.22.snmp > totalcare01.domain.local.48967: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
10:23:50.320184 IP totalcare01.domain.local.48967 > 10.55.21.20.snmp: C=ro_snmp_string GetNextRequest(25)
10:23:50.320806 IP 10.55.21.22.snmp > totalcare01.domain.local.48967: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"
10:24:00.329966 IP totalcare01.domain.local.48967 > 10.55.21.20.snmp: C=ro_snmp_string GetNextRequest(25)
10:24:00.430685 IP 10.55.21.22.snmp > totalcare01.domain.local.48967: C=ro_snmp_string GetResponse(201) system.sysDescr.0="Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00"

And for additional reference, here is the output from my Linux network management server:

[root@netmgmt01 ~]# snmpwalk -v2c -c '<removed>' 10.55.21.20
SNMPv2-MIB::sysDescr.0 = STRING: Cisco NX-OS(tm) ucs, Software (ucs-6100-k9-system), Version 5.0(3)N2(3.11k), RELEASE SOFTWARE Copyright (c) 2002-2013 by Cisco Systems, Inc. Compiled 8/28/2016 11:00:00
SNMPv2-MIB::sysObjectID.0 = OID: CISCO-SMI::ciscoModules.3.1.3.1062
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (385222233) 44 days, 14:03:42.33
SNMPv2-MIB::sysContact.0 = STRING: techemail@domain.com
SNMPv2-MIB::sysName.0 = STRING: UCS01-B
SNMPv2-MIB::sysLocation.0 = STRING: Colo
SNMPv2-MIB::sysServices.0 = INTEGER: 70
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (16) 0:00:00.16
...output truncated...