Prime alarms device is unreachable via SNMP and ping(ICMP) is successful

[DOUG PENNER]

I periodically get alarms from Prime: "Device <device name> is unreachable via SNMP and ping(ICMP) is successful." It was only occasional, only one device, always when it is likely pretty busy and always cleared within 10 minutes so I didn't think much of it. Last night I got this for 14 devices at the same time. None of them were the typical one I usually get and all of them were small edge devices that wouldn't have been busy. Again, it cleared within 10 minutes. Is this likely to be a normal timeout due to SNMP requests being low priority? If so, will adjusting the retry or timeout settings in the system settings/snmp settings help to eliminate the alarms? The devices are clearly not down so the alarms are annoying. Or is this a symptom of the core being overloaded and dropping packets? There are no other alerts or symptoms such as latency ever noticed at the same time as these alerts.

[Thomas Hausruckinger]

Had this Problem too, yes you can try to increase SNMP Timeout in the System Settings, but as far as I remember the SNMP Timeout in the Credential Profile is the Way to go.

[DOUG PENNER]

We don't have a credential profile. We changed the system settings timeout from 30 to 40 but that didn't help. Then we found each device has their own snmp timeout defaulted to 3 so we increased that. I'll report the result.

[muath1987]

hello guys

I have the same problem, may I know if you solved it and how ?

[msmithgrh]

Has anyone in the thread come up with a resolution to this? In my case, it is all with 3850 switches running newer than 16.3 code. Never had the problem on switches running 16.3 or older, or anything in the 3.x line.

I have tried a bunch of different things without any change:

- tried moving to SNMP v3

- tried using an SNMP filter to reduce the number of items that were pulled

- tried converting switches in bundle mode to install mode

- recently updated from PI3.2 to 3.3

I do see the SNMP process ramp up and get lots of CPU time when these events happen. But, the events are not consistent in their timing or duration, so they are very hard to troubleshoot.

I have a case open right now, and will post back if I find anything useful.

[david.johnson]

Did anyone get a concrete resolution with this issue?

[msmithgrh]

I am still working a case with TAC, so no resolution yet for us. I does appear to be happening less now, after the upgrade to PI 3.3, but does still occur once or twice within 24 hours on at least one of my stacks.

[david.johnson]

I updated my Prime software to 3.1.7 about 3 hours ago and haven't had a single SNMP alert since then.
So either rebooting the server Prime is on or installing that software update fixed the issue for me because I was getting the SNMP alert every 5-10 minutes for the past week.

[david.johnson]

I'm not even joking, just a minute ago four switches threw the SNMP alert.  I perhaps jynxed myself :(

[msmithgrh]

As I mentioned earlier in the thread, I had a case open with TAC. We tried a lot of things to resolve this, but just recently found something that was effective. We ended up creating an SNMP view to filter out some of the OID's. I have many recommended entries in there already, but these are the two that finally removed the items that appear to have been causing the problem:

Object ipNetToPhysicalEntry

OID      1.3.6.1.2.1.4.35.1

Object cafSessionEntry

OID      1.3.6.1.4.1.9.9.656.1.4.1.1

Since I have added these to my SNMP view, I have not had a report of non-response from the SNMP process, and have not seen a high-cpu condition related to the SNMP service.

I am using SNMP v3, and my complete configuration looks like this:

snmp-server user prime admin v3 auth md5 <v3communityString> (this does not show up in the config)

snmp-server group admin v3 auth read cutdown
snmp-server view cutdown iso included
snmp-server view cutdown snmpUsmMIB excluded
snmp-server view cutdown snmpVacmMIB excluded
snmp-server view cutdown internet.6.3.17 excluded
snmp-server view cutdown snmpCommunityMIB excluded
snmp-server view cutdown ip.21 excluded
snmp-server view cutdown ip.22 excluded
snmp-server view cutdown ipNetToPhysicalEntry excluded
snmp-server view cutdown ciscoFlashMIB excluded
snmp-server view cutdown ciscoProcessMIB excluded
snmp-server view cutdown cafSessionEntry excluded

if you are using v2c, you need to add a normal community string referencing the view:

snmp-server community <v2ccommunitystring> view cutdown RO

HTH