ISE 1.4 snmp polling causing high cpu on 6880 switch

[andrewswanson]

Hello

I'm running into an issue with 100% cpu on a Cisco 6880-X switch. The process causing high cpu is the SNMP Engine. The only device snmp polling the switch is Cisco ISE 1.4 patch 3 (3 PSN nodes). SNMP version used is v3 (auth/priv sha/aes) - I see the same issue with v2 but not with v1.

Issue occurs when using both available versions of 15.2:

• c6880x-ipservicesk9-mz.SPA.152-1.SY1.bin
• c6880x-ipservicesk9-mz.SPA.152-1.SY0a.bin

When cpu hits 100%, the 6880 doesn't respond to snmp queries. An output of "show snmp stats oid" shows the last mibs queried by ISE before SNMP crashes

20:47:38 BST Sep 17 2015         3                       lldpXMedLocMediaPolicyEntry.2
20:47:38 BST Sep 17 2015         3                       lldpRemEntry.4
20:47:38 BST Sep 17 2015         31                      ipNetToMediaEntry.3
20:47:38 BST Sep 17 2015         55                      ipNetToMediaEntry.2
20:47:38 BST Sep 17 2015         9                       ciscoImageEntry.2
20:47:38 BST Sep 17 2015         1                       cdpGlobal.5
20:47:38 BST Sep 17 2015         1                       cdpGlobal.4
20:47:38 BST Sep 17 2015         3                       cdpGlobal.3
20:47:38 BST Sep 17 2015         3                       cdpGlobal.2
20:47:38 BST Sep 17 2015         3                       cdpGlobal.1
20:47:38 BST Sep 17 2015         5                       cdpCacheEntry.24
20:47:38 BST Sep 17 2015         3                       cdpCacheEntry.12
20:47:38 BST Sep 17 2015         10                      cdpCacheEntry.11
20:47:38 BST Sep 17 2015         13                      cdpCacheEntry.10
20:47:38 BST Sep 17 2015         16                      cdpCacheEntry.9
20:47:38 BST Sep 17 2015         16                      cdpCacheEntry.8
20:47:38 BST Sep 17 2015         16                      cdpCacheEntry.7
20:47:38 BST Sep 17 2015         14                      cdpCacheEntry.6
20:47:38 BST Sep 17 2015         15                      cdpCacheEntry.5
20:47:38 BST Sep 17 2015         2                       cdpCacheEntry.4
20:47:38 BST Sep 17 2015         3                       cdpCacheEntry.3
20:47:38 BST Sep 17 2015         15                      ifPhysAddress
20:47:38 BST Sep 17 2015         3                       ipAddrEntry.2
20:47:38 BST Sep 17 2015         3                       ipAddrEntry.3
20:47:38 BST Sep 17 2015         3                       system.6
20:47:38 BST Sep 17 2015         3                       system.5
20:47:38 BST Sep 17 2015         3                       system.4
20:47:38 BST Sep 17 2015         130                     sysUpTime
20:47:38 BST Sep 17 2015         3                       system.2
20:47:38 BST Sep 17 2015         3                       system.1

I had lldp enabled on the 6880 but this is now disabled. I've restarted the SNMP process to see if this helps. The No. of times requested for each mib doesn't look high when the cpu hits 100% - has anyone else come across ISE causing SNMP/CPU issues?

Thanks
Andy

[andrewswanson]

Disabling lldp didn't resolve issue - 6880 switch at 100% cpu. Now running debug snmp packets after restarting snmp process

[andrewswanson]

The snmp debug showed something interesting. Originally, Prime Infrastructure 2.2 was managing the 6880 but I removed its IP Address from the snmp acl to rule it out of causing the high cpu/snmp issue.

The debug would show the snmp queuing a packet:

Sep 18 14:43:22.878: SW1: SNMP: Queuing packet to <IP-ADDRESS>
Sep 18 14:43:22.878: SW1: SNMP: V2 Trap, reqid 1762, errstat 0, erridx 0
 sysUpTime.0 = 16490046
 snmpTrapOID.0 = snmpTraps.5
 lsystem.5.0 = <PI-IP-ADDRESS>
 ciscoMgmt.412.1.1.1.0 = 1
 ciscoMgmt.412.1.1.2.0 = <PI-IP-ADDRESS>

A packet would be queued for the 3 PSN nodes and the PI node but the IP Address in the trap was always the IP Address of the PI Node (<PI-IP-ADDRESS>)

The 6880 would then send the traps to the 3 PSN nodes and the PI node and receive a packet back from the PI Node only:

Sep 18 14:43:23.126: SW1: SNMP: Packet sent via UDP to <PSN-1-IP-ADDRESS>
Sep 18 14:43:23.126: SW1: SNMP: Packet sent via UDP to <PSN-2-IP-ADDRESS>
Sep 18 14:43:23.130: SW1: SNMP: Packet sent via UDP to <PSN-3-IP-ADDRESS>
Sep 18 14:43:23.130: SW1: SNMP: Packet sent via UDP to <PI-IP-ADDRESS>
Sep 18 14:43:42.887: SW1: SNMP: Packet received via UDP from <PI-IP-ADDRESS> on Port-channel92

I looked through the 6880 config and found I had an "snmp-server host" entry for each of the 3 PSN and PI nodes:

snmp-server host <IP-ADDRESS> version 3 priv <SNMP-USER> auth-framework mac-notification snmp

I deleted the "snmp-server host" entry for PI and deleted the 6880 from PI. SNMP debug on the 6880 is now showing no traffic at all (switch is in development so I'd expect to see little snmp traffic with ISE).

CPU is looking fine so far  - looks like it wasn't an issue with ISE after all.

Cheers
Andy

[andrewswanson]

This didn't resolve 100% CPU issue. SNMP debug showed that 100% CPU occurred after the following request from ISE:

SNMP: Get-bulk request, reqid 1268581735, nonrptr 0, maxreps 10
  lldpRemEntry = NULL TYPE/VALUE

Applied the workaround fix for bug CSCtg62793  which solved the issue

[Tim Steele]

Good to know.  Thanks for posting your progress in resolving this.