Cat 6500 High CPU

[S891]

We experienced almost 95% to 98% cpu utilization on our pair of Core 6500 switches. 

The show proc cpu command was showing "IP input" process as the main reason and after that HSRP process was also high. 

I did not see any STP related issue. The switch log was showing continous flap of HSRP active to standby. 

The network almost became unresponsive and all connectivity through this pair of core 6500 was suffering. We had to shut down one of the switch and after that things started to normalize. 

Not sure what could be the issue or if there is a bug. We plan to bring up the switch at a quieter time and monitor it.

Can anyone share if they had similar experience and suggest troubleshooting steps for this problem?

Thanks

[Mark Malone]

Hi

can you collect the show proc cpu sorted when you bring the switch backup it will show which process maybe causing it

You could also run a EEM script if its intermittent which will collect the info and send it to flash to view when the issue occurs

As soon as cpu hits 70 % it will run and send a text file to flash , it may be like disk0 or 1 on your 6500 depending just do a dir and alter below if in a different location

 event manager applet CPU-HIGH
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3 get-type next entry-op gt entry-val 70 poll-interval 5
action 0.1 cli command “enable”
action 0.2 syslog msg “Capturing high cpu information to flash:”
action 1.0  cli command “term length 0”
action 1.1 cli command “show process cpu sorted | append flash:EEM_CPU_HIGH”
action 1.4 cli command “show ip traffic | append flash:EEM_CURR_TRAFFIC”
action 1.4 cli command “show logging | append flash:EEM_SYSLOG”
action 1.5  syslog msg “Removing EEM APPLET from running_config”
action 1.6 cli command “configure terminal”
action 1.7 cli command “no event manager applet CPU-HIGH”
action 1.8 cli command “end”
action 1.9 cli command “term default length”"

[jdesanti]

Hi,

IP input, means traffic being punted to the CPU to be processed, however high CPU utilization needs live tshooting to find out the root cause. 

If the CPU is running high due traffic, then you need to run a netdr capture.

Switch#debug netdr capture rx <<<<<<<<It is safe to run this debug. (no service impact)
Switch#sh netdr captured-packets<<<<<<<<<The output will be 4096 packets long.

Once you have the output, then you can copy/paste it on the NetDr parser to check the packets being punted.

http://netdr.54.227.241.219.xip.io/

Just keep in mind the capture needs to be done while the CPU is running high, otherwise the result will not be accurate.