C4500 high CPU high utilization

[pvandyk2005]

->Every night, in between 10:00 and 1:00 CPU is going crazy(over 90%)

(device is in production)

device#show processes cpu sorted
CPU utilization for five seconds: 98%/0%; one minute: 98%; five minutes: 98%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  52  1804256212 504419180       3576 81.67% 81.14% 80.16%   0 Cat4k Mgmt LoPri
  51   5030666163610116487          0 15.35% 15.12% 15.43%   0 Cat4k Mgmt HiPri

->Trying to look forward for what can cause this high CPU, i found this

device#show platform health
      %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
<output omited>
K2L2 Address Table R   2.00  84.19     12      9  100  500  117 110   46  22203:33
<output omited>

device#show platform cpu packet statistics all | be Total packet queues 16
<output omited>
Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                        2898183726       171       162       131        118
L2/L3Control                 293980982        16        10        12          9
Host Learning               2369682928      4815      4401      3697       1808
L3 Fwd High                          0         0         0         0          0

->This results indicate to look towards CAM table, expecting to have a high number of unicast or multicast addresses. However, this is not the case

device#show mac address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count:                  187
Static Unicast Address (User-defined) Count:    0
Static Unicast Address (System-defined) Count:  32
Total Unicast MAC Addresses In Use:             219
Total Unicast MAC Addresses Available:          55000
Multicast MAC Address Count:                    24
Total Multicast MAC Addresses Available:        32768

->Is there someone who have seen something similar before?

Thank you!

->As a refference for this investigation i've used following link regarding "High CPU Utilization on Cisco IOS Software-Based Catalyst 4500 Switches"
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

[IAN WHITMORE]

Haven't had this problem before but I have had problems where due to nightly backups saturating the network we used to lose connectivity to many switches and servers until the backup was complete. Then the network stabalized again (I've been asking for a backup network for 5 years....).

So it could be that the network is flooded during backups?

Otherwise it could be something programmed on a server somewhere that causes flapping so the network goes crazy sending L2 frames back and forth. When whatever it was stops then the network settles down again. Without knowing your network setup etc...I was guess that is IS a backup of some server somewhere.

We also had problems on our Cisco network when we integrated with HP Blades (c-class and p-class) but all those issues got resolved once we updated to the latest HP firmware. Having said that the other day the network came down becasue a Windows serser on a blade was restarted!!?!?! Beat that! For some reason the Cisco switch and HP blade stopped negotiating the trunk and the HP sent all it's traffic over the native VLAN and this wreaked havoc on the network for things like spanning-tree.

So check your backups. Ask if you can NOT run them for one night and see what happens...a big ask in a production network...or simply change some server to a different time frame and see what happens (outside production timeframe though).

If you haven't got many server it shouldn't take long to find which one it is (assuming it is backups casing the problem)...we had over 900 servers

Check all your switches logs and look for port flapping. That might give you a clue.

Regards,

Ian

[Shashank Singh]

Hi,

Use the following inbuilt non impacting debug to find out the packets going to the CPU(menioned in the same high CPU troubleshooting document)

Switch#debug platform packet all receive buffer

platform packet debugging is on

Switch#show platform cpu packet buffered

Focus on finding any common traits in the packets like srcmac etc.Once we know the bulk of the traffic goingto the CPU the source can be shut off

to bring the CPU down.

Hope that helps,

Cheers,

Shashank

[pvandyk2005]

->this helped a lot . I found a source mac generating a high amount of packets

30 second input rate 7442000 bits/sec, 11539 packets/sec
  30 second output rate 500573000 bits/sec, 41574 packets/sec

->About the source i don't know nothing yet, however. I did found the source of this high CPU utilization

Thank you!

[Shashank Singh · Accepted solution]

Hi,

Glad that you found the information useful and found the source of high CPU. Please mark this post 'answered' if it helped you.

Regards,

Shashank

[ibrahimbadr4669]

what's the root cause of the high utilization