02-21-2011 11:38 PM - edited 03-06-2019 03:40 PM
->Every night, in between 10:00 and 1:00 CPU is going crazy(over 90%)
(device is in production)
device#show processes cpu sorted
CPU utilization for five seconds: 98%/0%; one minute: 98%; five minutes: 98%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
52 1804256212 504419180 3576 81.67% 81.14% 80.16% 0 Cat4k Mgmt LoPri
51 5030666163610116487 0 15.35% 15.12% 15.43% 0 Cat4k Mgmt HiPri
->Trying to look forward for what can cause this high CPU, i found this
device#show platform health
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
<output omited>
K2L2 Address Table R 2.00 84.19 12 9 100 500 117 110 46 22203:33
<output omited>
device#show platform cpu packet statistics all | be Total packet queues 16
<output omited>
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp 2898183726 171 162 131 118
L2/L3Control 293980982 16 10 12 9
Host Learning 2369682928 4815 4401 3697 1808
L3 Fwd High 0 0 0 0 0
->This results indicate to look towards CAM table, expecting to have a high number of unicast or multicast addresses. However, this is not the case
device#show mac address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count: 187
Static Unicast Address (User-defined) Count: 0
Static Unicast Address (System-defined) Count: 32
Total Unicast MAC Addresses In Use: 219
Total Unicast MAC Addresses Available: 55000
Multicast MAC Address Count: 24
Total Multicast MAC Addresses Available: 32768
->Is there someone who have seen something similar before?
Thank you!
->As a refference for this investigation i've used following link regarding "High CPU Utilization on Cisco IOS Software-Based Catalyst 4500 Switches"
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml
Solved! Go to Solution.
02-23-2011 12:38 AM
Hi,
Glad that you found the information useful and found the source of high CPU. Please mark this post 'answered' if it helped you.
Regards,
Shashank
02-22-2011 12:59 AM
Haven't had this problem before but I have had problems where due to nightly backups saturating the network we used to lose connectivity to many switches and servers until the backup was complete. Then the network stabalized again (I've been asking for a backup network for 5 years....).
So it could be that the network is flooded during backups?
Otherwise it could be something programmed on a server somewhere that causes flapping so the network goes crazy sending L2 frames back and forth. When whatever it was stops then the network settles down again. Without knowing your network setup etc...I was guess that is IS a backup of some server somewhere.
We also had problems on our Cisco network when we integrated with HP Blades (c-class and p-class) but all those issues got resolved once we updated to the latest HP firmware. Having said that the other day the network came down becasue a Windows serser on a blade was restarted!!?!?! Beat that! For some reason the Cisco switch and HP blade stopped negotiating the trunk and the HP sent all it's traffic over the native VLAN and this wreaked havoc on the network for things like spanning-tree.
So check your backups. Ask if you can NOT run them for one night and see what happens...a big ask in a production network...or simply change some server to a different time frame and see what happens (outside production timeframe though).
If you haven't got many server it shouldn't take long to find which one it is (assuming it is backups casing the problem)...we had over 900 servers
Check all your switches logs and look for port flapping. That might give you a clue.
Regards,
Ian
02-22-2011 01:02 AM
Hi,
Use the following inbuilt non impacting debug to find out the packets going to the CPU(menioned in the same high CPU troubleshooting document)
Switch#debug platform packet all receive buffer
platform packet debugging is on
Switch#show platform cpu packet buffered
Focus on finding any common traits in the packets like srcmac etc.Once we know the bulk of the traffic goingto the CPU the source can be shut off
to bring the CPU down.
Hope that helps,
Cheers,
Shashank
02-23-2011 12:04 AM
->this helped a lot . I found a source mac generating a high amount of packets
30 second input rate 7442000 bits/sec, 11539 packets/sec
30 second output rate 500573000 bits/sec, 41574 packets/sec
->About the source i don't know nothing yet, however. I did found the source of this high CPU utilization
Thank you!
02-23-2011 12:38 AM
Hi,
Glad that you found the information useful and found the source of high CPU. Please mark this post 'answered' if it helped you.
Regards,
Shashank
11-06-2020 03:31 AM
what's the root cause of the high utilization
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide