I am having couple of Cisco 4500 switches running in VSS mode with IOS-XE Version 03.04.01.SG RELEASE SOFTWARE (fc3)
I am facing high CPU on both Core CPU on the switch , i am not able to detemine the exact problem. Please help me to understand whats the exact issue causing High CPU. I also observed that my OSPF neighbours also go down when there is High CPU , might be Hellp packets are getting dropped.
Core 0: CPU utilization for five seconds: 8%; one minute: 14%; five minutes: 15%
Core 1: CPU utilization for five seconds: 95%; one minute: 85%; five minutes: 83%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
8623 2245873 28634543 382 51.88 49.67 49.28 0 iosd
11946 701202 10255390 68 0.12 0.18 0.18 0 ffm
11409 283519 1944165 145 0.04 0.04 0.04 0 cpumemd
1 1018 883 1153 0.00 0.00 0.00 0 init
2 1 91 11 0.00 0.00 0.00 0 kthreadd
3 136 22434 6 0.00 0.00 0.00 0 migration/0
4 65 3413 19 0.00 0.00 0.00 0 ksoftirqd/0
5 140 24075 5 0.00 0.00 0.00 0 migration/1
6 39 3447 11 0.00 0.00 0.00 0 ksoftirqd/1
7 11973 591682 20 0.00 0.00 0.00 0 events/0
8 13473 600343 22 0.00 0.00 0.00 0 events/1
9 7 790 9 0.00 0.00 0.00 0 khelper
92 31 1322 23 0.00 0.00 0.00 0 kblockd/0
93 6 387 17 0.00 0.00 0.00 0 kblockd/1
106 1 51 26 0.00 0.00 0.00 0 khubd
109 0 23 0 0.00 0.00 0.00 0 kseriod
114 7 25 282 0.00 0.00 0.00 0 kmmcd
156 953 78149 12 0.00 0.00 0.00 0 pdflush
157 1 176 10 0.00 0.00 0.00 0 pdflush
switch#show platform cpu packet statistics
Executing the command on VSS member switch role = VSS Active, id = 1
RkGenericPacketMan:
Packet allocation failures: 0
Packet Buffer(SW Common) allocation failures: 0
Packet Buffer(SW ESMP) allocation failures: 0
Packet Buffer(SW EOBC) allocation failures: 0
Packet Buffer(SW SupToSup) allocation failures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
349393 4 1 3 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss 347223 4 1 3 0
L2 Router 1993 0 0 0 0
Input ACl Copy 59 0 0 0 0
VS Mgmt 30 0 0 0 0
Vs Tunnel Control 64 0 0 0 0
Sw Packet w/ TxInfo 24 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 25975 0 0 0 0
Normal 58 0 0 0 0
Medium 323326 4 1 3 0
High 1 0 0 0 0
Crucial 33 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
NoPimPhyportMap 23967 0 0 0 0
Tx Mode Drop 323352 4 1 3 0
Qos Policer Drop 24 0 0 0 0
Extra packet received on VSS-Standby 1986 0 0 0 0
No Vsl port found 10 0 0 0 0
L2 Ctrl Packet received as SA miss 22 0 0 0 0
L3 Ctrl Packet copied to Standby 32 0 0 0 0
Total packet queues 64
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping) 762148 11 3 3 0
Vslp 392192 0 0 0 0
Host Learning 187415 0 0 0 0
L2 Control 18150729 62 49 41 31
Input ACL log, unreach 93448 0 0 0 0
L3 Glean 141578 0 0 0 0
Vs Mgmt 1416702 6 1 2 1
L3 Receive 72926075 769 925 536 307
Vs Tunnel Ctrl Pkt 40226 0 0 0 0
Ttl Expired 21124 0 0 0 0
Vs Tunnel Data Pkt 50166294 96 1511 855 917
InputIf Fail 15 0 0 0 0
Bfd 8 0 0 0 0
Adj SameIf Fail 153072 1 0 0 0
L2 router to CPU, 7 35061823 3733 1688 1757 1554
L3 Fwd 111064647 1603 2477 1487 816
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L3 Receive 423 0 0 0 0
Vs Tunnel Data Pkt 1346 0 0 0 0
Adj SameIf Fail 806 0 0 0 0
L2 router to CPU, 7 6128876 0 0 0 230
L3 Fwd 7292 0 0 0 0
Executing the command on VSS member switch role = VSS Standby, id = 2
RkGenericPacketMan:
Packet allocation failures: 0
Packet Buffer(SW Common) allocation failures: 0
Packet Buffer(SW ESMP) allocation failures: 0
Packet Buffer(SW EOBC) allocation failures: 0
Packet Buffer(SW SupToSup) allocation failures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
72483 0 0 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Vs Tunnel Control 72483 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------
%Timer expired.
switch#- --------- --------- --------- ----------
Unknown 32729 0 0 0 0
Medium 39754 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
L2 Ctrl Packet received as SA miss 32729 0 0 0 0
L3 Ctrl Packet copied to Standby 39754 0 0 0 0
Total packet queues 64
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Vslp 391856 0 0 0 0
Host Learning 267027 3 0 2 0
L2 Control 1 0 0 0 0
Input ACL log, unreach 39754 0 0 0 0
Vs Mgmt 945900 7 0 1 1
Vs Tunnel Ctrl Pkt 26712010 94 83 64 53
Ttl Expired 1232 0 0 0 0
Vs Tunnel Data Pkt 387375 1 0 0 0
Bfd 4 0 0 0 0
L2 router to CPU, 7 49659705 139 1576 834 890
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2 router to CPU, 7 3057300 6 11 14 126
