12-10-2010 10:34 AM - edited 03-06-2019 02:28 PM
I have couple of 4500 switches which are running high CPU. Though i know it is process Cat4k Mgmt LoPri which is eating up CPU cycles, i am unable to determine what exactly is the root cause and how to fix it.
Below are the commands for your reference. How do i see which packets are being sent to CPU and what interface is sending them and why.
Cat4500-sw1#sh proc cpu sorted 5min | ex 0.00% 0.00% 0.00%
CPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes: 91%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
49 3924333802675976363 0 95.66% 95.51% 85.66% 0 Cat4k Mgmt LoPri
48 1496541140 550408613 2718 2.39% 2.59% 3.71% 0 Cat4k Mgmt HiPri
104 5036533161297436277 388 1.16% 1.04% 1.05% 0 Spanning Tree
37 239957700 77642786 3090 0.23% 0.24% 0.23% 0 IDB Work
7 72783040 10613546 6857 0.00% 0.10% 0.11% 0 Check heaps
195 45645708 295308027 154 0.07% 0.07% 0.07% 0 SNMP ENGINE
193 47040588 668827756 70 0.07% 0.05% 0.05% 0 IP SNMP
93 10969624 126007784 87 0.07% 0.03% 0.05% 0 CDP Protocol
100 9995124 313542997 31 0.07% 0.03% 0.00% 0 IP Input
94 108 392 275 0.07% 0.02% 0.00% 1 SSH Process
164 4732892 655867157 7 0.07% 0.01% 0.00% 0 PM Callback
55 3418728 17432919 196 0.00% 0.01% 0.00% 0 Compute load avg
41 21243520 1559662 13620 0.00% 0.01% 0.00% 0 Per-minute Jobs
79 3919768 101059994 38 0.00% 0.01% 0.00% 0 DTP Protocol
Cat4500-its-sw1#sh platform cpu packet statistics al
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
------------ --------------- --------------- --------------- ---------------
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 4 TxQueue 5 TxQueue 6 TxQueue 7
------------ --------------- --------------- --------------- ---------------
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0
Packets Enqueued Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
0 0 0 0 0
Packets Not Enqueued Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
0 0 0 0 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
25285077341 5 2 3 6
Packets Not Enqueued by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 0 0 0 0 0
Sa Miss 0 0 0 0 0
PVMapping Miss 0 0 0 0 0
Nfe 0 0 0 0 0
CTS 0 0 0 0 0
MTU Check Fail 0 0 0 0 0
ESMP 0 0 0 0 0
L2 Control 0 0 0 0 0
Adj Same If 0 0 0 0 0
Non Arpa 0 0 0 0 0
Expired Ttl 0 0 0 0 0
Ip Option 0 0 0 0 0
Input If Fail 0 0 0 0 0
Ucast Rpf Fail 0 0 0 0 0
SrcIdx Check Fail 0 0 0 0 0
Ipv6 Scope Fail 0 0 0 0 0
L2 Bridge 0 0 0 0 0
L2 Router 0 0 0 0 0
L3 Glean 0 0 0 0 0
L3 Forward 0 0 0 0 0
L3 Receive 0 0 0 0 0
Input Acl Fwd 0 0 0 0 0
Input ACl Copy 0 0 0 0 0
Input Acl Punt 0 0 0 0 0
Input Acl Err 0 0 0 0 0
Output Acl Fwd 0 0 0 0 0
Output Acl Copy 0 0 0 0 0
Output Acl Punt 0 0 0 0 0
Output Acl Err 0 0 0 0 0
Sw Packet w/ TxInfo 0 0 0 0 0
Sw Packet for Bridge 0 0 0 0 0
Sw Cpu Span 0 0 0 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 0 0 0 0 0
Sa Miss 24702605024 3 1 2 0
PVMapping Miss 0 0 0 0 0
Nfe 0 0 0 0 0
CTS 0 0 0 0 0
MTU Check Fail 0 0 0 0 0
ESMP 0 0 0 0 0
L2 Control 0 0 0 0 0
Adj Same If 0 0 0 0 0
Non Arpa 0 0 0 0 0
Expired Ttl 0 0 0 0 0
Ip Option 0 0 0 0 0
Input If Fail 0 0 0 0 0
Ucast Rpf Fail 0 0 0 0 0
SrcIdx Check Fail 0 0 0 0 0
Ipv6 Scope Fail 0 0 0 0 0
L2 Bridge 0 0 0 0 0
L2 Router 0 0 0 0 0
L3 Glean 0 0 0 0 0
L3 Forward 0 0 0 0 0
L3 Receive 0 0 0 0 0
Input Acl Fwd 582472317 2 0 0 2
Input ACl Copy 0 0 0 0 0
Input Acl Punt 0 0 0 0 0
Input Acl Err 0 0 0 0 0
Output Acl Fwd 0 0 0 0 0
Output Acl Copy 0 0 0 0 0
Output Acl Punt 0 0 0 0 0
Output Acl Err 0 0 0 0 0
Sw Packet w/ TxInfo 0 0 0 0 0
Sw Packet for Bridge 0 0 0 0 0
Sw Cpu Span 0 0 0 0 0
Packets Not Enqueued by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 0 0 0 0 0
Normal 0 0 0 0 0
Medium 0 0 0 0 0
High 0 0 0 0 0
Crucial 0 0 0 0 0
Super Crucial 0 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 0 0 0 0 0
Normal 319883415 0 0 0 0
Medium 24702605024 3 1 2 0
High 262588902 2 0 0 1
Crucial 0 0 0 0 0
Super Crucial 0 0 0 0 0
Packets Not Enqueued by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------- --------------------- --------- --------- --------- ----------
RxQueueNotAvail 0 0 0 0 0
RxNoBuffersAvail 0 0 0 0 0
RxBadCpuEvent 0 0 0 0 0
Testing 0 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
NoPimPhyportMap 0 0 0 0 0
NoPimPhyport 0 0 0 0 0
UnimplementedEvt 0 0 0 0 0
InvalidEvt 0 0 0 0 0
SrcAddrTableFilt 0 0 0 0 0
NoL2Vlan 0 0 0 0 0
STPDrop 153 0 0 0 0
UnknownLyr-Bridge 0 0 0 0 0
UnknownLyr-OutAcl 0 0 0 0 0
L2DstDrop 0 0 0 0 0
L2DstDropOutAcl 0 0 0 0 0
AclNoVlan 0 0 0 0 0
AclActionDrop 0 0 0 0 0
AclActionRedirect 0 0 0 0 0
AclActionUnsupptd 0 0 0 0 0
UnknownBridgeAct 0 0 0 0 0
NoTxInfo 0 0 0 0 0
NoDstPorts 582472185 2 0 0 2
NoFloodPorts 0 0 0 0 0
ControlNoL2DstEnt 0 0 0 0 0
InvalidAclAction 0 0 0 0 0
AclInputErr 0 0 0 0 0
OutAclEvntOnInAcl 0 0 0 0 0
AclOutputErr 0 0 0 0 0
ReservedPort 0 0 0 0 0
EsmpNoTxInfo 0 0 0 0 0
UnknownSource 0 0 0 0 0
NoDriverTxPacketInfo 0 0 0 0 0
NoPimPort 0 0 0 0 0
RouteLookupFailed 0 0 0 0 0
FragPktAllocFailed 0 0 0 0 0
Dont Fragment Drop 0 0 0 0 0
IpHdrException 0 0 0 0 0
SpanEvent 0 0 0 0 0
Dot1xUnauthCtrlPkt 0 0 0 0 0
No Sw port for ARP 0 0 0 0 0
No Sw port for ESMP 0 0 0 0 0
Tx Mode Drop 24702605003 3 1 2 0
Invalid TxInfo For Replica 0 0 0 0 0
Qos Policer Drop 0 0 0 0 0
Bad Crc 0 0 0 0 0
Invalid EVC packet 0 0 0 0 0
Unknown EVC packet 0 0 0 0 0
Invalid Vlan packet 0 0 0 0 0
Efp not present for Tx packet 0 0 0 0 0
No space available for EVC shim hdr 0 0 0 0 0
Total packet queues 64
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Mtu Fail 0 0 0 0 0
Esmp 0 0 0 0 0
Input ACL fwd(snooping) 0 0 0 0 0
L2 bridge to CPU, 0 0 0 0 0 0
L2 router to CPU, 0 0 0 0 0 0
L3 Glean, 0 0 0 0 0 0
L3 Fwd, 0 0 0 0 0 0
L3 Receive, 0 0 0 0 0 0
Host Learning 0 0 0 0 0
L2 Control 0 0 0 0 0
Input ACL log, unreach 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 1 0 0 0 0 0
L3 Glean, 1 0 0 0 0 0
L3 Fwd, 1 0 0 0 0 0
L3 Receive, 1 0 0 0 0 0
Unknown 0 0 0 0 0
Ip Option 0 0 0 0 0
Input ACL sw processing 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 2 0 0 0 0 0
L3 Glean, 2 0 0 0 0 0
L3 Fwd, 2 0 0 0 0 0
L3 Receive, 2 0 0 0 0 0
PV Mapping Miss 0 0 0 0 0
Ttl Expired 0 0 0 0 0
Input ACL Error 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 3 0 0 0 0 0
L3 Glean, 3 0 0 0 0 0
L3 Fwd, 3 0 0 0 0 0
L3 Receive, 3 0 0 0 0 0
Unknown 0 0 0 0 0
Non Arpa 0 0 0 0 0
Output ACL fwd(snooping) 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 4 0 0 0 0 0
L3 Glean, 4 0 0 0 0 0
L3 Fwd, 4 0 0 0 0 0
L3 Receive, 4 0 0 0 0 0
InputIf Fail 0 0 0 0 0
UcastRpfFail 0 0 0 0 0
Output ACL log, unreach 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 5 0 0 0 0 0
L3 Glean, 5 0 0 0 0 0
L3 Fwd, 5 0 0 0 0 0
L3 Receive, 5 0 0 0 0 0
Ipv6 Scope Check Fail 0 0 0 0 0
SrcIndex Check Fail 0 0 0 0 0
Output ACL sw processing 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 6 0 0 0 0 0
L3 Glean, 6 0 0 0 0 0
L3 Fwd, 6 0 0 0 0 0
L3 Receive, 6 0 0 0 0 0
Unknown 0 0 0 0 0
Adj SameIf Fail 0 0 0 0 0
Output ACL Error 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 7 0 0 0 0 0
L3 Glean, 7 0 0 0 0 0
L3 Fwd, 7 0 0 0 0 0
L3 Receive, 7 0 0 0 0 0
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Mtu Fail 0 0 0 0 0
Esmp 0 0 0 0 0
Input ACL fwd(snooping) 0 0 0 0 0
L2 bridge to CPU, 0 0 0 0 0 0
L2 router to CPU, 0 0 0 0 0 0
L3 Glean, 0 0 0 0 0 0
L3 Fwd, 0 0 0 0 0 0
L3 Receive, 0 0 0 0 0 0
Host Learning 0 0 0 0 0
L2 Control 0 0 0 0 0
Input ACL log, unreach 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 1 0 0 0 0 0
L3 Glean, 1 0 0 0 0 0
L3 Fwd, 1 0 0 0 0 0
L3 Receive, 1 0 0 0 0 0
Unknown 0 0 0 0 0
Ip Option 0 0 0 0 0
Input ACL sw processing 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 2 0 0 0 0 0
L3 Glean, 2 0 0 0 0 0
L3 Fwd, 2 0 0 0 0 0
L3 Receive, 2 0 0 0 0 0
PV Mapping Miss 0 0 0 0 0
Ttl Expired 0 0 0 0 0
Input ACL Error 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 3 0 0 0 0 0
L3 Glean, 3 0 0 0 0 0
L3 Fwd, 3 0 0 0 0 0
L3 Receive, 3 0 0 0 0 0
Unknown 0 0 0 0 0
Non Arpa 0 0 0 0 0
Output ACL fwd(snooping) 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 4 0 0 0 0 0
L3 Glean, 4 0 0 0 0 0
L3 Fwd, 4 0 0 0 0 0
L3 Receive, 4 0 0 0 0 0
InputIf Fail 0 0 0 0 0
UcastRpfFail 0 0 0 0 0
Output ACL log, unreach 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 5 0 0 0 0 0
L3 Glean, 5 0 0 0 0 0
L3 Fwd, 5 0 0 0 0 0
L3 Receive, 5 0 0 0 0 0
Ipv6 Scope Check Fail 0 0 0 0 0
SrcIndex Check Fail 0 0 0 0 0
Output ACL sw processing 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 6 0 0 0 0 0
L3 Glean, 6 0 0 0 0 0
L3 Fwd, 6 0 0 0 0 0
L3 Receive, 6 0 0 0 0 0
Unknown 0 0 0 0 0
Adj SameIf Fail 0 0 0 0 0
Output ACL Error 0 0 0 0 0
Unknown 0 0 0 0 0
L2 router to CPU, 7 0 0 0 0 0
L3 Glean, 7 0 0 0 0 0
L3 Fwd, 7 0 0 0 0 0
L3 Receive, 7 0 0 0 0 0
Packets Transmitted from CPU per Output Interface
Interface Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Gi1/1 0 0 0 0 0
Gi1/2 0 0 0 0 0
Gi1/3 0 0 0 0 0
Gi1/4 0 0 0 0 0
Gi1/5 0 0 0 0 0
Gi1/6 0 0 0 0 0
Gi1/7 0 0 0 0 0
Gi1/8 0 0 0 0 0
Gi1/9 0 0 0 0 0
Gi1/10 0 0 0 0 0
Gi1/11 0 0 0 0 0
Gi1/12 0 0 0 0 0
Gi1/13 0 0 0 0 0
Gi1/14 0 0 0 0 0
Gi1/15 0 0 0 0 0
Gi1/16 0 0 0 0 0
Gi1/17 0 0 0 0 0
Gi1/18 0 0 0 0 0
Gi1/19 0 0 0 0 0
Gi1/20 0 0 0 0 0
Gi1/21 0 0 0 0 0
Gi1/22 0 0 0 0 0
Gi1/23 0 0 0 0 0
Gi1/24 0 0 0 0 0
Gi1/25 0 0 0 0 0
Gi1/26 0 0 0 0 0
Gi1/27 0 0 0 0 0
Gi1/28 0 0 0 0 0
Gi1/29 0 0 0 0 0
Gi1/30 0 0 0 0 0
Gi1/31 0 0 0 0 0
Gi1/32 0 0 0 0 0
Gi1/33 0 0 0 0 0
Gi1/34 0 0 0 0 0
Gi1/35 0 0 0 0 0
Gi1/36 0 0 0 0 0
Gi1/37 0 0 0 0 0
Gi1/38 0 0 0 0 0
Gi1/39 0 0 0 0 0
Gi1/40 0 0 0 0 0
Gi1/41 0 0 0 0 0
Gi1/42 0 0 0 0 0
Gi1/43 0 0 0 0 0
Gi1/44 0 0 0 0 0
Gi1/45 0 0 0 0 0
Gi1/46 0 0 0 0 0
Gi1/47 0 0 0 0 0
Gi1/48 0 0 0 0 0
Gi2/1 0 0 0 0 0
Gi2/2 0 0 0 0 0
Gi2/3 0 0 0 0 0
Gi2/4 0 0 0 0 0
Gi2/5 0 0 0 0 0
Gi2/6 0 0 0 0 0
Gi2/7 0 0 0 0 0
Gi2/8 0 0 0 0 0
Gi2/9 0 0 0 0 0
Gi2/10 0 0 0 0 0
Gi2/11 0 0 0 0 0
Gi2/12 0 0 0 0 0
Gi2/13 0 0 0 0 0
Gi2/14 0 0 0 0 0
Gi2/15 0 0 0 0 0
Gi2/16 0 0 0 0 0
Gi2/17 0 0 0 0 0
Gi2/18 0 0 0 0 0
Gi2/19 0 0 0 0 0
Gi2/20 0 0 0 0 0
Gi2/21 0 0 0 0 0
Gi2/22 0 0 0 0 0
Gi2/23 0 0 0 0 0
Gi2/24 0 0 0 0 0
Gi2/25 0 0 0 0 0
Gi2/26 0 0 0 0 0
Gi2/27 0 0 0 0 0
Gi2/28 0 0 0 0 0
Gi2/29 0 0 0 0 0
Gi2/30 0 0 0 0 0
Gi2/31 0 0 0 0 0
Gi2/32 0 0 0 0 0
Gi2/33 0 0 0 0 0
Gi2/34 0 0 0 0 0
Gi2/35 0 0 0 0 0
Gi2/36 0 0 0 0 0
Gi2/37 0 0 0 0 0
Gi2/38 0 0 0 0 0
Gi2/39 0 0 0 0 0
Gi2/40 0 0 0 0 0
Gi2/41 0 0 0 0 0
Gi2/42 0 0 0 0 0
Gi2/43 0 0 0 0 0
Gi2/44 0 0 0 0 0
Gi2/45 0 0 0 0 0
Gi2/46 0 0 0 0 0
Gi2/47 0 0 0 0 0
Gi2/48 0 0 0 0 0
Gi5/1 0 0 0 0 0
Gi5/2 0 0 0 0 0
Gi5/3 0 0 0 0 0
Gi5/4 0 0 0 0 0
Gi5/5 0 0 0 0 0
Gi5/6 0 0 0 0 0
Gi5/7 0 0 0 0 0
Gi5/8 0 0 0 0 0
Gi5/9 0 0 0 0 0
Gi5/10 0 0 0 0 0
Gi5/11 0 0 0 0 0
Gi5/12 0 0 0 0 0
Gi5/13 0 0 0 0 0
Gi5/14 0 0 0 0 0
Gi5/15 0 0 0 0 0
Gi5/16 0 0 0 0 0
Gi5/17 0 0 0 0 0
Gi5/18 0 0 0 0 0
Gi5/19 0 0 0 0 0
Gi5/20 0 0 0 0 0
Gi5/21 0 0 0 0 0
Gi5/22 0 0 0 0 0
Gi5/23 0 0 0 0 0
Gi5/24 0 0 0 0 0
Gi5/25 0 0 0 0 0
Gi5/26 0 0 0 0 0
Gi5/27 0 0 0 0 0
Gi5/28 0 0 0 0 0
Gi5/29 0 0 0 0 0
Gi5/30 0 0 0 0 0
Gi5/31 0 0 0 0 0
Gi5/32 0 0 0 0 0
Gi5/33 0 0 0 0 0
Gi5/34 0 0 0 0 0
Gi5/35 0 0 0 0 0
Gi5/36 0 0 0 0 0
Gi5/37 0 0 0 0 0
Gi5/38 0 0 0 0 0
Gi5/39 0 0 0 0 0
Gi5/40 0 0 0 0 0
Gi5/41 0 0 0 0 0
Gi5/42 0 0 0 0 0
Gi5/43 0 0 0 0 0
Gi5/44 0 0 0 0 0
Gi5/45 0 0 0 0 0
Gi5/46 0 0 0 0 0
Gi5/47 0 0 0 0 0
Gi5/48 0 0 0 0 0
Gi6/1 0 0 0 0 0
Gi6/2 0 0 0 0 0
Gi6/3 0 0 0 0 0
Gi6/4 0 0 0 0 0
Gi6/5 0 0 0 0 0
Gi6/6 0 0 0 0 0
Gi6/7 0 0 0 0 0
Gi6/8 0 0 0 0 0
Gi6/9 0 0 0 0 0
Gi6/10 0 0 0 0 0
Gi6/11 0 0 0 0 0
Gi6/12 0 0 0 0 0
Gi6/13 0 0 0 0 0
Gi6/14 0 0 0 0 0
Gi6/15 0 0 0 0 0
Gi6/16 0 0 0 0 0
Gi6/17 0 0 0 0 0
Gi6/18 0 0 0 0 0
Gi6/19 0 0 0 0 0
Gi6/20 0 0 0 0 0
Gi6/21 0 0 0 0 0
Gi6/22 0 0 0 0 0
Gi6/23 0 0 0 0 0
Gi6/24 0 0 0 0 0
Gi6/25 0 0 0 0 0
Gi6/26 0 0 0 0 0
Gi6/27 0 0 0 0 0
Gi6/28 0 0 0 0 0
Gi6/29 0 0 0 0 0
Gi6/30 0 0 0 0 0
Gi6/31 0 0 0 0 0
Gi6/32 0 0 0 0 0
Gi6/33 0 0 0 0 0
Gi6/34 0 0 0 0 0
Gi6/35 0 0 0 0 0
Gi6/36 0 0 0 0 0
Gi6/37 0 0 0 0 0
Gi6/38 0 0 0 0 0
Gi6/39 0 0 0 0 0
Gi6/40 0 0 0 0 0
Gi6/41 0 0 0 0 0
Gi6/42 0 0 0 0 0
Gi6/43 0 0 0 0 0
Gi6/44 0 0 0 0 0
Gi6/45 0 0 0 0 0
Gi6/46 0 0 0 0 0
Gi6/47 0 0 0 0 0
Gi6/48 0 0 0 0 0
Gi7/1 0 0 0 0 0
Gi7/2 0 0 0 0 0
Gi7/3 0 0 0 0 0
Gi7/4 0 0 0 0 0
Gi7/5 0 0 0 0 0
Gi7/6 0 0 0 0 0
Gi7/7 0 0 0 0 0
Gi7/8 0 0 0 0 0
Gi7/9 0 0 0 0 0
Gi7/10 0 0 0 0 0
Gi7/11 0 0 0 0 0
Gi7/12 0 0 0 0 0
Gi7/13 0 0 0 0 0
Gi7/14 0 0 0 0 0
Gi7/15 0 0 0 0 0
Gi7/16 0 0 0 0 0
Gi7/17 0 0 0 0 0
Gi7/18 0 0 0 0 0
Gi7/19 0 0 0 0 0
Gi7/20 0 0 0 0 0
Gi7/21 0 0 0 0 0
Gi7/22 0 0 0 0 0
Gi7/23 0 0 0 0 0
Gi7/24 0 0 0 0 0
Gi7/25 0 0 0 0 0
Gi7/26 0 0 0 0 0
Gi7/27 0 0 0 0 0
Gi7/28 0 0 0 0 0
Gi7/29 0 0 0 0 0
Gi7/30 0 0 0 0 0
Gi7/31 0 0 0 0 0
Gi7/32 0 0 0 0 0
Gi7/33 0 0 0 0 0
Gi7/34 0 0 0 0 0
Gi7/35 0 0 0 0 0
Gi7/36 0 0 0 0 0
Gi7/37 0 0 0 0 0
Gi7/38 0 0 0 0 0
Gi7/39 0 0 0 0 0
Gi7/40 0 0 0 0 0
Gi7/41 0 0 0 0 0
Gi7/42 0 0 0 0 0
Gi7/43 0 0 0 0 0
Gi7/44 0 0 0 0 0
Gi7/45 0 0 0 0 0
Gi7/46 0 0 0 0 0
Gi7/47 0 0 0 0 0
Gi7/48 0 0 0 0 0
Te3/1 0 0 0 0 0
Te3/2 0 0 0 0 0
Gi3/3 0 0 0 0 0
Gi3/4 0 0 0 0 0
Gi3/5 0 0 0 0 0
Gi3/6 0 0 0 0 0
Te4/1 0 0 0 0 0
Te4/2 0 0 0 0 0
Gi4/3 0 0 0 0 0
Gi4/4 0 0 0 0 0
Gi4/5 0 0 0 0 0
Gi4/6 0 0 0 0 0
Packets Received at CPU per Input Interface
Interface Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Gi1/1 0 0 0 0 0
Gi1/2 0 0 0 0 0
Gi1/3 0 0 0 0 0
Gi1/4 0 0 0 0 0
Gi1/5 0 0 0 0 0
Gi1/6 0 0 0 0 0
Gi1/7 0 0 0 0 0
Gi1/8 0 0 0 0 0
Gi1/9 0 0 0 0 0
Gi1/10 0 0 0 0 0
Gi1/11 0 0 0 0 0
Gi1/12 0 0 0 0 0
Gi1/13 0 0 0 0 0
Gi1/14 0 0 0 0 0
Gi1/15 0 0 0 0 0
Gi1/16 0 0 0 0 0
Gi1/17 0 0 0 0 0
Gi1/18 0 0 0 0 0
Gi1/19 0 0 0 0 0
Gi1/20 0 0 0 0 0
Gi1/21 0 0 0 0 0
Gi1/22 0 0 0 0 0
Gi1/23 0 0 0 0 0
Gi1/24 0 0 0 0 0
Gi1/25 0 0 0 0 0
Gi1/26 0 0 0 0 0
Gi1/27 0 0 0 0 0
Gi1/28 0 0 0 0 0
Gi1/29 0 0 0 0 0
Gi1/30 0 0 0 0 0
Gi1/31 0 0 0 0 0
Gi1/32 0 0 0 0 0
Gi1/33 0 0 0 0 0
Gi1/34 0 0 0 0 0
Gi1/35 0 0 0 0 0
Gi1/36 0 0 0 0 0
Gi1/37 0 0 0 0 0
Gi1/38 0 0 0 0 0
Gi1/39 0 0 0 0 0
Gi1/40 0 0 0 0 0
Gi1/41 0 0 0 0 0
Gi1/42 0 0 0 0 0
Gi1/43 0 0 0 0 0
Gi1/44 0 0 0 0 0
Gi1/45 0 0 0 0 0
Gi1/46 0 0 0 0 0
Gi1/47 0 0 0 0 0
Gi1/48 0 0 0 0 0
Gi2/1 0 0 0 0 0
Gi2/2 0 0 0 0 0
Gi2/3 0 0 0 0 0
Gi2/4 0 0 0 0 0
Gi2/5 0 0 0 0 0
Gi2/6 0 0 0 0 0
Gi2/7 0 0 0 0 0
Gi2/8 0 0 0 0 0
Gi2/9 0 0 0 0 0
Gi2/10 0 0 0 0 0
Gi2/11 0 0 0 0 0
Gi2/12 0 0 0 0 0
Gi2/13 0 0 0 0 0
Gi2/14 0 0 0 0 0
Gi2/15 0 0 0 0 0
Gi2/16 0 0 0 0 0
Gi2/17 0 0 0 0 0
Gi2/18 0 0 0 0 0
Gi2/19 0 0 0 0 0
Gi2/20 0 0 0 0 0
Gi2/21 0 0 0 0 0
Gi2/22 0 0 0 0 0
Gi2/23 0 0 0 0 0
Gi2/24 0 0 0 0 0
Gi2/25 0 0 0 0 0
Gi2/26 0 0 0 0 0
Gi2/27 0 0 0 0 0
Gi2/28 0 0 0 0 0
Gi2/29 0 0 0 0 0
Gi2/30 0 0 0 0 0
Gi2/31 0 0 0 0 0
Gi2/32 0 0 0 0 0
Gi2/33 0 0 0 0 0
Gi2/34 0 0 0 0 0
Gi2/35 0 0 0 0 0
Gi2/36 0 0 0 0 0
Gi2/37 0 0 0 0 0
Gi2/38 0 0 0 0 0
Gi2/39 0 0 0 0 0
Gi2/40 0 0 0 0 0
Gi2/41 0 0 0 0 0
Gi2/42 0 0 0 0 0
Gi2/43 0 0 0 0 0
Gi2/44 0 0 0 0 0
Gi2/45 0 0 0 0 0
Gi2/46 0 0 0 0 0
Gi2/47 0 0 0 0 0
Gi2/48 0 0 0 0 0
Gi5/1 0 0 0 0 0
Gi5/2 0 0 0 0 0
Gi5/3 0 0 0 0 0
Gi5/4 0 0 0 0 0
Gi5/5 0 0 0 0 0
Gi5/6 0 0 0 0 0
Gi5/7 0 0 0 0 0
Gi5/8 0 0 0 0 0
Gi5/9 0 0 0 0 0
Gi5/10 0 0 0 0 0
Gi5/11 0 0 0 0 0
Gi5/12 0 0 0 0 0
Gi5/13 0 0 0 0 0
Gi5/14 0 0 0 0 0
Gi5/15 0 0 0 0 0
Gi5/16 0 0 0 0 0
Gi5/17 0 0 0 0 0
Gi5/18 0 0 0 0 0
Gi5/19 0 0 0 0 0
Gi5/20 0 0 0 0 0
Gi5/21 0 0 0 0 0
Gi5/22 0 0 0 0 0
Gi5/23 0 0 0 0 0
Gi5/24 0 0 0 0 0
Gi5/25 0 0 0 0 0
Gi5/26 0 0 0 0 0
Gi5/27 0 0 0 0 0
Gi5/28 0 0 0 0 0
Gi5/29 0 0 0 0 0
Gi5/30 0 0 0 0 0
Gi5/31 0 0 0 0 0
Gi5/32 0 0 0 0 0
Gi5/33 0 0 0 0 0
Gi5/34 0 0 0 0 0
Gi5/35 0 0 0 0 0
Gi5/36 0 0 0 0 0
Gi5/37 0 0 0 0 0
Gi5/38 0 0 0 0 0
Gi5/39 0 0 0 0 0
Gi5/40 0 0 0 0 0
Gi5/41 0 0 0 0 0
Gi5/42 0 0 0 0 0
Gi5/43 0 0 0 0 0
Gi5/44 0 0 0 0 0
Gi5/45 0 0 0 0 0
Gi5/46 0 0 0 0 0
Gi5/47 0 0 0 0 0
Gi5/48 0 0 0 0 0
Gi6/1 0 0 0 0 0
Gi6/2 0 0 0 0 0
Gi6/3 0 0 0 0 0
Gi6/4 0 0 0 0 0
Gi6/5 0 0 0 0 0
Gi6/6 0 0 0 0 0
Gi6/7 0 0 0 0 0
Gi6/8 0 0 0 0 0
Gi6/9 0 0 0 0 0
Gi6/10 0 0 0 0 0
Gi6/11 0 0 0 0 0
Gi6/12 0 0 0 0 0
Gi6/13 0 0 0 0 0
Gi6/14 0 0 0 0 0
Gi6/15 0 0 0 0 0
Gi6/16 0 0 0 0 0
Gi6/17 0 0 0 0 0
Gi6/18 0 0 0 0 0
Gi6/19 0 0 0 0 0
Gi6/20 0 0 0 0 0
Gi6/21 0 0 0 0 0
Gi6/22 0 0 0 0 0
Gi6/23 0 0 0 0 0
Gi6/24 0 0 0 0 0
Gi6/25 0 0 0 0 0
Gi6/26 0 0 0 0 0
Gi6/27 0 0 0 0 0
Gi6/28 0 0 0 0 0
Gi6/29 0 0 0 0 0
Gi6/30 0 0 0 0 0
Gi6/31 0 0 0 0 0
Gi6/32 0 0 0 0 0
Gi6/33 0 0 0 0 0
Gi6/34 0 0 0 0 0
Gi6/35 0 0 0 0 0
Gi6/36 0 0 0 0 0
Gi6/37 0 0 0 0 0
Gi6/38 0 0 0 0 0
Gi6/39 0 0 0 0 0
Gi6/40 0 0 0 0 0
Gi6/41 0 0 0 0 0
Gi6/42 0 0 0 0 0
Gi6/43 0 0 0 0 0
Gi6/44 0 0 0 0 0
Gi6/45 0 0 0 0 0
Gi6/46 0 0 0 0 0
Gi6/47 0 0 0 0 0
Gi6/48 0 0 0 0 0
Gi7/1 0 0 0 0 0
Gi7/2 0 0 0 0 0
Gi7/3 0 0 0 0 0
Gi7/4 0 0 0 0 0
Gi7/5 0 0 0 0 0
Gi7/6 0 0 0 0 0
Gi7/7 0 0 0 0 0
Gi7/8 0 0 0 0 0
Gi7/9 0 0 0 0 0
Gi7/10 0 0 0 0 0
Gi7/11 0 0 0 0 0
Gi7/12 0 0 0 0 0
Gi7/13 0 0 0 0 0
Gi7/14 0 0 0 0 0
Gi7/15 0 0 0 0 0
Gi7/16 0 0 0 0 0
Gi7/17 0 0 0 0 0
Gi7/18 0 0 0 0 0
Gi7/19 0 0 0 0 0
Gi7/20 0 0 0 0 0
Gi7/21 0 0 0 0 0
Gi7/22 0 0 0 0 0
Gi7/23 0 0 0 0 0
Gi7/24 0 0 0 0 0
Gi7/25 0 0 0 0 0
Gi7/26 0 0 0 0 0
Gi7/27 0 0 0 0 0
Gi7/28 0 0 0 0 0
Gi7/29 0 0 0 0 0
Gi7/30 0 0 0 0 0
Gi7/31 0 0 0 0 0
Gi7/32 0 0 0 0 0
Gi7/33 0 0 0 0 0
Gi7/34 0 0 0 0 0
Gi7/35 0 0 0 0 0
Gi7/36 0 0 0 0 0
Gi7/37 0 0 0 0 0
Gi7/38 0 0 0 0 0
Gi7/39 0 0 0 0 0
Gi7/40 0 0 0 0 0
Gi7/41 0 0 0 0 0
Gi7/42 0 0 0 0 0
Gi7/43 0 0 0 0 0
Gi7/44 0 0 0 0 0
Gi7/45 0 0 0 0 0
Gi7/46 0 0 0 0 0
Gi7/47 0 0 0 0 0
Gi7/48 0 0 0 0 0
Te3/1 0 0 0 0 0
Te3/2 0 0 0 0 0
Gi3/3 0 0 0 0 0
Gi3/4 0 0 0 0 0
Gi3/5 0 0 0 0 0
Gi3/6 0 0 0 0 0
Te4/1 0 0 0 0 0
Te4/2 0 0 0 0 0
Gi4/3 0 0 0 0 0
Gi4/4 0 0 0 0 0
Gi4/5 0 0 0 0 0
Gi4/6 0 0 0 0 0
12-10-2010 11:22 AM
From the outputs you provided I suspect packets are being software processed Input Acl punted packets are being dropped. These could be control packets like IGMP, OSPF etc. Or there is a ACL defined that is logging. If the problem is happening now can you collect show platform health and show platform cpu packet statistics?
If it is traffic being punted to the CPU try using the following debug: (built-in-sniffer)
- debug platform packet all receive buffer (safe to use in high CPU condition that is what it is built for)
- show platform CPU packet buffered to see what packets are being punted.
The output will provide src / dst and rx interface, etc...
Here is a whitepaper on TS high CPU on the 4500. Refer to the built in sniffer tool for more details.
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml
Regards,
Dale
12-10-2010 11:59 AM
Thanks for the reply Dale. I have few further queries.
Can you let me about this output from show plat cpu packet stat all.Apart from Input Acl Fwd event, i can see event Sa Miss and Packets Dropped In Processing by reasons NoDstPorts and Tx Mode Drop
.
.
.
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown 0 0 0 0 0
Sa Miss 24702624823 11 1 2 0
.
.
.
.
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
NoDstPorts 582487025 4 0 0 2
Tx Mode Drop 24702624802 11 1 2 0
.
.
.
Here is the output of show plat health
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Cat4500-sw1# sh platform health
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
GalChassisVp-review 3.00 0.14 10 45 100 500 0 0 0 1504:04
S2w-JobEventSchedule 10.00 0.00 10 0 100 500 0 0 0 0:00
Stub-JobEventSchedul 10.00 0.00 10 0 100 500 0 0 0 0:00
Lj-poll 1.00 0.02 2 0 100 500 0 0 0 267:57
StatValueMan Update 1.00 0.08 1 0 100 500 0 0 0 753:25
Pim-review 0.10 0.00 1 0 100 500 0 0 0 66:01
Ebm-host-review 1.00 0.00 8 7 100 500 0 0 0 3408:27
Ebm-host-util-review 1.00 0.00 10 0 100 500 0 0 0 2:13
Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 5:35
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:18
EbmHostRedundancyMan 2.00 0.00 20 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 0 100 500 0 0 0 0:00
IrmFibThrottler Thro 2.00 0.00 7 0 100 500 0 0 0 90:02
GalChassisVp Ondeman 2.00 0.00 2 0 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 10:20
KxAclPathMan update 2.00 0.00 10 0 100 500 0 0 0 6:44
KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00
GalK5TatooineStatsMa 0.70 0.01 4 0 100 500 0 0 0 274:11
MOL FastDropReview 2.00 0.00 15 0 100 500 0 0 0 0:00
RkiosL3PortMan AclFe 2.00 0.00 15 1 100 500 0 0 0 0:00
K5L3FlcMan FwdEntry 2.00 0.00 15 2 100 500 0 0 0 0:00
FwdEntry Zombie Revi 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3FlcMan Cam Shuff 4.00 0.00 25 0 100 500 0 0 0 0:00
K5L3Unciast IFE Revi 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3UnicastRpf IFE R 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3Unicast Fwd Entr 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3McastMan IrmMfib 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3McastMan ImeSync 2.00 0.00 15 0 100 500 0 0 0 3:42
K5L3Unicast Fwd Entr 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3Unicast Adj Grou 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3Unicast Adj Chan 2.00 0.00 15 0 100 500 0 0 0 0:00
K5L3Unicast Adj Tabl 2.00 8.38 15 8 100 500 9 10 7 68125:26
K5L3AdjStatsMan Revi 2.00 0.49 10 7 100 500 0 0 0 2929:28
K5L3McastMan RetSync 1.00 0.00 8 0 100 500 0 0 0 0:00
K5FlcHitMan review 2.00 0.02 5 4 100 500 0 0 0 1876:42
K5PortMan Regular Re 2.00 0.20 15 12 100 500 0 0 0 4508:03
K5PortMan Ondemand L 3.00 0.51 30 4 100 500 0 0 0 4894:51
K5PortMan Stats Revi 2.00 0.07 15 5 100 500 0 0 0 503:17
K5PortMan Tx Queue R 3.00 0.00 15 0 100 500 0 0 0 0:00
K5L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 1853:53
K5 L2 Aging Table Re 2.00 0.04 20 6 100 500 0 0 0 1594:39
K5 L2 Unicast Addres 2.00 0.00 10 5 100 500 0 0 0 16:46
K5 L2 Multicast Addr 2.00 0.00 10 0 100 500 0 0 0 17:13
K5 L2 Hardware Addre 2.00 0.00 10 8 100 500 0 1 1 12840:29
K5RetStatsMan Review 2.00 0.00 5 0 100 500 0 0 0 0:00
K5CpuMan Review 30.00 7.64 30 2001 100 500 15 20 34 530517:33
K5ForerunnerPacketMa 1.50 0.23 4 4 100 500 1 1 0 4169:38
K5ForerunnerPacketMa 0.70 0.28 4 4 100 500 0 0 0 2864:58
K5QosDhmMan Rate DBL 2.00 0.00 7 0 100 500 0 0 0 0:00
K5QosPolicerStatsMan 1.00 0.00 10 0 100 500 0 0 0 54:12
K5VlanStatsReview 2.00 0.51 10 6 100 500 0 0 0 7990:12
K5AclMan-labeledFlat 1.00 0.00 10 0 100 500 0 0 0 0:00
K5AclLabelMan-punted 1.00 0.00 10 0 100 500 0 0 0 6:52
K5AclCamMan stale en 1.00 0.00 10 0 100 500 0 0 0 0:00
K5AclCamStatsMan hw 3.00 0.10 10 5 100 500 0 0 0 4412:49
VSI DMA Local Jawa R 1.00 0.05 6 4 100 500 0 0 0 607:22
VSI DMA Remote Jawa 1.00 0.06 6 0 100 500 0 0 0 605:10
VSI DMA Darkside Rev 1.00 0.00 6 0 100 500 0 0 0 0:00
VSI DMA Slot-01 Revi 1.00 0.26 6 4 100 500 0 0 0 2901:13
VSI DMA Slot-02 Revi 1.00 0.26 6 4 100 500 0 0 0 2895:43
VSI DMA Slot-05 Revi 1.00 0.27 6 4 100 500 0 0 0 2948:02
VSI DMA Slot-06 Revi 1.00 0.25 6 4 100 500 0 0 0 2957:10
VSI DMA Slot-07 Revi 1.00 0.26 6 4 100 500 0 0 0 2905:59
GbmHostTable Review 2.00 0.00 8 0 100 500 0 0 0 17:56
GbmHostTableUtilizat 2.00 0.00 8 0 100 500 0 0 0 0:00
RkiosPortMan Port Re 2.00 0.17 12 6 100 500 0 0 0 3454:12
Rkios Module State R 4.00 0.02 40 2 100 500 0 0 0 283:17
Rkios Online Diag Re 4.00 0.02 40 0 100 500 0 0 0 276:13
RkiosIpPbr IrmPort R 2.00 0.00 10 5 100 500 0 0 0 368:38
RkiosIpWccp IrmWccpW 2.00 0.00 10 0 100 500 0 0 0 53:44
RkiosAclMan Review 3.00 0.03 30 0 100 500 0 0 0 535:53
MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00
LocalJawaVsiMan VsiR 0.20 0.00 2 0 100 500 0 0 0 0:00
QuackMan( 254 ) Revi 4.00 0.00 20 4 100 500 0 0 0 0:02
GlmBridgeMan(2) revi 0.50 0.01 2 0 100 500 0 0 0 106:46
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:20
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:00
Xgstub Stats Review 0.50 0.09 5 4 100 500 0 0 0 1059:45
EpmPortGroup(2) stat 0.50 0.01 2 0 100 500 0 0 0 95:41
EpmPortGroup(2) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(2) on d 0.50 0.00 4 8 100 500 0 0 0 0:00
X2Controller(2) 0.50 0.00 0 0 100 500 0 0 0 9:22
PluggableLinecard(2) 0.50 0.00 1 0 100 500 0 0 0 0:00
PluggableLinecard(2) 0.50 0.00 1 0 100 500 0 0 0 0:00
EpmPluggableGroup(2) 0.50 0.00 2 0 100 500 0 0 0 4:44
QuackMan( 2 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00
GlmBridgeMan(0) revi 0.50 0.00 2 0 100 500 0 0 0 108:09
GlmBridgeMan(1) revi 0.50 0.00 2 4 100 500 0 0 0 108:17
RemoteJawaVsiMan Vsi 0.20 0.00 2 0 100 500 0 0 0 0:00
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:24
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:10
Xgstub Stats Review 0.50 0.11 5 4 100 500 0 0 0 1201:52
EthPhyControllerMan( 0.40 0.00 4 0 100 500 0 0 0 17:49
EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 8:36
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 4:47
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 5:28
EpmPortGroup(0) stat 0.50 0.00 2 4 100 500 0 0 0 215:40
EpmPortGroup(0) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(0) on d 0.50 0.00 4 8 100 500 0 0 0 0:03
GlmBridgeMan(3) revi 0.50 0.01 2 0 100 500 0 0 0 106:54
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:24
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:17
Xgstub Stats Review 0.50 0.11 5 4 100 500 0 0 0 1201:10
EthPhyControllerMan( 0.40 0.00 4 4 100 500 0 0 0 17:40
EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 8:24
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 4:47
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 5:29
EpmPortGroup(1) stat 0.50 0.03 2 4 100 500 0 0 0 212:10
EpmPortGroup(1) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(1) on d 0.50 0.00 4 4 100 500 0 0 0 0:06
GlmBridgeMan(4) revi 0.50 0.00 2 0 100 500 0 0 0 111:48
GlmBridgeMan(5) revi 0.50 0.00 2 0 100 500 0 0 0 112:33
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:21
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:00
Xgstub Stats Review 0.50 0.09 5 0 100 500 0 0 0 1058:55
EpmPortGroup(3) stat 0.50 0.00 2 4 100 500 0 0 0 95:40
EpmPortGroup(3) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(3) on d 0.50 0.00 4 6 100 500 0 0 0 0:00
GlmBridgeMan(6) revi 0.50 0.00 2 4 100 500 0 0 0 108:02
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:24
Stub ondemand global 0.50 0.00 5 4 100 500 0 0 0 38:16
Xgstub Stats Review 0.50 0.11 5 4 100 500 0 0 0 1201:17
EthPhyControllerMan( 0.40 0.00 4 1 100 500 0 0 0 19:34
EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 15:26
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 4:46
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 5:29
EpmPortGroup(4) stat 0.50 0.00 2 0 100 500 0 0 0 100:24
EpmPortGroup(4) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(4) on d 0.50 0.00 4 3 100 500 0 0 0 8:09
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:24
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 43:03
Xgstub Stats Review 0.50 0.11 5 4 100 500 0 0 0 1200:22
EthPhyControllerMan( 0.40 0.00 4 0 100 500 0 0 0 18:29
EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 16:07
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 4:48
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 5:31
EpmPortGroup(5) stat 0.50 0.01 2 4 100 500 0 0 0 41:18
EpmPortGroup(5) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(5) on d 0.50 0.00 4 3 100 500 0 0 0 10:17
Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 2:24
Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:00
Xgstub Stats Review 0.50 0.11 5 4 100 500 0 0 0 1200:34
EthPhyControllerMan( 0.40 0.00 4 1 100 500 0 0 0 18:48
EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 7:40
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 4:47
EthPoeControllerMan( 0.20 0.00 2 0 100 500 0 0 0 5:29
EpmPortGroup(6) stat 0.50 0.00 2 0 100 500 0 0 0 27:28
EpmPortGroup(6) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00
EpmPortGroup(6) on d 0.50 0.00 4 0 100 500 0 0 0 0:00
X2Controller(3) 0.50 0.00 0 0 100 500 0 0 0 9:20
PluggableLinecard(3) 0.50 0.00 1 0 100 500 0 0 0 0:00
PluggableLinecard(3) 0.50 0.00 1 0 100 500 0 0 0 0:00
EpmPluggableGroup(3) 0.50 0.00 2 0 100 500 0 0 0 4:35
QuackMan( 3 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00
QuackMan( 0 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00
QuackMan( 1 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00
QuackMan( 4 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:01
QuackMan( 5 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:01
QuackMan( 6 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:01
-------------
%CPU Totals 234.69 21.45
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 600.00 99% 596.36 100%
Linecard 2's Store 600.00 99% 596.36 100%
Linecard 3's Store 600.00 34% 205.72 100%
Linecard 4's Store 600.00 34% 205.72 100%
Linecard 5's Store 600.00 99% 596.36 100%
Linecard 6's Store 600.00 99% 596.36 100%
Linecard 7's Store 600.00 99% 596.36 100%
TSM objects ------------------ ------------------
PacketInfoItem 859.37 0% 4.29 0%
VbufNodes2400 80.50 0% 0.00 0%
VbufNodes1600 55.50 0% 3.46 0%
VbufNodes400 288.00 1% 4.50 87%
VbufNodes64 60.00 0% 0.46 0%
VbufNodes4200 68.37 43% 29.91 100%
PacketBufRaw 184.29 100% 184.29 100%
PacketBufRaw 4644.56 100% 4644.56 100%
RkiosSysPacketBuf 250.00 0% 1.12 0%
Packet 2322.42 2% 54.73 98%
IndexCache 800.39 0% 0.00 0%
K5AclOpDescNode 10752.00 0% 0.00 0%
K5AclLabelListNode 768.00 0% 0.00 0%
K5AclIpv6PackedAddrH 1024.00 0% 0.00 0%
K5AdjGroups 960.00 0% 0.13 100%
K5L3FlcEntryAvlTree2 12902.39 0% 0.46 100%
IrmFibUnicastRpfList 5120.00 0% 0.15 100%
K5L3FwdEntrys 29120.00 0% 1.09 100%
K5L3FwdEntryAvlTree2 12480.00 0% 0.37 100%
K5L3FwdTreeEntrys 21840.00 0% 1.31 100%
K5L3FwdTreeEntryAvlT 49920.00 0% 0.23 100%
IrmMfibFastDropFlowM 576.00 0% 0.00 0%
K5QosTxQueSelTableBl 14.00 2% 0.38 100%
K5QosPolicerBlockNod 2.00 0% 0.00 0%
K5QosPolicerBlockMem 69.00 0% 0.00 0%
K5QosPolicerMemAlloc 448.00 0% 0.00 0%
K5QosFeatureInfoList 1536.00 0% 0.00 0%
K5QosLabelToFeatureE 1792.00 0% 0.00 0%
K5QosPathFeatureInfo 384.00 0% 0.00 0%
K5CpuPacketInfoItem 859.37 0% 0.00 0%
K5TxPacketInfo 384.00 0% 2.36 0%
MatEntrys 7680.00 0% 0.46 100%
MatEntryTableIterato 1.12 0% 0.03 0%
RkiosAclMan NamedGal 43.75 0% 0.00 0%
LpipAceListNode 192.00 0% 0.00 0%
Rkios Acl VlanMaps 128.00 0% 0.00 0%
Rkios Acl VlanMapEnt 859.37 0% 0.00 0%
Rkios EVC EFPs 1280.00 0% 0.00 0%
Rkios GBM BridgeDoma 256.00 0% 0.00 0%
Rkios GBM PseudoPort 768.00 0% 0.00 0%
Rkios QoS PolicyMaps 356.32 0% 0.00 0%
Rkios QoS ClassMaps 896.00 0% 0.02 100%
AclToIosFilterMapLis 384.00 0% 0.00 0%
Rkios QoS Policers 456.00 0% 0.00 0%
RkiosTableMap Galios 3.00 0% 0.00 0%
PimPhyports 1000.00 25% 257.00 100%
PimPorts 890.62 32% 285.89 100%
PimModules 156.00 2% 4.26 100%
PimSlots 5.00 2% 0.13 100%
PimChassis 38.37 6% 2.39 100%
PimQuack 0.35 26% 0.09 100%
EbmVlans 4832.00 0% 35.39 100%
EbmVlanGroupEntrys 1920.00 0% 0.35 33%
EbmPorts 240.00 30% 74.29 100%
EbmPortHostEntrys 384.00 0% 0.00 0%
EbmIeNodes 536.00 0% 4.71 100%
EbmPortVlanAclFeatur 1536.00 0% 0.00 0%
EbmSortedHostTableIt 1.87 0% 0.05 0%
EbmSortedGroupTableI 1.75 0% 0.05 0%
EbmHostRedundancyMan 1082.81 0% 0.00 0%
IrmVrfs 317.00 0% 1.23 100%
IrmFibLoadBalances 1152.00 0% 0.00 0%
IrmFibAdjs 1920.00 0% 0.50 100%
IrmPortEtherAddrEntr 500.00 0% 0.00 0%
IrmFibEntries 12288.00 0% 0.51 100%
IrmMfibEntryMemMan 13312.00 0% 0.10 100%
IrmWccpMemMan 34.06 0% 0.00 0%
AclOp 2048.00 0% 0.00 0%
AclOpAceSet 2559.96 0% 0.00 0%
AclClassifier 1280.00 0% 1.17 100%
AclFeature 2570.04 0% 1.63 100%
Acl 1408.00 0% 1.28 100%
Ace24 9215.85 0% 3.58 100%
Ace48 15359.76 0% 0.11 100%
AceActionDescStorage 1023.98 0% 0.00 0%
AclListNode 256.00 0% 0.23 100%
AceListNode 25600.00 0% 0.03 0%
AclClassifierActionL 1152.00 0% 0.91 100%
AclLayerFeatureListN 512.00 0% 0.00 0%
AclClassifierListNod 256.00 0% 0.00 0%
TableMapMan NameToTa 27.00 0% 0.00 0%
TableMapAllocator 116.00 0% 0.00 0%
FlatAcl 512.00 0% 0.00 0%
FlatAce24 10752.00 0% 0.00 0%
FlatAce48 16896.00 0% 0.00 0%
FlatAceActionListNod 256000.00 0% 0.07 0%
FlatAclOpSetStorage 3072.00 0% 0.00 0%
FlatAclCacheNode 4224.00 0% 0.00 0%
FlatAclListNode 256.00 0% 0.00 0%
QosFeatureClassifier 135.26 0% 0.00 0%
QosFeatureClassifier 2367.14 0% 0.00 0%
QosClassifierActionL 896.00 0% 0.00 0%
QosPortVlanAclFeatur 1152.00 0% 0.00 0%
QosEvcEfpAllocator 4352.00 0% 0.00 0%
QoS Policers 5624.00 0% 0.00 0%
MOL EVC EFPs 9728.00 0% 0.00 0%
EvcEfpEncapVlanRange 512.00 0% 0.00 0%
MOL GBM BridgeDomain 2048.00 0% 0.00 0%
MOL GBM PseudoPort 5376.00 0% 0.00 0%
GbmSortedHostTableIt 1.75 0% 0.00 0%
GbmBridgeDomainHostE 12288.00 0% 0.00 0%
KxAclPath 1024.00 0% 0.00 0%
KxAclPathListNode 256.00 0% 0.00 0%
KxAclConstPathListNo 256.00 0% 0.00 0%
EpmPortGroups 0.81 87% 0.71 100%
EpmPortMans 61.27 99% 61.03 100%
KxAclLabeledFlatAcl 3968.00 0% 0.00 0%
KxAclLabeledFlatAclE 3072.00 0% 0.00 0%
EbmVlanHostEntrys 3437.50 9% 384.12 83%
VsiBuffers3072 300.00 0% 3.00 0%
VsiBuffers1024 750.00 64% 486.00 100%
VsiBuffers128 250.00 0% 21.50 0%
VsiBuffers16 31.25 3% 22.00 4%
VsiTransactions(Max 82.03 4% 8.32 40%
VsiTransactions(Max 106.64 35% 88.96 42%
VsiTransactions(Max 15.42 0% 0.18 0%
VsiTransactions(Max 5.90 0% 0.00 0%
VsiTransactions(Max 4.83 0% 0.00 0%
VsiTransactionRespon 11.89 18% 4.88 44%
VsiS2wRequestMans 54.68 0% 5.07 0%
VsiVliRequestMans 147.65 26% 50.83 75%
VsiMdio22RequestMans 273.43 0% 105.00 0%
VsiMdio45RequestMans 27.34 0% 0.15 0%
VsiJtagRequestMans 4.92 0% 0.00 0%
GalGbicEntrys 11.85 0% 0.09 100%
EbmVlanBroadcastFloo 1408.00 0% 10.31 100%
Event Nodes 128.00 0% 0.00 0%
Event Nodes 128.00 0% 0.21 0%
RkisoIpPbrRouteMaps 97.65 0% 0.00 0%
Rkios EVC EFPs 768.00 0% 0.00 0%
K5AclLabelSignatureM 11648.00 0% 0.00 0%
K5AclLabelMapEntryPa 1408.00 0% 0.00 0%
TableMapMan NameToTa 27.00 0% 0.00 0%
TableMapAllocator 116.00 0% 0.00 0%
InpTosMarkTbl BlockA 1.75 12% 0.21 100%
InpCosMarkTbl BlockA 1.75 12% 0.21 100%
InpExpMarkTbl BlockA 1.75 12% 0.21 100%
OutTosMarkTbl BlockA 1.75 12% 0.21 100%
OutCosMarkTbl BlockA 1.75 12% 0.21 100%
OutExpMarkTbl BlockA 1.75 12% 0.21 100%
CommandTables 48.00 14% 6.75 100%
------------------ ------------------
TSM totals 645533.02 1% 6870.98 95%
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
12-12-2010 10:03 PM
Sachin,
The SA Miss is a packet flooded in the vlan and a copy sent the CPU for mac learning. The Tx mode drop (almost exact match to the amount of drops for SA MIss ) is that same packet getting dropped. Since the packet was already bridged by hardware, software drops the packet after doing source learning. The NoDstPorts are packets that cannot be forwarded. Example TTL=1 or hairpinned traffic that cannot be forwarded out the same interface it was received.
I can see from the outputs you're running the command when the CPU is normal. These outputs will help determine the root cause while the problem is happening. From the information so far I think it is best you get a idea of what traffic is punted to the CPU when running high. The debug packet all recieve buffer would be the ideal tool to use. You can also do an inband span and monitor all traffic to the CPU.
Once you identify what traffic is causing the problem it will be much easier to identifiy the root cause. Right now there isn't enough information to tell.
Take Care,
Dale
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide