02-09-2014 08:28 PM - edited 03-07-2019 06:06 PM
Hello Forum Members,
A little bit Question,
Our 4506 Switch act as a core, for last 3 weeks our CPU performance is on normal always at 65-70% utilization in weekday and 30-35% in Weekend. but this weekend at saturday our switch reach 97-99% for more than 1 hours.
How can i investigate this case? i monitor this switch with NMS Software.
Is there any way to find the root cause for this case.
Need Forum members help.
thanx
hamzah
02-10-2014 02:09 AM
Hi Hamzah,
You can use these commands to monitor the CPU:
Show proc cpu sort | ex 0.00
Show platform history
Using the first command will tell you what process is eating up the CPU.
Give me the output of:
Show proc cpu sort | ex 0.00
show platform health
Thanks
Ankur
"Please rate the post if found useful"
02-10-2014 06:44 PM
#Show proc cpu sort | ex 0.00
28 2955212 455433 6488 77.67% 47.85% 40.66% 0 Cat4k Mgmt LoPri
27 945920 2123818 445 12.23% 11.28% 10.98% 0 Cat4k Mgmt HiPri
38 382720 357889 1069 3.91% 4.34% 4.30% 0 Spanning Tree
44 100136 336531 297 1.51% 1.12% 1.06% 0 IP Input
81 26604 176099 151 0.63% 0.34% 0.29% 0 Standby (HSRP)
16 53696 97695 549 0.63% 0.73% 0.66% 0 ARP Input
33 2340 61380 38 0.15% 0.04% 0.01% 0 Net Input
69 4312 61583 70 0.15% 0.05% 0.03% 0 VLAN Manager
50 7356 19463 377 0.15% 0.12% 0.11% 0 CEF process
93 8984 92831 96 0.07% 0.05% 0.05% 0 IP-EIGRP(0): PDM
49 2960 3447 858 0.07% 0.07% 0.06% 0 Adj Manager
94 6384 94908 67 0.07% 0.07% 0.07% 0 IP-EIGRP(0): HEL
##############################################
#sh platf hea
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
Lj-poll 1.00 0.03 2 1 100 500 0 0 0 0:02
GalChassisVp-review 3.00 0.20 10 22 100 500 0 0 0 0:13
S2w-JobEventSchedule 10.00 0.79 10 9 100 500 0 0 0 1:08
Stub-JobEventSchedul 10.00 1.94 10 5 100 500 1 1 1 2:33
Pim-review 0.10 0.01 1 0 100 500 0 0 0 0:00
Ebm-host-review 1.00 0.38 8 4 100 500 0 0 0 0:07
Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 0:00
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 5 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 0:30
KxAclPathMan update 2.00 0.00 10 1 100 500 0 0 0 0:01
KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00
TagMan-InformMtegRev 1.00 0.00 5 0 100 500 0 0 0 0:00
TagMan-RecreateMtegR 1.00 0.00 10 4 100 500 0 0 0 0:00
K2CpuMan Review 30.00 11.08 30 47 100 500 8 8 7 12:31
K2AccelPacketMan: Tx 10.00 4.58 20 2 100 500 4 4 3 5:43
K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00
K2AclMan-taggedFlatA 1.00 0.00 10 2 100 500 0 0 0 0:00
K2AclCamMan stale en 1.00 0.00 10 5 100 500 0 0 0 0:00
K2AclCamMan hw stats 3.00 0.09 10 5 100 500 0 0 0 0:38
K2AclCamMan kx stats 1.00 0.00 10 5 100 500 0 0 0 0:13
K2AclCamMan Audit re 1.00 0.00 10 5 100 500 0 0 0 0:17
K2AclPolicerTableMan 1.00 0.00 10 0 100 500 0 0 0 0:00
K2L2 Address Table R 2.00 1.03 10 8 100 500 0 0 0 0:16
K2L2 New Static Addr 2.00 0.00 10 7 100 500 0 0 0 0:00
K2L2 New Multicast A 2.00 0.00 10 7 100 500 0 0 0 0:00
K2L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 0:00
K2 L2 Destination Ca 2.00 0.00 10 0 100 500 0 0 0 0:00
K2PortMan Review 2.00 3.20 15 11 100 500 2 2 1 3:26
Gigaport0 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport1 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport2 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport3 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport4 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport5 Review 0.40 0.06 4 1 100 500 0 0 0 0:07
Gigaport6 Review 0.40 0.08 4 2 100 500 0 0 0 0:06
Gigaport7 Review 0.40 0.10 4 1 100 500 0 0 0 0:07
Gigaport8 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport9 Review 0.40 0.07 4 1 100 500 0 0 0 0:05
Gigaport10 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport11 Review 0.40 0.09 4 1 100 500 0 0 0 0:06
Gigaport12 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport13 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport14 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport15 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport16 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport17 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport18 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport19 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport20 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport21 Review 0.40 0.06 4 1 100 500 0 0 0 0:07
Gigaport22 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport23 Review 0.40 0.06 4 1 100 500 0 0 0 0:06
Gigaport24 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport25 Review 0.40 0.09 4 1 100 500 0 0 0 0:06
Gigaport26 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport27 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
Gigaport28 Review 0.40 0.09 4 1 100 500 0 0 0 0:06
Gigaport29 Review 0.40 0.09 4 1 100 500 0 0 0 0:06
Gigaport30 Review 0.40 0.08 4 1 100 500 0 0 0 0:06
Gigaport31 Review 0.40 0.07 4 1 100 500 0 0 0 0:06
K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib IrmFib Review 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Unpunt Rev 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Consistency Ch 1.00 0.00 5 2 100 500 2 2 1 1:58
K2FibAdjMan Stats Re 2.00 0.14 10 7 100 500 0 0 0 0:47
K2FibAdjMan Host Mov 2.00 1.72 10 7 100 500 1 0 0 1:36
K2FibAdjMan Adj Chan 2.00 0.00 10 1 100 500 0 0 0 0:00
K2FibMulticast Signa 2.00 0.10 10 2 100 500 0 0 0 0:03
K2FibMulticast Entry 2.00 0.00 10 6 100 500 0 0 0 0:00
K2FibMulticast Irm M 2.00 0.00 10 7 100 500 0 0 0 0:00
K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 0:00
K2FibPbr route map r 2.00 0.03 10 9 100 500 0 0 0 0:17
K2FibPbr flat acl pr 2.00 0.01 10 9 100 500 0 0 0 0:04
K2FibPbr consolidati 2.00 0.00 10 0 100 500 0 0 0 0:01
K2FibPerVlanPuntMan 2.00 0.00 15 2 100 500 0 0 0 0:00
K2FibFlowCache flow 2.00 0.02 10 8 100 500 0 0 0 0:06
K2FibFlowCache flow 2.00 18.85 10 8 100 500 40 42 23 38:23
K2FibFlowCache adj r 2.00 0.03 10 5 100 500 0 0 0 0:10
K2FibFlowCache flow 2.00 0.00 10 2 100 500 0 0 0 0:01
K2MetStatsMan Review 2.00 0.58 5 2 100 500 0 0 0 0:28
K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00
K2QosDblMan Rate DBL 2.00 0.12 7 0 100 500 0 0 0 0:09
IrmFibThrottler Thro 2.00 0.00 7 5 100 500 0 0 0 0:01
K2 VlanStatsMan Revi 2.00 1.04 15 4 100 500 0 0 0 1:08
K2 Packet Memory Dia 2.00 0.00 15 8 100 500 0 0 0 1:15
K2 L2 Aging Table Re 2.00 0.06 20 3 100 500 0 0 0 0:13
RkiosPortMan Port Re 2.00 0.24 12 39 100 500 0 0 0 0:21
Rkios Module State R 4.00 0.02 40 5 100 500 0 0 0 0:02
RkiosIpPbr IrmPort R 2.00 0.11 10 8 100 500 0 0 0 0:15
RkiosAclMan Review 3.00 0.03 30 1 100 500 0 0 0 0:04
MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
ILC Manager Review 5.00 0.00 10 0 100 500 0 0 0 0:00
ILC S2wMan Review 5.00 0.00 15 0 100 500 0 0 0 0:00
EthHoleLinecardMan(1 2.00 0.01 10 8 100 500 0 0 0 0:02
EthHoleLinecardMan(2 2.00 0.14 10 5 100 500 0 0 0 0:21
EthHoleLinecardMan(3 2.00 0.17 10 5 100 500 0 0 0 0:21
EthHoleLinecardMan(4 2.00 0.35 10 4 100 500 0 0 0 0:21
-------------
%CPU Totals 237.80 49.75
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 3% 8.56 100%
Linecard 2's Store 258.00 3% 9.48 100%
Linecard 3's Store 258.00 3% 9.48 100%
Linecard 4's Store 258.00 3% 9.48 100%
Linecard 5's Store 258.00 60% 156.43 100%
Linecard 6's Store 258.00 60% 156.40 100%
TSM objects ------------------ ------------------
RkiosSysPacketBuf 250.00 0% 250.00 0%
PacketBufRaw 20355.00 100% 20355.00 100%
PacketBufRawJumbo 732.81 25% 183.20 100%
Packet 1026.56 0% 1026.56 0%
PacketInfoItem 390.62 0% 1.95 0%
VbufNodes2400 80.75 0% 20.18 0%
VbufNodes1600 55.75 0% 13.93 0%
VbufNodes400 73.00 5% 18.25 21%
VbufNodes64 62.00 0% 31.00 0%
PimPhyports 968.75 24% 238.31 100%
PimPorts 789.06 31% 244.60 100%
PimModules 148.00 2% 9.25 37%
PimSlots 4.00 2% 0.25 37%
PimChassis 16.87 6% 1.05 100%
EbmVlans 2560.00 1% 36.25 100%
EbmVlanHostEntrys 2048.00 4% 86.62 97%
EbmVlanGroupEntrys 1856.00 0% 58.00 10%
EbmPorts 168.00 30% 50.85 100%
EbmPortHostEntrys 320.00 0% 20.00 0%
EbmIeNodes 536.00 0% 5.23 80%
EbmPortVlanAclFeatur 31.99 0% 0.02 0%
EbmSortedHostTableIt 1.87 0% 1.87 0%
EbmSortedGroupTableI 1.75 0% 1.75 0%
IrmVrfs 6.60 1% 6.60 1%
IrmFibAdjs 768.00 3% 96.00 28%
IrmPortEtherAddrEntr 375.00 0% 0.25 100%
IrmFibEntries 10240.00 1% 140.42 96%
AclL4Op 384.00 0% 384.00 0%
AclL4OpTriplet 256.00 0% 256.00 0%
AclClassifier 768.00 0% 768.00 0%
AclFeature 351.00 0% 351.00 0%
Acl 384.00 0% 384.00 0%
Ace 1280.00 0% 1280.00 0%
AceActionDescStorage 256.00 0% 0.00 0%
AclListNode 256.00 0% 256.00 0%
AceListNode 1280.00 0% 4.01 85%
AclClassifierActionL 512.00 0% 512.00 0%
AclLayerFeatureListN 512.00 0% 0.10 69%
FlatAcl 512.00 0% 0.18 66%
FlatAce 3840.00 0% 27.59 86%
FlatAceActionListNod 7680.00 0% 11.97 86%
FlatAclL4OpSetStorag 1024.00 0% 0.01 0%
FlatAclCacheNode 1024.00 0% 0.25 100%
QoS Policers 160.00 0% 40.00 0%
KxAclPath 1024.00 0% 0.06 100%
KxAclPathListNode 256.00 0% 0.01 0%
GalGbicEntrys 10.39 0% 10.39 0%
CommandTables 48.00 20% 9.93 100%
K2FibPbrFlatRouteMap 481.46 0% 0.76 100%
K2FibPbrExpandedFlat 2304.00 0% 13.88 99%
K2FibPbrFlatRouteMap 320.00 0% 0.54 100%
K2FibVrfs 38.33 1% 0.58 100%
K2TxPacket 384.00 0% 0.75 0%
K2TxPacketInfo 256.00 0% 0.50 0%
MatEntrys 5632.00 0% 10.31 100%
MatEntryTableIterato 1.12 0% 1.12 0%
RkiosAclMan NamedGal 42.18 0% 42.18 0%
Rkios Acl VlanMaps 272.00 0% 272.00 0%
Rkios Acl VlanMapEnt 1015.62 0% 1015.62 0%
Rkios QoS PolicyMaps 34.50 0% 6.00 0%
Rkios QoS ClassMaps 896.00 0% 896.00 0%
AclToIosFilterMapLis 384.00 0% 384.00 0%
Rkios QoS Policers 32.00 0% 8.00 0%
Event Nodes 84.00 0% 80.00 0%
Event Nodes 84.00 1% 80.00 1%
KxAclTagPairNode 2176.00 0% 0.06 100%
KxAclMappingTableEnt 96.02 0% 96.02 0%
KxAclMappingTableEnt 64.01 0% 64.01 0%
KxAclTaggedFlatAcl 768.00 0% 0.02 100%
RkisoIpPbrRouteMaps 97.65 0% 0.01 100%
IrmFlows 224.00 60% 135.62 99%
------------------ ------------------
TSM totals 81370.74 26% 30301.10 71%
02-10-2014 07:47 PM
Hi Hamzah,
From the first command you see that process Cat4k Mgmt LoPri is utilizing more CPU. shoe plaftform health shows K2CpuMan Review and K2PortMan Review to be causing high CPU utilization. The below URL will help you troubleshoot CPU utilizationstep by step.
Regards,
Sathvik K V
02-11-2014 12:42 AM
Hamzah,
Run the command show platform cpu packet statistics to check what packets are being punted on the CPU. You may also capture the CPU bound packets using a cpu span session and check the traffic (Don't run the capture when CPU is already at 99%, run it when it goes little bit down). Once you have that traffic you will easily be able to isolate the cause of high CPU. Rest everything is explained in the document given by Sathvik.
Thanks
Ankur
"Please rate the post if found useful"
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: