05-14-2013 12:04 AM - edited 03-07-2019 01:20 PM
hi all,
Our Customer used Distribution switches are DistSW4507-01 and DistSW4507-02. These switches used HSRP with Dist-02 is active, Dist-01 is standby.
But, the past few days then the CPU of Dist-01 increase to 60%, you can see result as follow:
DistSW-4507-01#sh processes cpu hi
54515 5 5 5 5 5 545 5 5 5 515 5 515 5 5 5 5 5 5 515 5 515
9414049494949494941494949494049494049494949494949404949404
100
90
80
70
60
50 * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
40 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *
30 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *
20 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
5555555555555555555555555555555555555555554554444445555545444444444444
5555454554444343443332233222221211211000009009999980000090999998877787
100
90
80
70
60 **** * **
50 **********************************************************************
40 **********************************************************************
30 **********************************************************************
20 **********************************************************************
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
DistSW-4507-01#sh processes cpu sorted
CPU utilization for five seconds: 54%/0%; one minute: 11%; five minutes: 9%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
41 263762272 258671478 1019 52.15% 8.30% 6.38% 0 Cat4k Mgmt LoPri
40 68970600 388841616 177 2.07% 2.05% 2.05% 0 Cat4k Mgmt HiPri
85 1575364 11474716 137 0.07% 0.02% 0.00% 0 Spanning Tree
69 40 7066609 0 0.07% 0.00% 0.00% 0 UDLD
82 2082608 61332016 33 0.07% 0.04% 0.03% 0 IP Input
135 1170540 54087423 21 0.07% 0.05% 0.07% 0 HSRP (Standby)
7 5794312 955116 6066 0.00% 0.06% 0.05% 0 Check heaps
8 0 3 0 0.00% 0.00% 0.00% 0 Pool Manager
Please help me resolve,
thank all,
Quynh
05-14-2013 12:05 AM
Normal,
CPU of the Dist-01 is very low as follow:
DistSW-4507-01#sh processes cpu sorted | exclude 0.00
CPU utilization for five seconds: 6%/0%; one minute: 7%; five minutes: 8%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
41 263764144 258675016 1019 3.43% 4.46% 5.60% 0 Cat4k Mgmt LoPri
40 68971748 388848159 177 2.15% 2.07% 2.05% 0 Cat4k Mgmt HiPri
135 1170540 54088269 21 0.07% 0.05% 0.07% 0 HSRP (Standby)
149 4660 64237 72 0.07% 0.11% 0.27% 1 Virtual Exec
77 1652764 3311459 499 0.07% 0.09% 0.08% 0 CDP Protocol
05-14-2013 12:41 AM
Hi Dao,
Could you please provide me with the following outputs:
show process cpu sorted | ex 0.00%
Show platform cpu packet statistics
Show platform health | ex 0.00
Also the below link has very good document on troubleshooting high cpu on cat 4500 kindly go through the same which help you in finding the root cause:
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2
HTH
Regards
Inayath
*Plz rate all usefull posts.
05-14-2013 04:02 AM
Mgmt High & Low priority process'' processes control plane as well as data packets that need to be software-switched or processed.Try the following.
Hardcode the L3 interfaces with the commands no ip redirect, no ip unreachable and no ip proxy-arp. It should come to normal.
--Pls rate helpful posts--
05-26-2013 09:52 PM
hi InayathUlla Sharieff,
These are the result of the commands:
DistSW-4507-01#show platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
------------ --------------- --------------- --------------- ---------------
0 0 0 0 287751
RkiosSysPacketMan:
Packet allocation falures: 0
Packet Buffer(Software Common) allocation falures: 0
Packet Buffer(Software ESMP) allocation falures: 0
Packet Buffer(Software EOBC) allocation falures: 0
IOS Packet Buffer Wrapper allocation falures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
431 0 0 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Input Acl 420 0 0 0 0
SA Miss 11 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Medium 11 0 0 0 0
High 420 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
SrcAddrTableFilt 8 0 0 0 0
L2DstDrop 3 0 0 0 0
NoFloodPorts 420 0 0 0 0
Total packet queues 16
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp 166023605 25 18 13 6
L2/L3Control 99934266 15 7 3 0
Host Learning 4353 0 0 0 0
L3 Fwd Low 9 0 0 0 0
L2 Fwd Low 331699 0 0 0 0
L3 Rx Low 2333469 2 0 0 0
DistSW-4507-01#show platform health | exclude 0.00
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
GalChassisVp-review 3.00 0.14 10 23 100 500 0 0 0 147:38
Lj-poll 1.00 0.02 2 0 100 500 0 0 0 27:18
StatValueMan Update 1.00 0.14 1 0 100 500 0 0 0 59:21
K2AclCamMan hw stats 3.00 0.55 10 5 100 500 0 0 0 437:25
K2AclPolicerTableMan 1.00 0.84 10 8 100 500 0 0 0 518:21
K2PortMan Review 2.00 0.87 15 10 100 500 0 1 0 1032:56
Gigaport9 Review 0.40 0.05 4 1 100 500 0 0 0 76:47
Gigaport11 Review 0.40 0.05 4 1 100 500 0 0 0 77:06
Gigaport16 Review 0.40 0.04 4 1 100 500 0 0 0 72:41
Gigaport18 Review 0.40 0.04 4 1 100 500 0 0 0 72:46
Gigaport25 Review 0.40 0.04 4 1 100 500 0 0 0 72:48
Gigaport39 Review 0.40 0.04 4 1 100 500 0 0 0 72:46
Gigaport48 Review 0.40 0.04 4 1 100 500 0 0 0 71:04
Gigaport49 Review 0.40 0.04 4 1 100 500 0 0 0 69:14
Gigaport50 Review 0.40 0.04 4 1 100 500 0 0 0 70:43
Gigaport51 Review 0.40 0.05 4 1 100 500 0 0 0 73:51
Gigaport52 Review 0.40 0.04 4 1 100 500 0 0 0 68:05
Gigaport53 Review 0.40 0.05 4 0 100 500 0 0 0 74:59
K2Fib Vrf Unpunt Rev 2.00 0.01 15 0 100 500 0 0 0 9:06
K2FibAdjMan Stats Re 2.00 0.50 10 8 100 500 0 0 0 213:07
K2FibMulticast Signa 2.00 0.04 10 1 100 500 0 0 0 39:10
K2FibPbr route map r 2.00 0.01 20 4 100 500 0 0 0 153:35
K2FibPbr consolidati 2.00 0.02 10 0 100 500 0 0 0 12:00
K2FibFlowCache flow 2.00 0.01 10 0 100 500 0 0 0 8:47
K2FibFlowCache adj r 2.00 0.01 10 0 100 500 0 0 0 8:47
K2MetStatsMan Review 2.00 0.07 5 2 100 500 0 0 0 55:00
K2QosDblMan Rate DBL 2.00 0.18 7 0 100 500 0 0 0 170:57
IrmFibThrottler Thro 2.00 0.01 7 0 100 500 0 0 0 9:00
K2 VlanStatsMan Revi 2.00 0.13 15 1 100 500 0 0 0 199:13
K2 Packet Memory Dia 2.00 0.04 15 8 100 500 0 0 0 754:50
K2 L2 Aging Table Re 2.00 0.03 20 2 100 500 0 0 0 138:37
K2NetflowMan - hw fl 5.00 0.03 20 1 100 500 0 0 0 29:32
K2PacketBufMonitor-P 1.00 2.58 10 1 100 500 2 2 1 2499:28
RkiosPortMan Port Re 2.00 0.05 12 8 100 500 0 0 0 115:32
Rkios Module State R 4.00 0.01 40 1 100 500 0 0 0 29:42
Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 27:44
RkiosIpPbr IrmPort R 2.00 0.14 10 2 100 500 0 0 0 44:24
RkiosAclMan Review 3.00 0.03 30 0 100 500 0 0 0 56:38
EthHoleLinecardMan(1 1.66 0.01 10 24 100 500 0 0 0 31:04
EthHoleLinecardMan(3 1.66 0.13 10 4 100 500 0 0 0 235:10
-------------
%CPU Totals 239.93 9.50
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 9% 24.63 100%
Linecard 3's Store 258.00 9% 24.79 100%
Linecard 4's Store 258.00 65% 168.57 100%
TSM objects ------------------ ------------------
PacketBufRaw 25219.50 100% 25219.50 100%
PacketBufRawJumbo 732.81 25% 183.20 100%
Packet 1158.75 2% 30.11 99%
PacketInfoItem 390.62 0% 0.01 0%
VbufNodes1600 55.75 0% 12.19 0%
VbufNodes400 292.00 0% 5.70 30%
VbufNodes64 62.00 0% 1.93 0%
MatEntryTableIterato 1.12 0% 0.03 0%
Rkios QoS ClassMaps 896.00 0% 0.02 100%
PimPhyports 882.81 25% 226.00 100%
PimPorts 796.87 32% 255.00 100%
PimSlots 5.00 2% 0.13 100%
PimChassis 34.37 6% 2.14 100%
EbmPorts 184.00 31% 57.32 100%
EbmIeNodes 536.00 0% 4.18 100%
EbmSortedHostTableIt 1.87 0% 0.05 0%
EbmSortedGroupTableI 1.75 0% 0.05 0%
IrmVrfs 27.00 0% 0.10 100%
IrmFibAdjs 768.00 0% 0.46 100%
AclClassifier 768.00 0% 0.18 100%
AclFeature 1216.82 0% 0.42 100%
Acl 384.00 0% 0.10 100%
AclListNode 256.00 0% 0.06 100%
AclClassifierActionL 512.00 0% 0.09 100%
CommandTables 48.00 23% 11.10 100%
K2FibVrfs 152.00 0% 0.59 100%
K2TxPacket 384.00 0% 0.02 0%
K2TxPacketInfo 256.00 0% 0.03 0%
EbmVlanHostEntrys 3437.50 0% 2.37 94%
GalGbicEntrys 11.85 0% 0.09 100%
Event Nodes 128.00 0% 3.15 1%
Event Nodes 128.00 1% 7.15 22%
RkiosNetflowFlows 3072.00 0% 4.62 0%
------------------ ------------------
TSM totals 194196.69 13% 26043.75 99%
05-26-2013 09:55 PM
Hi Dao,
Is the CPU high right now? When you took this output?
Regards
Inayath
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: