Problem with CPU on Cat 4507

Dao Quynh · ‎05-14-2013

hi all,

Our Customer used Distribution switches are DistSW4507-01 and DistSW4507-02. These switches used HSRP with Dist-02 is active, Dist-01 is standby.

But, the past few days then the CPU of Dist-01 increase to 60%, you can see result as follow:

DistSW-4507-01#sh processes cpu hi

54515 5 5 5 5 5 545 5 5 5 515 5 515 5 5 5 5 5 5 515 5 515

9414049494949494941494949494049494049494949494949404949404

100

90

80

70

60

50 * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

40 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

30 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

20 *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

5555555555555555555555555555555555555555554554444445555545444444444444

5555454554444343443332233222221211211000009009999980000090999998877787

100

90

80

70

60 **** * **

50 **********************************************************************

40 **********************************************************************

30 **********************************************************************

20 **********************************************************************

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

DistSW-4507-01#sh processes cpu sorted

CPU utilization for five seconds: 54%/0%; one minute: 11%; five minutes: 9%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

41 263762272 258671478 1019 52.15% 8.30% 6.38% 0 Cat4k Mgmt LoPri

40 68970600 388841616 177 2.07% 2.05% 2.05% 0 Cat4k Mgmt HiPri

85 1575364 11474716 137 0.07% 0.02% 0.00% 0 Spanning Tree

69 40 7066609 0 0.07% 0.00% 0.00% 0 UDLD

82 2082608 61332016 33 0.07% 0.04% 0.03% 0 IP Input

135 1170540 54087423 21 0.07% 0.05% 0.07% 0 HSRP (Standby)

7 5794312 955116 6066 0.00% 0.06% 0.05% 0 Check heaps

8 0 3 0 0.00% 0.00% 0.00% 0 Pool Manager

Please help me resolve,

thank all,

Quynh

Dao Quynh · ‎05-14-2013

Normal,

CPU of the Dist-01 is very low as follow:

DistSW-4507-01#sh processes cpu sorted | exclude 0.00

CPU utilization for five seconds: 6%/0%; one minute: 7%; five minutes: 8%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

41 263764144 258675016 1019 3.43% 4.46% 5.60% 0 Cat4k Mgmt LoPri

40 68971748 388848159 177 2.15% 2.07% 2.05% 0 Cat4k Mgmt HiPri

135 1170540 54088269 21 0.07% 0.05% 0.07% 0 HSRP (Standby)

149 4660 64237 72 0.07% 0.11% 0.27% 1 Virtual Exec

77 1652764 3311459 499 0.07% 0.09% 0.08% 0 CDP Protocol

InayathUlla Sharieff · ‎05-14-2013

Hi Dao,

Could you please provide me with the following outputs:

show process cpu sorted | ex 0.00%

Show platform cpu packet statistics

Show platform health | ex 0.00

Also the below link has very good document on troubleshooting high cpu on cat 4500 kindly go through the same which help you in finding the root cause:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2

HTH

Regards

Inayath

*Plz rate all usefull posts.

Prasan Venky · ‎05-14-2013

Mgmt High & Low priority process'' processes control plane as well as data packets that need to be software-switched or processed.Try the following.

Hardcode the L3 interfaces with the commands no ip redirect, no ip unreachable and no ip proxy-arp. It should come to normal.

--Pls rate helpful posts--

Dao Quynh · ‎05-26-2013

hi InayathUlla Sharieff,

These are the result of the commands:

DistSW-4507-01#show platform cpu packet statistics

Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3

------------ --------------- --------------- --------------- ---------------

0 0 0 0 287751

RkiosSysPacketMan:

Packet allocation falures: 0

Packet Buffer(Software Common) allocation falures: 0

Packet Buffer(Software ESMP) allocation falures: 0

Packet Buffer(Software EOBC) allocation falures: 0

IOS Packet Buffer Wrapper allocation falures: 0

Packets Dropped In Processing Overall

Total 5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

431 0 0 0 0

Packets Dropped In Processing by CPU event

Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Input Acl 420 0 0 0 0

SA Miss 11 0 0 0 0

Packets Dropped In Processing by Priority

Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Medium 11 0 0 0 0

High 420 0 0 0 0

Packets Dropped In Processing by Reason

Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt 8 0 0 0 0

L2DstDrop 3 0 0 0 0

NoFloodPorts 420 0 0 0 0

Total packet queues 16

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Esmp 166023605 25 18 13 6

L2/L3Control 99934266 15 7 3 0

Host Learning 4353 0 0 0 0

L3 Fwd Low 9 0 0 0 0

L2 Fwd Low 331699 0 0 0 0

L3 Rx Low 2333469 2 0 0 0

DistSW-4507-01#show platform health | exclude 0.00

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

GalChassisVp-review 3.00 0.14 10 23 100 500 0 0 0 147:38

Lj-poll 1.00 0.02 2 0 100 500 0 0 0 27:18

StatValueMan Update 1.00 0.14 1 0 100 500 0 0 0 59:21

K2AclCamMan hw stats 3.00 0.55 10 5 100 500 0 0 0 437:25

K2AclPolicerTableMan 1.00 0.84 10 8 100 500 0 0 0 518:21

K2PortMan Review 2.00 0.87 15 10 100 500 0 1 0 1032:56

Gigaport9 Review 0.40 0.05 4 1 100 500 0 0 0 76:47

Gigaport11 Review 0.40 0.05 4 1 100 500 0 0 0 77:06

Gigaport16 Review 0.40 0.04 4 1 100 500 0 0 0 72:41

Gigaport18 Review 0.40 0.04 4 1 100 500 0 0 0 72:46

Gigaport25 Review 0.40 0.04 4 1 100 500 0 0 0 72:48

Gigaport39 Review 0.40 0.04 4 1 100 500 0 0 0 72:46

Gigaport48 Review 0.40 0.04 4 1 100 500 0 0 0 71:04

Gigaport49 Review 0.40 0.04 4 1 100 500 0 0 0 69:14

Gigaport50 Review 0.40 0.04 4 1 100 500 0 0 0 70:43

Gigaport51 Review 0.40 0.05 4 1 100 500 0 0 0 73:51

Gigaport52 Review 0.40 0.04 4 1 100 500 0 0 0 68:05

Gigaport53 Review 0.40 0.05 4 0 100 500 0 0 0 74:59

K2Fib Vrf Unpunt Rev 2.00 0.01 15 0 100 500 0 0 0 9:06

K2FibAdjMan Stats Re 2.00 0.50 10 8 100 500 0 0 0 213:07

K2FibMulticast Signa 2.00 0.04 10 1 100 500 0 0 0 39:10

K2FibPbr route map r 2.00 0.01 20 4 100 500 0 0 0 153:35

K2FibPbr consolidati 2.00 0.02 10 0 100 500 0 0 0 12:00

K2FibFlowCache flow 2.00 0.01 10 0 100 500 0 0 0 8:47

K2FibFlowCache adj r 2.00 0.01 10 0 100 500 0 0 0 8:47

K2MetStatsMan Review 2.00 0.07 5 2 100 500 0 0 0 55:00

K2QosDblMan Rate DBL 2.00 0.18 7 0 100 500 0 0 0 170:57

IrmFibThrottler Thro 2.00 0.01 7 0 100 500 0 0 0 9:00

K2 VlanStatsMan Revi 2.00 0.13 15 1 100 500 0 0 0 199:13

K2 Packet Memory Dia 2.00 0.04 15 8 100 500 0 0 0 754:50

K2 L2 Aging Table Re 2.00 0.03 20 2 100 500 0 0 0 138:37

K2NetflowMan - hw fl 5.00 0.03 20 1 100 500 0 0 0 29:32

K2PacketBufMonitor-P 1.00 2.58 10 1 100 500 2 2 1 2499:28

RkiosPortMan Port Re 2.00 0.05 12 8 100 500 0 0 0 115:32

Rkios Module State R 4.00 0.01 40 1 100 500 0 0 0 29:42

Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 27:44

RkiosIpPbr IrmPort R 2.00 0.14 10 2 100 500 0 0 0 44:24

RkiosAclMan Review 3.00 0.03 30 0 100 500 0 0 0 56:38

EthHoleLinecardMan(1 1.66 0.01 10 24 100 500 0 0 0 31:04

EthHoleLinecardMan(3 1.66 0.13 10 4 100 500 0 0 0 235:10

-------------

%CPU Totals 239.93 9.50

Allocation ceiling Current allocation

------------------ ------------------

kbytes % in use kbytes % in use

Linecard 1's Store 258.00 9% 24.63 100%

Linecard 3's Store 258.00 9% 24.79 100%

Linecard 4's Store 258.00 65% 168.57 100%

TSM objects ------------------ ------------------

PacketBufRaw 25219.50 100% 25219.50 100%

PacketBufRawJumbo 732.81 25% 183.20 100%

Packet 1158.75 2% 30.11 99%

PacketInfoItem 390.62 0% 0.01 0%

VbufNodes1600 55.75 0% 12.19 0%

VbufNodes400 292.00 0% 5.70 30%

VbufNodes64 62.00 0% 1.93 0%

MatEntryTableIterato 1.12 0% 0.03 0%

Rkios QoS ClassMaps 896.00 0% 0.02 100%

PimPhyports 882.81 25% 226.00 100%

PimPorts 796.87 32% 255.00 100%

PimSlots 5.00 2% 0.13 100%

PimChassis 34.37 6% 2.14 100%

EbmPorts 184.00 31% 57.32 100%

EbmIeNodes 536.00 0% 4.18 100%

EbmSortedHostTableIt 1.87 0% 0.05 0%

EbmSortedGroupTableI 1.75 0% 0.05 0%

IrmVrfs 27.00 0% 0.10 100%

IrmFibAdjs 768.00 0% 0.46 100%

AclClassifier 768.00 0% 0.18 100%

AclFeature 1216.82 0% 0.42 100%

Acl 384.00 0% 0.10 100%

AclListNode 256.00 0% 0.06 100%

AclClassifierActionL 512.00 0% 0.09 100%

CommandTables 48.00 23% 11.10 100%

K2FibVrfs 152.00 0% 0.59 100%

K2TxPacket 384.00 0% 0.02 0%

K2TxPacketInfo 256.00 0% 0.03 0%

EbmVlanHostEntrys 3437.50 0% 2.37 94%

GalGbicEntrys 11.85 0% 0.09 100%

Event Nodes 128.00 0% 3.15 1%

Event Nodes 128.00 1% 7.15 22%

RkiosNetflowFlows 3072.00 0% 4.62 0%

------------------ ------------------

TSM totals 194196.69 13% 26043.75 99%

InayathUlla Sharieff · ‎05-26-2013

Hi Dao,

Is the CPU high right now? When you took this output?

Regards

Inayath