Re: High CPU utilization on Catalyst WS-C4510R - SupV 10GE

skolaric1 · ‎12-13-2012

Hello @ll,

I´m stuck in a problem of high CPU utilization on company core SW (Catalyst WS-C4510R - SupV 10GE).

I did found a Cisco documentation but dont know all the tweeks to go in the right direction, so please advise.

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

Problem description:

-----------------------------

In the tipical work time CPU is going towards 90-100%.

Cat4510R#sho proc cpu hist

7777778888888888666667777766666777776666655555888885555577

6111115555599999777772222255555888885555566666222226666677

100

90 **********

80 * ********** ***** ***** **

70 ***************************************** ***** **

60 **********************************************************

50 **********************************************************

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

8888788997979888988899998899999988899998988977799999999889

2410835004074868977201938699451358387988322984829999936953

100 * * ** * **** * *#### *

90 *** * ******* *******#****** **#*** * **####*****

80 ********* ************##**##*#*#***#*####**** **########*#

70 ###*##*############*#####################*##*#*###########

60 ##########################################################

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

9797777777779999999999999887778677779999999999999676677678999999999999

9391544523251878898999999523221965302268898889999987711713044579898999

100 * * ************ *********** *********

90 * * ****#######*** ****#######** *******#****

80 * * * * ***##########*** * ** ***########** * *****########

70 *************###########**************#########***************########

60 #***********#############************###########*************#########

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

So the process table is showing that a Cat4k Mgmt LoPri process is to high!

Cat4510R#sho processes cpu sorted | ex 0.00

CPU utilization for five seconds: 72%/0%; one minute: 70%; five minutes: 69%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

59 29609297091294930243 2286 56.51% 55.03% 54.07% 0 Cat4k Mgmt LoPri

58 10723807763629619862 0 10.39% 10.21% 10.31% 0 Cat4k Mgmt HiPri

122 179994293 179943139 1000 1.27% 1.27% 1.27% 0 Spanning Tree

14 78054756 307767893 253 0.95% 0.80% 0.81% 0 ARP Input

97 14644995 150039341 97 0.23% 0.22% 0.23% 0 UDLD

244 18308895 866421282 21 0.23% 0.23% 0.23% 0 HSRP Common

39 42524100 15006257 2833 0.23% 0.23% 0.23% 0 IDB Work

115 34837482 408724320 85 0.23% 0.25% 0.24% 0 IP Input

165 14539549 34482044 421 0.15% 0.17% 0.16% 0 CEF: IPv4 proces

42 33422178 181259402 184 0.15% 0.23% 0.23% 0 Per-Second Jobs

81 115126332183926165 0 0.15% 0.11% 0.10% 0 Ethernet Msec Ti

121 304 223 1363 0.15% 0.18% 0.08% 1 SSH Process

239 10015807 61202677 163 0.07% 0.12% 0.10% 0 SNMP ENGINE

247 111225081674456698 6 0.07% 0.12% 0.13% 0 IP SLAs Event Pr

237 11940919 164510615 72 0.07% 0.10% 0.09% 0 IP SNMP

108 12804707 37625172 340 0.07% 0.09% 0.06% 0 CDP Protocol

Cat4510R#

As the next "sho platform health" command is showing the "K2L2 Address Table R" is doing the CPU hard time.

But where I can find more on this processes? What is the meaning?

Cat4510R#show platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

GalChassisVp-review 3.00 0.15 10 20 100 500 0 0 0 339:56

S2w-JobEventSchedule 10.00 2.09 10 24 100 500 1 2 1 4271:21

Stub-JobEventSchedul 10.00 3.40 10 26 100 500 2 3 2 7013:47

Lj-poll 1.00 0.02 2 0 100 500 0 0 0 54:47

StatValueMan Update 1.00 0.10 1 0 100 500 0 0 0 181:21

Pim-review 0.10 0.00 1 0 100 500 0 0 0 16:51

Ebm-host-review 1.00 0.01 8 4 100 500 0 0 0 98:44

Ebm-host-util-review 1.00 0.00 10 0 100 500 0 0 0 0:27

Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 1:21

Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:05

EbmHostRedundancyMan 2.00 0.00 20 0 100 500 0 0 0 0:00

Acl-Flattener 1.00 0.00 10 0 100 500 0 0 0 0:00

GalChassisVp Ondeman 2.00 0.00 2 0 100 500 0 0 0 0:00

KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 10:33

KxAclPathMan update 2.00 0.00 10 5 100 500 0 0 0 17:23

KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00

TagMan-RecreateMtegR 1.00 0.00 10 5 100 500 0 0 0 0:00

K2CpuMan Review 30.00 7.72 30 45 100 500 7 8 6 13975:35

K2AccelPacketMan: Tx 10.00 4.18 20 0 100 500 3 4 3 5792:44

K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00

K2AclMan-labeledFlat 1.00 0.00 10 1 100 500 0 0 0 0:00

K2AclCamMan stale en 1.00 0.00 10 5 100 500 0 0 0 0:00

K2AclCamMan consolid 1.00 0.00 10 0 100 500 0 0 0 0:00

K2AclCamMan hw stats 3.00 0.38 10 5 100 500 0 0 0 838:03

K2AclCamMan kx stats 1.00 0.67 10 5 100 500 0 0 0 307:02

K2AclCamMan Audit re 1.00 0.00 10 0 100 500 0 0 0 0:11

K2AclPolicerTableMan 1.00 0.24 10 5 100 500 0 0 0 943:00

K2L2 Address Table R 2.00 49.74 12 5 100 500 66 69 42 110622:49

K2L2 New Static Addr 2.00 0.00 10 2 100 500 0 0 0 1:56

K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 21:50

K2L2 Dynamic Address 2.00 0.00 10 5 100 500 0 0 0 17:27

K2L2 PPE Vlan Table 2.00 0.17 10 5 100 500 0 0 0 780:33

K2L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 125:18

K2PortMan Review 3.00 2.81 15 10 100 500 2 3 2 6262:07

Gigaport0 Review 0.40 0.04 4 1 100 500 0 0 0 138:28

Gigaport1 Review 0.40 0.00 4 1 100 500 0 0 0 23:24

Gigaport2 Review 0.40 0.00 4 1 100 500 0 0 0 22:24

Gigaport3 Review 0.40 0.04 4 11 100 500 0 0 0 142:21

Gigaport4 Review 0.40 0.00 4 1 100 500 0 0 0 23:24

Gigaport5 Review 0.40 0.04 4 1 100 500 0 0 0 136:37

Gigaport6 Review 0.40 0.04 4 1 100 500 0 0 0 135:57

Gigaport7 Review 0.40 0.04 4 1 100 500 0 0 0 136:41

Gigaport8 Review 0.40 0.05 4 1 100 500 0 0 0 138:11

Gigaport9 Review 0.40 0.38 4 1 100 500 0 0 0 711:44

Gigaport10 Review 0.40 0.01 4 1 100 500 0 0 0 22:21

Gigaport11 Review 0.40 0.04 4 1 100 500 0 0 0 138:22

Gigaport12 Review 0.40 0.00 4 1 100 500 0 0 0 24:02

Gigaport13 Review 0.40 0.04 4 12 100 500 0 0 0 143:25

Gigaport14 Review 0.40 0.00 4 1 100 500 0 0 0 22:19

Gigaport15 Review 0.40 0.00 4 1 100 500 0 0 0 22:18

Gigaport16 Review 0.40 0.00 4 1 100 500 0 0 0 22:23

Gigaport17 Review 0.40 0.01 4 1 100 500 0 0 0 22:21

Gigaport18 Review 0.40 0.00 4 1 100 500 0 0 0 23:24

Gigaport19 Review 0.40 0.04 4 14 100 500 0 0 0 145:32

Gigaport20 Review 0.40 0.01 4 1 100 500 0 0 0 22:20

Gigaport21 Review 0.40 0.04 4 1 100 500 0 0 0 135:20

Gigaport22 Review 0.40 0.04 4 1 100 500 0 0 0 139:44

Gigaport23 Review 0.40 0.01 4 1 100 500 0 0 0 22:19

Gigaport24 Review 0.40 0.01 4 1 100 500 0 0 0 24:01

Gigaport25 Review 0.40 0.00 4 1 100 500 0 0 0 22:24

Gigaport26 Review 0.40 0.04 4 13 100 500 0 0 0 140:43

Gigaport27 Review 0.40 0.01 4 1 100 500 0 0 0 22:23

Gigaport28 Review 0.40 0.04 4 14 100 500 0 0 0 141:21

Gigaport29 Review 0.40 0.00 4 1 100 500 0 0 0 24:05

Gigaport30 Review 0.40 0.01 4 1 100 500 0 0 0 23:18

Gigaport31 Review 0.40 0.00 4 1 100 500 0 0 0 23:19

Gigaport32 Review 0.40 0.04 4 1 100 500 0 0 0 136:33

Gigaport33 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport34 Review 0.40 0.01 4 1 100 500 0 0 0 22:21

Gigaport35 Review 0.40 0.00 4 1 100 500 0 0 0 23:20

Gigaport36 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport37 Review 0.40 0.01 4 1 100 500 0 0 0 22:26

Gigaport38 Review 0.40 0.00 4 1 100 500 0 0 0 22:25

Gigaport39 Review 0.40 0.00 4 1 100 500 0 0 0 24:05

Gigaport40 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport41 Review 0.40 0.01 4 1 100 500 0 0 0 22:21

Gigaport42 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport43 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport44 Review 0.40 0.01 4 1 100 500 0 0 0 22:20

Gigaport45 Review 0.40 0.04 4 1 100 500 0 0 0 136:16

Gigaport46 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport47 Review 0.40 0.01 4 1 100 500 0 0 0 22:19

Gigaport48 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport49 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport50 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport51 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport52 Review 0.40 0.31 4 1 100 500 0 0 0 628:30

Gigaport53 Review 0.40 0.31 4 19 100 500 0 0 0 627:08

K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:00

K2Fib IrmFib Review 2.00 0.00 15 7 100 500 0 0 0 0:00

K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:07

K2Fib Vrf Unpunt Rev 2.00 0.01 15 0 100 500 0 0 0 18:22

K2Fib Consistency Ch 1.00 1.36 5 2 100 500 7 1 0 2405:58

K2FibAdjMan Stats Re 2.00 0.60 10 7 100 500 0 0 0 606:52

K2FibAdjMan Host Mov 2.00 0.28 10 7 100 500 0 0 0 2047:23

K2FibAdjMan Adj Chan 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibMulticast Signa 2.00 0.01 10 1 100 500 0 0 0 59:47

K2FibMulticast Entry 2.00 0.00 10 4 100 500 0 0 0 0:38

K2FibMulticast Irm M 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 0:00

K2FibPbr route map r 2.00 0.59 20 5 100 500 0 0 0 622:54

K2FibPbr flat acl pr 2.00 0.03 20 1 100 500 0 0 0 109:39

K2FibPbr consolidati 2.00 0.01 10 0 100 500 0 0 0 24:29

K2FibPerVlanPuntMan 2.00 0.00 15 2 100 500 0 0 0 0:00

K2FibWccpMan Service 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan Cache E 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan Redirec 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan AdjRepo 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibFC Aging review 2.00 0.01 10 0 100 500 0 0 0 17:43

K2FibFC DelFlow revi 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibFC Adj review 2.00 0.01 10 0 100 500 0 0 0 17:39

K2FibFC overload/rel 2.00 0.00 10 0 100 500 0 0 0 5:24

K2MetStatsMan Review 2.00 0.03 5 2 100 500 0 0 0 109:29

K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00

K2QosDblMan Rate DBL 2.00 0.18 7 1 100 500 0 0 0 383:09

K2 VlanStatsMan Revi 2.00 0.19 15 1 100 500 0 0 0 590:09

K2 Packet Memory Dia 2.00 0.00 15 8 100 500 0 0 0 1461:58

K2 L2 Aging Table Re 2.00 0.06 20 2 100 500 0 0 0 274:58

K2NetflowMan - hw fl 5.00 0.57 20 11 100 500 0 0 0 938:28

K2FibEntry activity 1.00 0.23 10 5 100 500 0 0 0 353:17

K2PacketBufMonitor-P 3.00 0.58 10 0 100 500 0 0 0 1139:35

RkiosPortMan Port Re 2.00 0.23 12 11 100 500 0 0 0 865:23

Rkios Module State R 4.00 0.01 40 6 100 500 0 0 0 62:26

Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 60:17

MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00

RkiosIpPbr IrmPort R 2.00 0.02 10 2 100 500 0 0 0 102:30

RkiosIpWccp IrmWccpW 2.00 0.01 10 0 100 500 0 0 0 10:57

RkiosAclMan Review 3.00 0.03 30 0 100 500 0 0 0 114:48

EthHoleLinecardMan(1 1.11 0.02 10 24 100 500 0 0 0 64:49

EthHoleLinecardMan(3 1.11 0.23 10 5 100 500 0 0 0 554:20

EthHoleLinecardMan(4 1.11 0.55 10 4 100 500 0 0 0 1634:33

EthHoleLinecardMan(5 1.11 0.88 10 5 100 500 0 0 0 1635:00

EthHoleLinecardMan(6 1.11 0.20 10 4 100 500 0 0 0 554:21

GalLinecardDiag Mana 1.00 0.00 10 0 100 500 0 0 0 0:00

-------------

%CPU Totals 216.15 80.71

Allocation ceiling Current allocation

------------------ ------------------

kbytes % in use kbytes % in use

Linecard 1's Store 258.00 17% 44.07 100%

Linecard 2's Store 258.00 0% 0.00 0%

Linecard 3's Store 258.00 15% 41.16 100%

Linecard 4's Store 258.00 38% 100.26 100%

Linecard 5's Store 258.00 38% 100.26 100%

Linecard 6's Store 258.00 15% 41.17 100%

Linecard 7's Store 258.00 84% 217.13 100%

Linecard 8's Store 258.00 74% 191.85 100%

Linecard 9's Store 258.00 84% 217.13 100%

Linecard 10's Store 258.00 0% 0.00 0%

TSM objects ------------------ ------------------

PacketInfoItem 781.25 0% 5.62 0%

VbufNodes2400 80.50 0% 0.00 0%

VbufNodes1600 55.50 0% 12.14 0%

VbufNodes400 288.00 1% 11.25 40%

VbufNodes64 60.00 0% 6.56 0%

VbufNodes4200 68.37 0% 0.00 0%

PacketBufRaw 25219.50 100% 25219.50 100%

PacketBufRawJumbo 184.29 100% 184.29 100%

RkiosSysPacketBuf 250.00 0% 0.81 0%

Packet 1925.39 2% 50.29 99%

MatEntrys 9600.00 0% 6.44 63%

MatEntryTableIterato 1.00 0% 0.03 0%

PimPhyports 1093.75 40% 437.50 100%

PimPorts 921.87 46% 427.75 100%

PimModules 262.00 3% 8.18 100%

PimSlots 18.00 3% 0.70 100%

PimChassis 39.87 6% 2.49 100%

PimQuack 1.64 0% 0.00 0%

EbmVlans 5408.00 0% 40.92 100%

EbmVlanGroupEntrys 3968.00 0% 4.23 77%

EbmPorts 392.00 45% 177.24 100%

EbmPortHostEntrys 448.00 0% 0.00 0%

EbmIeNodes 540.00 0% 5.27 100%

EbmPortVlanAclFeatur 3072.00 0% 0.00 0%

EbmPortVlanMap Alloc 64.00 0% 0.00 0%

EbmSortedHostTableIt 1.75 0% 0.05 0%

EbmSortedGroupTableI 1.75 0% 0.05 0%

EbmHostRedundancyMan 1082.81 0% 0.00 0%

IrmVrfs 318.00 0% 2.48 100%

IrmFibLoadBalances 1280.00 0% 0.11 66%

IrmFibAdjs 1920.00 3% 61.60 95%

IrmPortEtherAddrEntr 500.00 0% 0.15 20%

IrmFibEntries 12288.00 0% 77.81 95%

IrmMfibEntryMemMan 6656.00 0% 0.40 100%

IrmWccpMemMan 34.25 0% 0.00 0%

IrmWccpServiceGroupL 0.06 0% 0.00 0%

AclOp 2176.00 0% 0.00 0%

AclOpAceSet 3264.00 0% 0.00 0%

AclClassifier 1280.00 0% 1.60 100%

AclFeature 2574.79 0% 2.07 100%

Acl 1536.00 0% 1.96 100%

Ace24 10880.00 0% 6.01 100%

Ace48 17408.00 0% 0.37 100%

AclFlowLabelListNode 7616.00 0% 0.00 0%

AceActionDescStorage 1088.00 0% 0.00 0%

AclListNode 512.00 0% 0.32 97%

AceListNode 25600.00 0% 0.26 26%

AclClassifierActionL 2560.00 0% 1.48 97%

AclLayerFeatureListN 512.00 0% 0.07 44%

AclClassifierListNod 512.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 128.00 0% 0.00 0%

FlatAcl 512.00 0% 0.12 25%

FlatAce24 11264.00 0% 2.57 30%

FlatAce48 17408.00 0% 0.00 0%

FlatAceActionListNod 307200.00 0% 1.45 29%

FlatAclOpSetStorage 3072.00 0% 0.00 0%

FlatAclCacheNode 4224.00 0% 0.25 100%

FlatAclListNode 256.00 0% 0.04 0%

QosFeatureClassifier 343.51 0% 0.00 0%

QosFeatureClassifier 687.03 0% 0.00 0%

QosClassifierActionL 9618.43 0% 0.00 0%

QosNestedClassifierA 15114.68 0% 2.01 100%

QosPortVlanAclFeatur 1224.00 0% 0.00 0%

QoS Policers 37000.00 0% 0.00 0%

Qos FlowFnf 7.81 0% 0.00 0%

KxAclPath 1024.00 0% 1.40 100%

KxAclPathListNode 1280.00 0% 0.00 0%

KxAclConstPathListNo 1280.00 0% 0.34 0%

IndexCache 800.78 0% 0.00 0%

ConfigToHwAfMap 203.27 0% 0.00 0%

HwToCfgAceMap 816.00 0% 0.00 0%

CommandTables 48.00 22% 10.59 100%

K2FibWccpServiceGrou 162.59 0% 0.00 0%

K2FibPbrFlatRouteMap 498.42 0% 0.00 0%

K2FibPbrExpandedFlat 2304.00 0% 0.00 0%

K2FibPbrFlatRouteMap 320.00 0% 0.00 0%

K2FibVrfs 152.00 0% 1.18 100%

K2 Netflow FPT Entri 11.97 0% 0.00 0%

K2TxPacket 384.00 0% 0.02 0%

K2TxPacketInfo 256.00 0% 2.26 0%

KxAclLabeledFlatAcl 3584.00 0% 0.21 100%

KxAclLabeledFlatAclE 3072.00 0% 0.09 100%

EbmVlanHostEntrys 3222.65 2% 106.23 77%

GalGbicEntrys 18.60 1% 0.28 83%

Rkios QoS PolicyMaps 357.57 0% 0.03 100%

AclClassifierIdToCla 48.00 0% 0.01 100%

Rkios QoS ClassMaps 1024.00 0% 0.03 100%

AclToIosFilterMapLis 384.00 0% 0.01 100%

Rkios QoS Policers 3500.00 0% 0.00 0%

RkiosAclMan NamedGal 43.75 0% 0.00 0%

EpmPolicyListNode 120.00 0% 0.00 0%

EpmAceListNode 192.00 0% 0.00 0%

RkiosAclSecurityEpmP 4080.00 0% 0.00 0%

Rkios Acl VlanMaps 128.00 0% 0.00 0%

Rkios Acl VlanMapEnt 937.50 0% 0.00 0%

RkiosTableMap Galios 3.00 0% 0.00 0%

IrmMfibIntrfs 3072.00 0% 0.00 0%

Event Nodes 160.00 0% 3.53 24%

Event Nodes 160.00 1% 15.62 16%

RkiosNetflowFlows 3072.00 0% 176.35 7%

IrmFlows 384.00 0% 0.00 0%

K2AclMappingTableEnt 106.82 0% 0.02 0%

K2AclMappingTableEnt 71.21 0% 0.00 0%

K2AclTagPairSignatur 1792.00 0% 0.10 100%

RkisoIpPbrRouteMaps 97.65 0% 0.00 0%

K2 Microflow Policer 383.81 0% 0.00 0%

------------------ ------------------

TSM totals 603986.51 4% 27083.01 99%

Cat4510R#

I did also some sniffing and enabled debugging with in-build CPU cniffer tool and stored the result in CPUHOG-Buffers.txt file (attached) but have no idea to look for a hint to solve the problem.

My suspection is that due to a flat colapsed core architecture (Core:C4510R; acces L2 switches) some of the server applications is responsible of the CPUHOG issue.

Regards,

Silvi.

skolaric1 · ‎01-24-2013

The issue is solved, commands I have used to clarify the cause:

1) mac address-table notification mac-move

2) sho logg

As described here: http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml