Ask the Expert:Troubleshooting High CPU and Other Issues in the

ciscomoderator · ‎10-05-2012

With Nickolay Karpyshev and Ivan Shirshin

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about architecture and troubleshooting of Industry's Most Widely Deployed Modular Access Platform Cisco Catalyst 4500 with Cisco Experts Nikolay Karpyshev and Ivan Shirshin.

Nikolay and Ivan are Customer Support Engineers in the high touch technology support team (HTTS) at Cisco specialized in LAN Switching and Routing. They support the Cisco Switches Nexus 7000, Catalyst 6500, 3750, 3560, 4500, 2900 and variety of routing platforms, and work as senior and escalation engineers. Both Nikolay and Ivan were previously a part of Cisco Sales Associate program. They hold Cisco Certifications: CCNP, CCSP, and CCDP.

Remember to use the rating system to let Nikolay and Ivan know if you have received an adequate response.

Nikolay and Ivan might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Network Infrastructure sub-community discussion forum shortly after the event. This event lasts through through October 19, 2012. Visit this forum often to view responses to your questions and the questions of other community members.

Claus Andersen · ‎10-08-2012

Hi Experts

I have some problems with the 4500 platform and high cpu load.

normaly i will use

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tcam to troubleshoot the issue, but i always seems to hit a dead end.

Here is some output:

xxxx-xxx-xx#sh proc cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

49 24659898802830920636 0 86.10% 85.93% 85.85% 0 Cat4k Mgmt LoPri

48 1459019441878837282 77 11.74% 11.69% 11.71% 0 Cat4k Mgmt HiPri

36 406413592 95089543 4274 0.55% 0.50% 0.49% 0 IDB Work

39 73667932 102528426 718 0.23% 0.12% 0.11% 0 Per-Second Jobs

169 164910162928907666 1 0.15% 0.14% 0.15% 0 PM Callback

3 56 65 861 0.07% 0.08% 0.01% 1 SSH Process

218 1068680 83542081 12 0.07% 0.00% 0.00% 0 EIGRP-IPv4: Hell

84 1900108 101130341 18 0.07% 0.02% 0.02% 0 UDLD

201 25186108 202200624 124 0.07% 0.05% 0.07% 0 IGMP Input

102 65327852 428594577 152 0.07% 0.03% 0.00% 0 IP Input

159 207344 696207 297 0.07% 0.05% 0.07% 0 DHCP Security He

15 58316156 153764063 379 0.07% 0.06% 0.07% 0 ARP Input

152 172752 1450236 119 0.07% 0.00% 0.00% 0 TPLUS

13 0 1 0 0.00% 0.00% 0.00% 0 ifIndex Receive

14 26960 2180488 12 0.00% 0.00% 0.00% 0 EEM ED Syslog

12 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

17 0 5 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

16 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API

11 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

20 96 241 398 0.00% 0.00% 0.00% 0 Entity MIB API

21 1304 186940 6 0.00% 0.00% 0.00% 0 IFS Agent Manage

22 500 1580218 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

23 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

24 114212 94744201 1 0.00% 0.00% 0.00% 0 IPC Periodic Tim

10 780 41 19024 0.00% 0.00% 0.00% 0 PrstVbl

9 0 2 0 0.00% 0.00% 0.00% 0 Timers

18 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

28 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv

19 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

30 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

25 0 1 0 0.00% 0.00% 0.00% 0 IPC Managed Time

32 142432 94744243 1 0.00% 0.00% 0.00% 0 Dynamic ARP Insp

33 119568 94744130 1 0.00% 0.00% 0.00% 0 GraphIt

34 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

35 716052 83086833 8 0.00% 0.01% 0.00% 0 Net Background

xxxx-xxx-xx#

xxxx-xxx-xx#sh platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

GalChassisVp-review 3.00 0.15 10 48 100 500 0 0 0 2114:40

S2w-JobEventSchedule 10.00 5.33 10 29 100 500 4 4 3 55466:23

Stub-JobEventSchedul 10.00 5.01 10 11 100 500 5 4 3 69132:27

Lj-poll 1.00 0.02 2 0 100 500 0 0 0 340:37

StatValueMan Update 1.00 0.03 1 0 100 500 0 0 0 1050:39

Pim-review 0.10 0.02 1 0 100 500 0 0 0 99:32

Ebm-host-review 1.00 0.14 8 3 100 500 0 0 0 249:44

Ebm-host-util-review 1.00 0.00 10 0 100 500 0 0 0 2:46

Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 8:26

Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:32

EbmHostRedundancyMan 2.00 0.00 20 0 100 500 0 0 0 0:00

Acl-Flattener 1.00 0.21 10 7 100 500 0 0 0 19:49

IrmFibThrottler Thro 2.00 0.00 7 5 100 500 0 0 0 132:24

GalChassisVp Ondeman 2.00 0.00 2 0 100 500 0 0 0 0:00

KxAclPathMan create/ 1.00 2.10 10 6 100 500 2 2 2 211:43

KxAclPathMan update 2.00 47.97 10 74 100 500 30 32 28 7308:38

KxAclPathMan reprogr 1.00 0.00 2 1 100 500 0 0 0 7:14

TagMan-RecreateMtegR 1.00 1.48 10 94 100 500 1 2 2 284:13

K2CpuMan Review 30.00 4.84 30 44 100 500 5 5 4 106894:03

K2AccelPacketMan: Tx 10.00 2.07 20 4 100 500 2 1 1 52344:25

K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00

K2AclMan-labeledFlat 1.00 3.15 10 37 100 500 1 6 5 400:26

K2AclCamMan stale en 1.00 9.41 10 93 100 500 21 18 16 2340:01

K2AclCamMan consolid 1.00 1.11 10 9 100 500 30 24 13 1914:38

K2AclCamMan hw stats 3.00 1.48 10 8 100 500 0 1 0 10858:55

K2AclCamMan kx stats 1.00 0.17 10 8 100 500 1 1 0 9470:11

K2AclCamMan Audit re 1.00 0.00 10 0 100 500 0 0 0 1:07

K2AclPolicerTableMan 1.00 0.24 10 5 100 500 0 0 0 7993:02

K2L2 Address Table R 2.00 17.48 12 7 100 500 14 14 10 4245:01

K2L2 New Static Addr 2.00 2.03 10 5 100 500 2 2 1 187:55

K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 80:08

K2L2 Dynamic Address 2.00 0.02 10 2 100 500 0 0 0 4:39

K2L2 Vlan Table Revi 2.00 0.64 12 8 100 500 0 0 0 86:57

K2PortMan Review 3.00 2.29 15 13 100 500 3 3 2 36205:34

Gigaport0 Review 0.40 0.00 4 2 100 500 0 0 0 139:58

Gigaport1 Review 0.40 0.01 4 2 100 500 0 0 0 137:38

Gigaport2 Review 0.40 0.01 4 2 100 500 0 0 0 137:32

Gigaport3 Review 0.40 0.00 4 2 100 500 0 0 0 137:28

Gigaport4 Review 0.40 0.01 4 2 100 500 0 0 0 137:38

Gigaport5 Review 0.40 0.01 4 2 100 500 0 0 0 137:43

Gigaport6 Review 0.40 0.00 4 2 100 500 0 0 0 137:42

Gigaport7 Review 0.40 0.01 4 2 100 500 0 0 0 137:25

Gigaport8 Review 0.40 0.00 4 2 100 500 0 0 0 137:30

Gigaport9 Review 0.40 0.00 4 2 100 500 0 0 0 137:33

Gigaport10 Review 0.40 0.00 4 2 100 500 0 0 0 142:18

Gigaport11 Review 0.40 0.00 4 2 100 500 0 0 0 137:27

Gigaport12 Review 0.40 0.00 4 2 100 500 0 0 0 145:41

Gigaport13 Review 0.40 0.00 4 2 100 500 0 0 0 137:40

Gigaport14 Review 0.40 0.01 4 2 100 500 0 0 0 137:20

Gigaport15 Review 0.40 0.01 4 2 100 500 0 0 0 137:27

Gigaport16 Review 0.40 0.01 4 2 100 500 0 0 0 141:32

Gigaport17 Review 0.40 0.00 4 2 100 500 0 0 0 142:20

Gigaport18 Review 0.40 0.01 4 2 100 500 0 0 0 137:33

Gigaport19 Review 0.40 0.00 4 2 100 500 0 0 0 140:26

Gigaport20 Review 0.40 0.00 4 2 100 500 0 0 0 142:12

Gigaport21 Review 0.40 0.00 4 2 100 500 0 0 0 139:14

Gigaport22 Review 0.40 0.00 4 2 100 500 0 0 0 139:24

Gigaport23 Review 0.40 0.00 4 2 100 500 0 0 0 142:13

Gigaport24 Review 0.40 0.00 4 2 100 500 0 0 0 137:33

Gigaport25 Review 0.40 0.01 4 2 100 500 0 0 0 137:59

Gigaport26 Review 0.40 0.01 4 2 100 500 0 0 0 138:06

Gigaport27 Review 0.40 0.00 4 2 100 500 0 0 0 142:42

Gigaport28 Review 0.40 0.01 4 2 100 500 0 0 0 137:42

Gigaport29 Review 0.40 0.01 4 2 100 500 0 0 0 138:15

Gigaport30 Review 0.40 0.00 4 2 100 500 0 0 0 144:18

Gigaport31 Review 0.40 0.01 4 2 100 500 0 0 0 137:49

Gigaport32 Review 0.40 0.00 4 2 100 500 0 0 0 137:39

Gigaport33 Review 0.40 0.00 4 2 100 500 0 0 0 146:07

Gigaport34 Review 0.40 0.00 4 2 100 500 0 0 0 137:26

Gigaport35 Review 0.40 0.01 4 2 100 500 0 0 0 137:31

Gigaport36 Review 0.40 0.00 4 2 100 500 0 0 0 141:23

Gigaport37 Review 0.40 0.00 4 2 100 500 0 0 0 137:33

Gigaport38 Review 0.40 0.01 4 2 100 500 0 0 0 138:07

Gigaport39 Review 0.40 0.01 4 2 100 500 0 0 0 144:49

Gigaport40 Review 0.40 0.00 4 2 100 500 0 0 0 143:27

Gigaport41 Review 0.40 0.00 4 2 100 500 0 0 0 142:49

Gigaport42 Review 0.40 0.00 4 2 100 500 0 0 0 141:56

Gigaport43 Review 0.40 0.00 4 2 100 500 0 0 0 143:45

Gigaport44 Review 0.40 0.00 4 2 100 500 0 0 0 140:52

Gigaport45 Review 0.40 0.00 4 2 100 500 0 0 0 137:33

Gigaport46 Review 0.40 0.00 4 2 100 500 0 0 0 140:46

Gigaport47 Review 0.40 0.00 4 2 100 500 0 0 0 137:30

Gigaport48 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport49 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport50 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport51 Review 0.40 0.00 4 0 100 500 0 0 0 0:00

Gigaport52 Review 0.40 0.38 4 4 100 500 0 0 0 4552:51

Gigaport53 Review 0.40 0.30 4 17 100 500 0 0 0 3891:56

K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:00

K2Fib IrmFib Review 2.00 0.00 15 4 100 500 0 0 0 0:00

K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:03

K2Fib Vrf Unpunt Rev 2.00 0.00 15 0 100 500 0 0 0 113:48

K2Fib Consistency Ch 1.00 0.00 5 6 100 500 2 3 2 36559:04

K2FibAdjMan Stats Re 2.00 0.05 10 6 100 500 0 0 0 2769:15

K2FibAdjMan Host Mov 2.00 0.44 10 4 100 500 0 0 0 95:43

K2FibAdjMan Adj Chan 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibMulticast Signa 2.00 0.09 10 1 100 500 0 0 0 375:37

K2FibMulticast Entry 2.00 0.00 10 7 100 500 0 0 0 593:15

K2FibMulticast Irm M 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 24:19

K2FibPbr route map r 2.00 0.07 20 7 100 500 0 0 0 3167:28

K2FibPbr flat acl pr 2.00 0.00 20 1 100 500 0 0 0 639:48

K2FibPbr consolidati 2.00 0.00 10 0 100 500 0 0 0 146:36

K2FibPerVlanPuntMan 2.00 0.00 15 2 100 500 0 0 0 0:00

K2FibWccpMan Service 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan Cache E 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan Redirec 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibWccpMan AdjRepo 2.00 0.00 20 0 100 500 0 0 0 0:00

K2FibFC Aging review 2.00 0.00 10 0 100 500 0 0 0 110:09

K2FibFC DelFlow revi 2.00 0.00 10 0 100 500 0 0 0 0:00

K2FibFC Adj review 2.00 0.00 10 0 100 500 0 0 0 109:52

K2FibFC overload/rel 2.00 0.00 10 0 100 500 0 0 0 33:33

K2MetStatsMan Review 2.00 0.20 5 2 100 500 0 0 0 753:11

K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00

K2QosDblMan Rate DBL 2.00 0.15 7 1 100 500 0 0 0 2307:44

K2 VlanStatsMan Revi 2.00 0.23 15 4 100 500 0 0 0 3060:11

K2 Packet Memory Dia 2.00 0.00 15 12 100 500 0 0 0 9342:50

K2 L2 Aging Table Re 2.00 0.04 20 2 100 500 0 0 0 1733:54

K2NetflowMan - hw fl 5.00 0.09 20 0 100 500 0 0 0 351:33

K2FibEntry activity 1.00 0.01 10 7 100 500 0 0 0 2094:31

K2PacketBufMonitor-P 3.00 0.52 10 4 100 500 0 0 0 7013:10

GbmHostTable Review 2.00 0.00 8 0 100 500 0 0 0 22:34

GbmHostTableUtilizat 2.00 0.00 8 0 100 500 0 0 0 0:00

GbmHostTableAging Re 2.00 0.00 8 0 100 500 0 0 0 33:35

RkiosPortMan Port Re 2.00 0.79 12 10 100 500 1 0 0 9792:46

Rkios Module State R 4.00 0.02 40 38 100 500 0 0 0 379:39

Rkios Online Diag Re 4.00 0.02 40 0 100 500 0 0 0 358:06

RkiosIpPbr IrmPort R 2.00 0.07 10 1 100 500 0 0 0 606:48

RkiosIpWccp IrmWccpW 2.00 0.00 10 0 100 500 0 0 0 68:03

RkiosAclMan Review 3.00 0.03 30 3 100 500 0 0 0 696:44

MatMan Review 0.50 0.00 4 1 100 500 0 0 0 0:00

EthHoleLinecardMan(1 1.11 0.01 10 29 100 500 0 0 0 390:16

EthHoleLinecardMan(3 1.11 0.44 10 6 100 500 0 0 0 7275:19

EthHoleLinecardMan(4 1.11 0.96 10 5 100 500 0 0 0 7282:02

EthHoleLinecardMan(5 1.11 0.56 10 6 100 500 0 0 0 7301:29

EthHoleLinecardMan(6 1.11 0.72 10 6 100 500 0 0 0 7280:02

EthHoleLinecardMan(7 1.11 0.44 10 6 100 500 0 0 0 7292:52

EthHoleLinecardMan(8 1.11 0.71 10 6 100 500 0 0 0 7291:33

EthHoleLinecardMan(9 1.11 0.63 10 6 100 500 0 0 0 5236:04

EthHoleLinecardMan(1 1.11 0.19 10 12 100 500 0 0 0 5196:33

-------------

%CPU Totals 224.60 116.19

Allocation ceiling Current allocation

------------------ ------------------

kbytes % in use kbytes % in use

Linecard 1's Store 258.00 12% 33.38 100%

Linecard 2's Store 258.00 0% 0.00 0%

Linecard 3's Store 258.00 67% 173.00 100%

Linecard 4's Store 258.00 67% 172.99 100%

Linecard 5's Store 258.00 67% 173.00 100%

Linecard 6's Store 258.00 67% 172.99 100%

Linecard 7's Store 258.00 67% 173.00 100%

Linecard 8's Store 258.00 67% 172.99 100%

Linecard 9's Store 258.00 67% 173.00 100%

Linecard 10's Store 258.00 67% 172.99 100%

TSM objects ------------------ ------------------

PacketInfoItem 859.37 0% 11.00 0%

VbufNodes2400 80.50 0% 0.00 0%

VbufNodes1600 55.50 0% 3.46 0%

VbufNodes400 288.00 1% 5.62 90%

VbufNodes64 60.00 0% 11.71 0%

VbufNodes4200 68.37 0% 0.00 0%

PacketBufRaw 25219.50 100% 25219.50 100%

PacketBufRawJumbo 184.29 100% 184.29 100%

RkiosSysPacketBuf 250.00 0% 1.15 2%

Packet 2079.42 2% 54.31 99%

MatEntrys 7680.00 0% 57.65 98%

MatEntryTableIterato 1.00 0% 0.03 0%

RkiosAclMan NamedGal 43.75 0% 0.12 62%

EpmAceListNode 192.00 0% 0.00 0%

Rkios Acl VlanMaps 128.00 0% 0.06 100%

Rkios Acl VlanMapEnt 859.37 0% 0.17 100%

Rkios EVC EFPs 1280.00 0% 0.00 0%

Rkios GBM BridgeDoma 256.00 0% 0.00 0%

Rkios GBM PseudoPort 768.00 0% 0.00 0%

RkiosQosPortVlanToEf 767.98 0% 0.00 0%

Rkios QoS PolicyMaps 356.32 1% 4.14 100%

AclClassifierIdToCla 6.00 0% 0.03 100%

Rkios QoS ClassMaps 896.00 0% 0.08 100%

AclToIosFilterMapLis 384.00 0% 0.03 100%

Rkios QoS Policers 3500.00 0% 29.20 100%

RkiosTableMap Galios 3.00 0% 0.00 0%

PimPhyports 1023.43 40% 410.39 100%

PimPorts 906.25 46% 421.40 100%

PimModules 160.00 3% 5.62 100%

PimSlots 6.00 3% 0.23 100%

PimChassis 38.12 6% 2.38 100%

PimQuack 0.35 0% 0.00 0%

EbmVlans 5024.00 0% 33.11 100%

EbmVlanGroupEntrys 7679.88 0% 7.85 50%

EbmPorts 328.00 45% 148.30 100%

EbmPortHostEntrys 448.00 1% 6.58 98%

EbmIeNodes 540.00 0% 5.27 100%

EbmPortVlanAclFeatur 2432.00 1% 38.14 98%

Ebm PV Maps 192.00 0% 0.00 0%

EbmSortedHostTableIt 1.75 0% 0.10 0%

EbmSortedGroupTableI 1.75 0% 0.05 0%

EbmHostRedundancyMan 1082.81 0% 0.00 0%

IrmVrfs 315.00 0% 2.46 100%

IrmFibLoadBalances 1152.00 0% 0.00 0%

IrmFibAdjs 1920.00 0% 10.00 98%

IrmPortEtherAddrEntr 500.00 0% 0.00 0%

IrmFibEntries 12288.00 0% 23.67 55%

IrmMfibEntryMemMan 6656.00 0% 14.52 25%

IrmWccpMemMan 34.03 0% 0.00 0%

AclOp 2048.00 0% 0.17 63%

AclOpAceSet 2559.96 0% 0.46 33%

AclClassifier 1280.00 1% 20.23 98%

AclFeature 2570.04 6% 177.97 99%

Acl 1408.00 1% 22.21 98%

Ace24 9215.85 0% 73.96 98%

Ace48 15359.76 0% 0.00 0%

AceActionDescStorage 1023.98 0% 0.00 0%

AclListNode 256.00 1% 4.05 98%

AceListNode 25600.00 0% 367.72 47%

AclClassifierActionL 1152.00 5% 68.02 99%

AclLayerFeatureListN 512.00 5% 45.42 64%

AclClassifierListNod 256.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 116.00 0% 0.00 0%

FlatAcl 512.00 4% 27.78 80%

FlatAce24 10752.00 6% 1294.20 52%

FlatAce48 16896.00 0% 0.00 0%

FlatAceActionListNod 281600.00 0% 2022.36 47%

FlatAclOpSetStorage 3072.00 0% 19.07 0%

FlatAclCacheNode 4224.00 4% 227.90 80%

FlatAclListNode 256.00 0% 7.73 0%

QosFeatureClassifier 515.13 1% 5.56 100%

QosFeatureClassifier 1030.26 1% 11.12 100%

QosClassifierActionL 14423.71 0% 38.93 100%

QosPortVlanAclFeatur 18544.78 0% 0.00 0%

QosEvcEfpAllocator 4352.00 0% 0.00 0%

QoS Policers 37000.00 0% 308.71 100%

MOL EvcEfp 11776.00 0% 0.00 0%

EvcEfpEncapVlanRange 1024.00 0% 0.00 0%

MOL GBM BridgeDomain 2304.00 0% 0.00 0%

MOL GBM PseudoPort 5888.00 0% 0.00 0%

GbmSortedHostTableIt 1.62 0% 0.00 0%

GbmBridgeDomainHostE 16384.00 0% 0.00 0%

KxAclPath 1024.00 2% 30.40 99%

KxAclPathListNode 384.00 0% 4.77 0%

KxAclConstPathListNo 384.00 0% 2.00 0%

IndexCache 800.78 0% 0.00 0%

ConfigToHwAfMap 202.89 0% 0.00 0%

HwToCfgAceMap 767.98 0% 0.00 0%

CommandTables 48.00 22% 10.59 100%

K2FibWccpServiceGrou 162.53 0% 0.00 0%

K2FibPbrFlatRouteMap 498.42 0% 0.00 0%

K2FibPbrExpandedFlat 2304.00 0% 0.00 0%

K2FibPbrFlatRouteMap 320.00 0% 0.00 0%

K2FibVrfs 152.00 0% 1.18 100%

K2 Netflow FPT Entri 11.97 0% 0.00 0%

K2TxPacket 384.00 0% 0.11 0%

K2TxPacketInfo 256.00 0% 4.01 0%

KxAclLabeledFlatAcl 3584.00 2% 169.31 62%

KxAclLabeledFlatAclE 3072.00 1% 72.56 62%

EbmVlanHostEntrys 3437.50 0% 16.18 94%

GalGbicEntrys 18.60 96% 18.04 99%

Event Nodes 128.00 1% 3.87 56%

Event Nodes 128.00 3% 50.96 8%

RkisoIpPbrRouteMaps 97.65 0% 0.00 0%

Rkios QoS EVC EFPs 768.00 0% 0.00 0%

RkiosNetflowFlows 3072.00 0% 0.00 0%

IrmFlows 384.00 0% 0.00 0%

K2AclMappingTableEnt 106.82 0% 5.63 0%

K2AclMappingTableEnt 71.21 0% 0.00 0%

K2AclTagPairSignatur 1536.00 2% 72.56 62%

K2 Microflow Policer 383.81 0% 0.00 0%

------------------ ------------------

TSM totals 607424.16 4% 31918.72 93%

xxxx-xxx-xx#

xxxx-xxx-xx#sh platform cpu packet statistics

Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3

------------ --------------- --------------- --------------- ---------------

0 0 52675 0 27187274

RkiosSysPacketMan:

Packet allocation failures: 0

Packet Buffer(Software Common) allocation failures: 0

Packet Buffer(Software ESMP) allocation failures: 0

Packet Buffer(Software EOBC) allocation failures: 0

Packet Buffer(Software SupToSup) allocation failures: 0

IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total 5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

113234211 2 0 1 0

Packets Dropped In Processing by CPU event

Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Input Acl 107839073 2 0 1 0

Output Acl 46 0 0 0 0

SA Miss 2063975 0 0 0 0

L2 Forward 37 0 0 0 0

Packets Dropped In Processing by Priority

Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Normal 38271103 1 0 0 0

Medium 5405345 0 0 0 0

High 72786628 0 0 0 0

Crucial 102215 0 0 0 0

Packets Dropped In Processing by Reason

Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt 26 0 0 0 0

L2DstDrop 498 0 0 0 0

L2DstDropInAcl 113 0 0 0 0

AclActionDrop 63141355 1 0 0 0

NoDstPorts 944 0 0 0 0

NoFloodPorts 50091236 0 0 0 0

RouteLookupFailed 39 0 0 0 0

Total packet queues 16

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Esmp 31341641630 388 424 338 324

L2/L3Control 1911638290 10 8 7 7

Host Learning 2433531 0 0 0 0

L3 Fwd High 76459 0 0 0 0

L3 Fwd Medium 13194150 0 0 0 0

L3 Fwd Low 35932825 0 0 0 0

L2 Fwd High 7349 0 0 0 0

L2 Fwd Medium 2452 0 0 0 0

L2 Fwd Low 62706217 0 0 0 0

L3 Rx High 157444 0 0 0 0

L3 Rx Low 74128765 1 0 0 0

RPF Failure 78278 0 0 0 0

ACL sw processing 643672985 0 0 0 0

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

L2/L3Control 2666463 0 0 0 0

Host Learning 13458739 0 0 0 0

L3 Fwd Low 61375 0 0 0 0

L3 Rx Low 41 0 0 0 0

ACL sw processing 5509018 0 0 0 0

xxxx-xxx-xx#

/claus

Ivan Shirshin · ‎10-08-2012

Hi Claus,

We see in the logs that the CPU is high due to management processes, specifically in "KxAclPathMan update" and "K2L2 Address Table R" jobs:

xxxx-xxx-xx#sh proc cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

49 24659898802830920636 0 86.10% 85.93% 85.85% 0 Cat4k Mgmt LoPri

48 1459019441878837282 77 11.74% 11.69% 11.71% 0 Cat4k Mgmt HiPri

xxxx-xxx-xx#sh platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

KxAclPathMan update 2.00 47.97 10 74 100 500 30 32 28 7308:38

K2AclMan-labeledFlat 1.00 3.15 10 37 100 500 1 6 5 400:26

K2AclCamMan stale en 1.00 9.41 10 93 100 500 21 18 16 2340:01

K2L2 Address Table R 2.00 17.48 12 7 100 500 14 14 10 4245:01

"K2L2 Address Table R" is related to L2 entries and adjacency updates. The learning of MAC addresses on this box is done in software, so if we keep on learning too many mac addresses you will see a spike of CPU on this process. This may happen in the following conditions:

- constant receipt of spanning tree topology change BPDUs: on PVST that reduces cam aging and you will see entries being deleted too fast, on rapid-PVST/MST this flushes the entries. Overall a number of entries may need to be deleted/added again.

- traffic hitting a static route which points to a broadcast multiaccess interface (such as a vlan interface) so that the box needs to ARP for end user addresses constantly

- mac-address moves: where a user will move from one port to another , or a mac address may be looping in the network.

The "KxAclPathMan update" is related to path update that is expected to run whenever configuration changes, e.g. ACLs or QoS service-policy. It is normal for them to use high CPU for some time.

However, the utilization due to these processes should drop and stay low under steady conditions.

ACTION PLAN:

Such symptoms could also be an indication of a bug in IOS and the issue requires further investigation, for which we need additional information.

1. Could you send collect and attach to the thread the following outputs, please? Note some of them are collected 2nd time but it is required to compare statistics.

show tech // in separate file

show platform health

show platform hardware acl statistics utilization brief

show platform software acl input path

show platform cpu packet statistics

show int

show log

2. Also, it would be good to see what kind of traffic is currently hitting the CPU using internal inband capture tool:

- debug platform packet all buffer

- show platform cpu packet buffered

It is low impact tool and does not affect CPU utilization much.

3. Finally, check if you can determine any change on the switch as a potential trigger for the issue and if there is some config change activity reoccurring on this device.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Claus Andersen · ‎10-08-2012

Hi

There has been no changes to the switch. We see the same problem on other 4500 switches with same IOS and hardware config also.

Ivan Shirshin · ‎10-08-2012

Hi Claus,

After checking the outputs I see 2 things increasing CPU consumption:

1. K2Fib Consistency Ch 1.00 18.13 5 6 100 500 0 1 2 36560:45

This is taking more CPU than expected due to the because of a background diagnostic job that runs every two minutes. This background job verifies the health of the L3 forwarding hardware, verifying that things are functioning as intended.

This job runs at a low priority, and will not interfere with other more important software tasks, such as forwarding packets or computing spanning trees. It shows high CPU in this case because the switch CPU is almost totally idle, so no other tasks are consuming that CPU.

Such spikes were fixed by a code enhancement CSCsy05491 (fix CPU spikes for K2 fib consistency checker), code fix was integrated into 12.2(53)SG, 12.2(52)SG and later releases. It smoothes the CPU usage by this job.

2. K2AclCamMan stale en 1.00 11.54 10 93 100 500 20 21 16 2351:58

The stale entry review is responsible for removing no-longer-needed entries from the hardware when classification features change. The consolidation review moves entries to the top of the TCAM to help with updating any partially-programmed policies to use the newly-freed space.

Can you check if STP is stable and if any MACs are flapping between vlans? The 2nd (mac move) could be verified using the "mac address-table notification mac-move" command. It generates alarms every time mac move is detected:

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/46sg/configuration/guide/swadmin.html#wp1252082

3. I see a lot of TCP packets destined to the local physical interface fa3/1 in the CPU buffer, coming from 192.168.67.136 and 192.168.67.135:

Index 24:

1097 days 10:55:49:874231 - RxVlan: 1006, RxPort: Fa3/1

Priority: Normal, Tag: No Tag, Event: L3 Forward, Flags: 0x440, Size: 64

Eth: Src 00:13:C4:06:56:5D Dst 00:1C:58:2E:DB:BF Type/Len 0x0800

Ip: ver:IpVersion4 len:20 tos:0 totLen:40 id:17107 fragOffset:0 ttl:59 proto:tcp

src: 192.168.67.136 dst: 192.168.89.100 firstFragment lastFragment

Remaining data:

0: 0xB2 0x69 0x0 0x16 0x12 0x4B 0xE5 0x7 0x4A 0x2B

10: 0x8D 0x34 0x50 0x10 0x1D 0x50 0xF3 0x14 0x0 0x0

20: 0x0 0x0 0x0 0x0 0x0 0x0 0xB1 0x17 0xEF 0x22

Index 0:

1097 days 10:55:48:772480 - RxVlan: 1006, RxPort: Fa3/1

Priority: Normal, Tag: No Tag, Event: L3 Forward, Flags: 0x440, Size: 64

Eth: Src 00:13:C4:06:56:5D Dst 00:1C:58:2E:DB:BF Type/Len 0x0800

Ip: ver:IpVersion4 len:20 tos:0 totLen:40 id:2015 fragOffset:0 ttl:123 proto:tcp

src: 192.168.67.135 dst: 192.168.89.100 firstFragment lastFragment

Remaining data:

0: 0x0 0x31 0x4C 0x51 0x6 0x36 0x7 0xCD 0xAE 0xB8

10: 0xA6 0x80 0x50 0x10 0xFA 0x63 0xE7 0x75 0x0 0x0

20: 0x0 0x0 0x0 0x0 0x0 0x0 0x6D 0x48 0x17 0x16

interface FastEthernet3/1

no switchport

ip address 192.168.89.100 255.255.255.240

duplex full

!

Check that IP sources and reason for such traffic to be sent to the switch interface, stop or block it and check the CPU utilization after that.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Claus Andersen · ‎10-08-2012

1) I will find out if it is posible to upgrade the IOS to a newer release

2) STP is stable and we dont see mac move

3) 192.168.67.135 is an internal ACS (tacacs+ server) and 192.168.67.136 is a internal jumphost, so it is not posible to block trafic from thos two hosts :-)

/claus

Claus Andersen · ‎10-12-2012

What software version are you recommending 15.0.2sg5?

/claus

Ivan Shirshin · ‎10-12-2012

Hi Claus,

15.0(2)SG2 is considered as a stable release and SG5 has mostly minor changes in the code to fix identified bugs. If you do not have any preference for specific features, that is a good release. I also checked DRAM and flash requirements and your HW is compatible.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Akhtar Samo · ‎10-10-2012

Hello Nick and Ivan,

Good to see this session again.

Can you suggest all possible tools to troubleshoot high cpu due to interrupts ? and their impacts. I had tried CPU SPAN, but it seemed CPU intensive. Do we have NETDR in 4500 ?

Regards,

Akhtar

Ivan Shirshin · ‎10-10-2012

Hi Akhtar!

When seeing high CPU, use the following commands to check what jobs are consuming CPU and if there are specific packet types being dropped:

- show platform health

- show platform cpu stat

E.g.:

switch#show platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

K5CpuMan Review 30.00 70.81 30 17 100 500 91 66 9 19:17

switch#show platform cpu packet statistics

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Ip Option 15167180 112673 84910 27502 0

As for capturing the CPU traffic, use the internal inband capture tool by configuring the following commands. It has low effect on the CPU utilization and can be used during high CPU situation:

switch#debug platform packet all buffer

platform packet debugging is on

switch#show platform cpu packet buffered

Total Received Packets Buffered: 1024

-------------------------------------

Index 0:

3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1

Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64

Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800

Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcp

src: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragment

Remaining data:

0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0

10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37 0x0 0x0

20: 0x0 0x1 0xB5 0x77 0x6A 0x7E

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Akhtar Samo · ‎10-11-2012

Thanks for your response. If you could kindly respond to my IOS query as well.

Regards,

Akhtar

Ivan Shirshin · ‎10-12-2012

Hi Akhtar,

You are welcome! Replied to your question in the post below. Sorry for delay with reply, somehow your notification for your post gone missing.

Kind Regards,

Ivan Shirshin

Kind Regards,
Ivan

Akhtar Samo · ‎10-10-2012

Hi,

We have been currently using IOS 12.2(54)SG1 on the SUP6-E, can you recommend the next stable release from CPU utilization perspective.

Regards,

Akhtar

Fahad Wasi · ‎10-12-2012

Hi Ivan and Nick,

I have some questions, do switches have the same type of hardware structure as routers have?

I mean what is the main difference between hardware of routers and switches?

Thanks

Ivan Shirshin · ‎10-12-2012

Hi Fahad,

Different Cisco platforms have different hardware structure. Even between the routers, e.g. 7200 router is CPU based platform, while 7600 is based on hardware forwarding and has separate switch and router processors.

Nexus switches are even further with different HW architecture and SW based on Linux.

The extent of the difference depends on what you are comparing, e.g. 6500 and 7600 have the same foundation. Are you comparing some specific platforms?

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Ask the Expert:Troubleshooting High CPU and Other Issues in the Cisco Catalyst 4500 Series Switches