cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

16437
Views
14
Helpful
33
Replies
Highlighted
Community Manager

Ask the Expert:Troubleshooting High CPU and Other Issues in the Cisco Catalyst 4500 Series Switches

With Nickolay Karpyshev and Ivan Shirshin

Read the bioRead the bioWelcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about architecture and troubleshooting of Industry's Most Widely Deployed Modular Access Platform Cisco Catalyst 4500 with Cisco Experts Nikolay Karpyshev and Ivan Shirshin.

Nikolay and Ivan are Customer Support Engineers in the high touch technology support team (HTTS) at Cisco specialized in LAN Switching and Routing. They support the Cisco Switches Nexus 7000, Catalyst 6500, 3750, 3560, 4500, 2900 and variety of routing platforms, and work as senior and escalation engineers. Both Nikolay and Ivan were previously a part of Cisco Sales Associate program. They hold Cisco Certifications: CCNP, CCSP, and CCDP.

Remember to use the rating system to let Nikolay and Ivan know if you have received an adequate response. 

Nikolay and Ivan might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Network Infrastructure  sub-community discussion forum shortly after the event. This event lasts through through October 19, 2012. Visit this forum often to view responses to your questions and the questions of other community members.

33 REPLIES 33
Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Experts

I have some problems with the 4500 platform and high cpu load.

normaly i will use

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tcam to troubleshoot the issue, but i always seems to hit a dead end.

Here is some output:

xxxx-xxx-xx#sh proc cpu  sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  49  24659898802830920636          0 86.10% 85.93% 85.85%   0 Cat4k Mgmt LoPri

  48   1459019441878837282         77 11.74% 11.69% 11.71%   0 Cat4k Mgmt HiPri

  36   406413592  95089543       4274  0.55%  0.50%  0.49%   0 IDB Work        

  39    73667932 102528426        718  0.23%  0.12%  0.11%   0 Per-Second Jobs 

169    164910162928907666          1  0.15%  0.14%  0.15%   0 PM Callback     

   3          56        65        861  0.07%  0.08%  0.01%   1 SSH Process     

218     1068680  83542081         12  0.07%  0.00%  0.00%   0 EIGRP-IPv4: Hell

  84     1900108 101130341         18  0.07%  0.02%  0.02%   0 UDLD            

201    25186108 202200624        124  0.07%  0.05%  0.07%   0 IGMP Input      

102    65327852 428594577        152  0.07%  0.03%  0.00%   0 IP Input        

159      207344    696207        297  0.07%  0.05%  0.07%   0 DHCP Security He

  15    58316156 153764063        379  0.07%  0.06%  0.07%   0 ARP Input       

152      172752   1450236        119  0.07%  0.00%  0.00%   0 TPLUS           

  13           0         1          0  0.00%  0.00%  0.00%   0 ifIndex Receive 

  14       26960   2180488         12  0.00%  0.00%  0.00%   0 EEM ED Syslog   

  12           0         1          0  0.00%  0.00%  0.00%   0 Crash writer    

  17           0         5          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  16           0         1          0  0.00%  0.00%  0.00%   0 CEF MIB API     

  11           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun

  20          96       241        398  0.00%  0.00%  0.00%   0 Entity MIB API  

  21        1304    186940          6  0.00%  0.00%  0.00%   0 IFS Agent Manage

  22         500   1580218          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach

  23           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager

  24      114212  94744201          1  0.00%  0.00%  0.00%   0 IPC Periodic Tim

  10         780        41      19024  0.00%  0.00%  0.00%   0 PrstVbl         

   9           0         2          0  0.00%  0.00%  0.00%   0 Timers          

  18           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit

  28           0         1          0  0.00%  0.00%  0.00%   0 IPC Session Serv

  19           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager  

  30           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect 

  25           0         1          0  0.00%  0.00%  0.00%   0 IPC Managed Time

  32      142432  94744243          1  0.00%  0.00%  0.00%   0 Dynamic ARP Insp

  33      119568  94744130          1  0.00%  0.00%  0.00%   0 GraphIt         

  34           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd  

  35      716052  83086833          8  0.00%  0.01%  0.00%   0 Net Background  

xxxx-xxx-xx#

xxxx-xxx-xx#sh platform health

                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

GalChassisVp-review    3.00   0.15     10     48  100  500    0   0    0  2114:40

S2w-JobEventSchedule  10.00   5.33     10     29  100  500    4   4    3  55466:23

Stub-JobEventSchedul  10.00   5.01     10     11  100  500    5   4    3  69132:27

Lj-poll                1.00   0.02      2      0  100  500    0   0    0  340:37

StatValueMan Update    1.00   0.03      1      0  100  500    0   0    0  1050:39

Pim-review             0.10   0.02      1      0  100  500    0   0    0  99:32

Ebm-host-review        1.00   0.14      8      3  100  500    0   0    0  249:44

Ebm-host-util-review   1.00   0.00     10      0  100  500    0   0    0  2:46

Ebm-port-review        0.10   0.00      1      0  100  500    0   0    0  8:26

Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:32

EbmHostRedundancyMan   2.00   0.00     20      0  100  500    0   0    0  0:00

Acl-Flattener          1.00   0.21     10      7  100  500    0   0    0  19:49

IrmFibThrottler Thro   2.00   0.00      7      5  100  500    0   0    0  132:24

GalChassisVp Ondeman   2.00   0.00      2      0  100  500    0   0    0  0:00

KxAclPathMan create/   1.00   2.10     10      6  100  500    2   2    2  211:43

KxAclPathMan update    2.00  47.97     10     74  100  500   30  32   28  7308:38

KxAclPathMan reprogr   1.00   0.00      2      1  100  500    0   0    0  7:14

TagMan-RecreateMtegR   1.00   1.48     10     94  100  500    1   2    2  284:13

K2CpuMan Review       30.00   4.84     30     44  100  500    5   5    4  106894:03

K2AccelPacketMan: Tx  10.00   2.07     20      4  100  500    2   1    1  52344:25

K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00

K2AclMan-labeledFlat   1.00   3.15     10     37  100  500    1   6    5  400:26

K2AclCamMan stale en   1.00   9.41     10     93  100  500   21  18   16  2340:01

K2AclCamMan consolid   1.00   1.11     10      9  100  500   30  24   13  1914:38

K2AclCamMan hw stats   3.00   1.48     10      8  100  500    0   1    0  10858:55

K2AclCamMan kx stats   1.00   0.17     10      8  100  500    1   1    0  9470:11

K2AclCamMan Audit re   1.00   0.00     10      0  100  500    0   0    0  1:07

K2AclPolicerTableMan   1.00   0.24     10      5  100  500    0   0    0  7993:02

K2L2 Address Table R   2.00  17.48     12      7  100  500   14  14   10  4245:01

K2L2 New Static Addr   2.00   2.03     10      5  100  500    2   2    1  187:55

K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  80:08

K2L2 Dynamic Address   2.00   0.02     10      2  100  500    0   0    0  4:39

K2L2 Vlan Table Revi   2.00   0.64     12      8  100  500    0   0    0  86:57

K2PortMan Review       3.00   2.29     15     13  100  500    3   3    2  36205:34

Gigaport0 Review       0.40   0.00      4      2  100  500    0   0    0  139:58

Gigaport1 Review       0.40   0.01      4      2  100  500    0   0    0  137:38

Gigaport2 Review       0.40   0.01      4      2  100  500    0   0    0  137:32

Gigaport3 Review       0.40   0.00      4      2  100  500    0   0    0  137:28

Gigaport4 Review       0.40   0.01      4      2  100  500    0   0    0  137:38

Gigaport5 Review       0.40   0.01      4      2  100  500    0   0    0  137:43

Gigaport6 Review       0.40   0.00      4      2  100  500    0   0    0  137:42

Gigaport7 Review       0.40   0.01      4      2  100  500    0   0    0  137:25

Gigaport8 Review       0.40   0.00      4      2  100  500    0   0    0  137:30

Gigaport9 Review       0.40   0.00      4      2  100  500    0   0    0  137:33

Gigaport10 Review      0.40   0.00      4      2  100  500    0   0    0  142:18

Gigaport11 Review      0.40   0.00      4      2  100  500    0   0    0  137:27

Gigaport12 Review      0.40   0.00      4      2  100  500    0   0    0  145:41

Gigaport13 Review      0.40   0.00      4      2  100  500    0   0    0  137:40

Gigaport14 Review      0.40   0.01      4      2  100  500    0   0    0  137:20

Gigaport15 Review      0.40   0.01      4      2  100  500    0   0    0  137:27

Gigaport16 Review      0.40   0.01      4      2  100  500    0   0    0  141:32

Gigaport17 Review      0.40   0.00      4      2  100  500    0   0    0  142:20

Gigaport18 Review      0.40   0.01      4      2  100  500    0   0    0  137:33

Gigaport19 Review      0.40   0.00      4      2  100  500    0   0    0  140:26

Gigaport20 Review      0.40   0.00      4      2  100  500    0   0    0  142:12

Gigaport21 Review      0.40   0.00      4      2  100  500    0   0    0  139:14

Gigaport22 Review      0.40   0.00      4      2  100  500    0   0    0  139:24

Gigaport23 Review      0.40   0.00      4      2  100  500    0   0    0  142:13

Gigaport24 Review      0.40   0.00      4      2  100  500    0   0    0  137:33

Gigaport25 Review      0.40   0.01      4      2  100  500    0   0    0  137:59

Gigaport26 Review      0.40   0.01      4      2  100  500    0   0    0  138:06

Gigaport27 Review      0.40   0.00      4      2  100  500    0   0    0  142:42

Gigaport28 Review      0.40   0.01      4      2  100  500    0   0    0  137:42

Gigaport29 Review      0.40   0.01      4      2  100  500    0   0    0  138:15

Gigaport30 Review      0.40   0.00      4      2  100  500    0   0    0  144:18

Gigaport31 Review      0.40   0.01      4      2  100  500    0   0    0  137:49

Gigaport32 Review      0.40   0.00      4      2  100  500    0   0    0  137:39

Gigaport33 Review      0.40   0.00      4      2  100  500    0   0    0  146:07

Gigaport34 Review      0.40   0.00      4      2  100  500    0   0    0  137:26

Gigaport35 Review      0.40   0.01      4      2  100  500    0   0    0  137:31

Gigaport36 Review      0.40   0.00      4      2  100  500    0   0    0  141:23

Gigaport37 Review      0.40   0.00      4      2  100  500    0   0    0  137:33

Gigaport38 Review      0.40   0.01      4      2  100  500    0   0    0  138:07

Gigaport39 Review      0.40   0.01      4      2  100  500    0   0    0  144:49

Gigaport40 Review      0.40   0.00      4      2  100  500    0   0    0  143:27

Gigaport41 Review      0.40   0.00      4      2  100  500    0   0    0  142:49

Gigaport42 Review      0.40   0.00      4      2  100  500    0   0    0  141:56

Gigaport43 Review      0.40   0.00      4      2  100  500    0   0    0  143:45

Gigaport44 Review      0.40   0.00      4      2  100  500    0   0    0  140:52

Gigaport45 Review      0.40   0.00      4      2  100  500    0   0    0  137:33

Gigaport46 Review      0.40   0.00      4      2  100  500    0   0    0  140:46

Gigaport47 Review      0.40   0.00      4      2  100  500    0   0    0  137:30

Gigaport48 Review      0.40   0.00      4      0  100  500    0   0    0  0:00

Gigaport49 Review      0.40   0.00      4      0  100  500    0   0    0  0:00

Gigaport50 Review      0.40   0.00      4      0  100  500    0   0    0  0:00

Gigaport51 Review      0.40   0.00      4      0  100  500    0   0    0  0:00

Gigaport52 Review      0.40   0.38      4      4  100  500    0   0    0  4552:51

Gigaport53 Review      0.40   0.30      4     17  100  500    0   0    0  3891:56

K2Fib cam usage revi   2.00   0.00     15      0  100  500    0   0    0  0:00

K2Fib IrmFib Review    2.00   0.00     15      4  100  500    0   0    0  0:00

K2Fib AdjRepop Revie   2.00   0.00     15      0  100  500    0   0    0  0:03

K2Fib Vrf Unpunt Rev   2.00   0.00     15      0  100  500    0   0    0  113:48

K2Fib Consistency Ch   1.00   0.00      5      6  100  500    2   3    2  36559:04

K2FibAdjMan Stats Re   2.00   0.05     10      6  100  500    0   0    0  2769:15

K2FibAdjMan Host Mov   2.00   0.44     10      4  100  500    0   0    0  95:43

K2FibAdjMan Adj Chan   2.00   0.00     10      0  100  500    0   0    0  0:00

K2FibMulticast Signa   2.00   0.09     10      1  100  500    0   0    0  375:37

K2FibMulticast Entry   2.00   0.00     10      7  100  500    0   0    0  593:15

K2FibMulticast Irm M   2.00   0.00     10      0  100  500    0   0    0  0:00

K2FibFastDropMan Rev   2.00   0.00      7      0  100  500    0   0    0  24:19

K2FibPbr route map r   2.00   0.07     20      7  100  500    0   0    0  3167:28

K2FibPbr flat acl pr   2.00   0.00     20      1  100  500    0   0    0  639:48

K2FibPbr consolidati   2.00   0.00     10      0  100  500    0   0    0  146:36

K2FibPerVlanPuntMan    2.00   0.00     15      2  100  500    0   0    0  0:00

K2FibWccpMan Service   2.00   0.00     20      0  100  500    0   0    0  0:00

K2FibWccpMan Cache E   2.00   0.00     20      0  100  500    0   0    0  0:00

K2FibWccpMan Redirec   2.00   0.00     20      0  100  500    0   0    0  0:00

K2FibWccpMan AdjRepo   2.00   0.00     20      0  100  500    0   0    0  0:00

K2FibFC Aging review   2.00   0.00     10      0  100  500    0   0    0  110:09

K2FibFC DelFlow revi   2.00   0.00     10      0  100  500    0   0    0  0:00

K2FibFC Adj review     2.00   0.00     10      0  100  500    0   0    0  109:52

K2FibFC overload/rel   2.00   0.00     10      0  100  500    0   0    0  33:33

K2MetStatsMan Review   2.00   0.20      5      2  100  500    0   0    0  753:11

K2FibMulticast MET S   2.00   0.00     10      0  100  500    0   0    0  0:00

K2QosDblMan Rate DBL   2.00   0.15      7      1  100  500    0   0    0  2307:44

K2 VlanStatsMan Revi   2.00   0.23     15      4  100  500    0   0    0  3060:11

K2 Packet Memory Dia   2.00   0.00     15     12  100  500    0   0    0  9342:50

K2 L2 Aging Table Re   2.00   0.04     20      2  100  500    0   0    0  1733:54

K2NetflowMan - hw fl   5.00   0.09     20      0  100  500    0   0    0  351:33

K2FibEntry activity    1.00   0.01     10      7  100  500    0   0    0  2094:31

K2PacketBufMonitor-P   3.00   0.52     10      4  100  500    0   0    0  7013:10

GbmHostTable Review    2.00   0.00      8      0  100  500    0   0    0  22:34

GbmHostTableUtilizat   2.00   0.00      8      0  100  500    0   0    0  0:00

GbmHostTableAging Re   2.00   0.00      8      0  100  500    0   0    0  33:35

RkiosPortMan Port Re   2.00   0.79     12     10  100  500    1   0    0  9792:46

Rkios Module State R   4.00   0.02     40     38  100  500    0   0    0  379:39

Rkios Online Diag Re   4.00   0.02     40      0  100  500    0   0    0  358:06

RkiosIpPbr IrmPort R   2.00   0.07     10      1  100  500    0   0    0  606:48

RkiosIpWccp IrmWccpW   2.00   0.00     10      0  100  500    0   0    0  68:03

RkiosAclMan Review     3.00   0.03     30      3  100  500    0   0    0  696:44

MatMan Review          0.50   0.00      4      1  100  500    0   0    0  0:00

EthHoleLinecardMan(1   1.11   0.01     10     29  100  500    0   0    0  390:16

EthHoleLinecardMan(3   1.11   0.44     10      6  100  500    0   0    0  7275:19

EthHoleLinecardMan(4   1.11   0.96     10      5  100  500    0   0    0  7282:02

EthHoleLinecardMan(5   1.11   0.56     10      6  100  500    0   0    0  7301:29

EthHoleLinecardMan(6   1.11   0.72     10      6  100  500    0   0    0  7280:02

EthHoleLinecardMan(7   1.11   0.44     10      6  100  500    0   0    0  7292:52

EthHoleLinecardMan(8   1.11   0.71     10      6  100  500    0   0    0  7291:33

EthHoleLinecardMan(9   1.11   0.63     10      6  100  500    0   0    0  5236:04

EthHoleLinecardMan(1   1.11   0.19     10     12  100  500    0   0    0  5196:33

                     -------------

%CPU Totals          224.60 116.19

                       Allocation ceiling        Current allocation

                       ------------------        ------------------

                       kbytes    % in use        kbytes    % in use

Linecard 1's Store        258.00      12%           33.38      100%

Linecard 2's Store        258.00       0%            0.00        0%

Linecard 3's Store        258.00      67%          173.00      100%

Linecard 4's Store        258.00      67%          172.99      100%

Linecard 5's Store        258.00      67%          173.00      100%

Linecard 6's Store        258.00      67%          172.99      100%

Linecard 7's Store        258.00      67%          173.00      100%

Linecard 8's Store        258.00      67%          172.99      100%

Linecard 9's Store        258.00      67%          173.00      100%

Linecard 10's Store       258.00      67%          172.99      100%

TSM objects            ------------------        ------------------

PacketInfoItem            859.37       0%           11.00        0%

VbufNodes2400              80.50       0%            0.00        0%

VbufNodes1600              55.50       0%            3.46        0%

VbufNodes400              288.00       1%            5.62       90%

VbufNodes64                60.00       0%           11.71        0%

VbufNodes4200              68.37       0%            0.00        0%

PacketBufRaw            25219.50     100%        25219.50      100%

PacketBufRawJumbo         184.29     100%          184.29      100%

RkiosSysPacketBuf         250.00       0%            1.15        2%

Packet                   2079.42       2%           54.31       99%

MatEntrys                7680.00       0%           57.65       98%

MatEntryTableIterato        1.00       0%            0.03        0%

RkiosAclMan NamedGal       43.75       0%            0.12       62%

EpmAceListNode            192.00       0%            0.00        0%

Rkios Acl VlanMaps        128.00       0%            0.06      100%

Rkios Acl VlanMapEnt      859.37       0%            0.17      100%

Rkios EVC EFPs           1280.00       0%            0.00        0%

Rkios GBM BridgeDoma      256.00       0%            0.00        0%

Rkios GBM PseudoPort      768.00       0%            0.00        0%

RkiosQosPortVlanToEf      767.98       0%            0.00        0%

Rkios QoS PolicyMaps      356.32       1%            4.14      100%

AclClassifierIdToCla        6.00       0%            0.03      100%

Rkios QoS ClassMaps       896.00       0%            0.08      100%

AclToIosFilterMapLis      384.00       0%            0.03      100%

Rkios QoS Policers       3500.00       0%           29.20      100%

RkiosTableMap Galios        3.00       0%            0.00        0%

PimPhyports              1023.43      40%          410.39      100%

PimPorts                  906.25      46%          421.40      100%

PimModules                160.00       3%            5.62      100%

PimSlots                    6.00       3%            0.23      100%

PimChassis                 38.12       6%            2.38      100%

PimQuack                    0.35       0%            0.00        0%

EbmVlans                 5024.00       0%           33.11      100%

EbmVlanGroupEntrys       7679.88       0%            7.85       50%

EbmPorts                  328.00      45%          148.30      100%

EbmPortHostEntrys         448.00       1%            6.58       98%

EbmIeNodes                540.00       0%            5.27      100%

EbmPortVlanAclFeatur     2432.00       1%           38.14       98%

Ebm PV Maps               192.00       0%            0.00        0%

EbmSortedHostTableIt        1.75       0%            0.10        0%

EbmSortedGroupTableI        1.75       0%            0.05        0%

EbmHostRedundancyMan     1082.81       0%            0.00        0%

IrmVrfs                   315.00       0%            2.46      100%

IrmFibLoadBalances       1152.00       0%            0.00        0%

IrmFibAdjs               1920.00       0%           10.00       98%

IrmPortEtherAddrEntr      500.00       0%            0.00        0%

IrmFibEntries           12288.00       0%           23.67       55%

IrmMfibEntryMemMan       6656.00       0%           14.52       25%

IrmWccpMemMan              34.03       0%            0.00        0%

AclOp                    2048.00       0%            0.17       63%

AclOpAceSet              2559.96       0%            0.46       33%

AclClassifier            1280.00       1%           20.23       98%

AclFeature               2570.04       6%          177.97       99%

Acl                      1408.00       1%           22.21       98%

Ace24                    9215.85       0%           73.96       98%

Ace48                   15359.76       0%            0.00        0%

AceActionDescStorage     1023.98       0%            0.00        0%

AclListNode               256.00       1%            4.05       98%

AceListNode             25600.00       0%          367.72       47%

AclClassifierActionL     1152.00       5%           68.02       99%

AclLayerFeatureListN      512.00       5%           45.42       64%

AclClassifierListNod      256.00       0%            0.00        0%

TableMapMan NameToTa       27.00       0%            0.00        0%

TableMapAllocator         116.00       0%            0.00        0%

FlatAcl                   512.00       4%           27.78       80%

FlatAce24               10752.00       6%         1294.20       52%

FlatAce48               16896.00       0%            0.00        0%

FlatAceActionListNod   281600.00       0%         2022.36       47%

FlatAclOpSetStorage      3072.00       0%           19.07        0%

FlatAclCacheNode         4224.00       4%          227.90       80%

FlatAclListNode           256.00       0%            7.73        0%

QosFeatureClassifier      515.13       1%            5.56      100%

QosFeatureClassifier     1030.26       1%           11.12      100%

QosClassifierActionL    14423.71       0%           38.93      100%

QosPortVlanAclFeatur    18544.78       0%            0.00        0%

QosEvcEfpAllocator       4352.00       0%            0.00        0%

QoS Policers            37000.00       0%          308.71      100%

MOL EvcEfp              11776.00       0%            0.00        0%

EvcEfpEncapVlanRange     1024.00       0%            0.00        0%

MOL GBM BridgeDomain     2304.00       0%            0.00        0%

MOL GBM PseudoPort       5888.00       0%            0.00        0%

GbmSortedHostTableIt        1.62       0%            0.00        0%

GbmBridgeDomainHostE    16384.00       0%            0.00        0%

KxAclPath                1024.00       2%           30.40       99%

KxAclPathListNode         384.00       0%            4.77        0%

KxAclConstPathListNo      384.00       0%            2.00        0%

IndexCache                800.78       0%            0.00        0%

ConfigToHwAfMap           202.89       0%            0.00        0%

HwToCfgAceMap             767.98       0%            0.00        0%

CommandTables              48.00      22%           10.59      100%

K2FibWccpServiceGrou      162.53       0%            0.00        0%

K2FibPbrFlatRouteMap      498.42       0%            0.00        0%

K2FibPbrExpandedFlat     2304.00       0%            0.00        0%

K2FibPbrFlatRouteMap      320.00       0%            0.00        0%

K2FibVrfs                 152.00       0%            1.18      100%

K2 Netflow FPT Entri       11.97       0%            0.00        0%

K2TxPacket                384.00       0%            0.11        0%

K2TxPacketInfo            256.00       0%            4.01        0%

KxAclLabeledFlatAcl      3584.00       2%          169.31       62%

KxAclLabeledFlatAclE     3072.00       1%           72.56       62%

EbmVlanHostEntrys        3437.50       0%           16.18       94%

GalGbicEntrys              18.60      96%           18.04       99%

Event Nodes               128.00       1%            3.87       56%

Event Nodes               128.00       3%           50.96        8%

RkisoIpPbrRouteMaps        97.65       0%            0.00        0%

Rkios QoS EVC EFPs        768.00       0%            0.00        0%

RkiosNetflowFlows        3072.00       0%            0.00        0%

IrmFlows                  384.00       0%            0.00        0%

K2AclMappingTableEnt      106.82       0%            5.63        0%

K2AclMappingTableEnt       71.21       0%            0.00        0%

K2AclTagPairSignatur     1536.00       2%           72.56       62%

K2 Microflow Policer      383.81       0%            0.00        0%

                       ------------------        ------------------

TSM totals             607424.16       4%        31918.72       93%

xxxx-xxx-xx# 

xxxx-xxx-xx#sh platform cpu packet statistics

Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport  TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3

------------ --------------- --------------- --------------- ---------------

           0               0           52675               0        27187274

RkiosSysPacketMan:

Packet allocation failures: 0

Packet Buffer(Software Common) allocation failures: 0

Packet Buffer(Software ESMP) allocation failures: 0

Packet Buffer(Software EOBC) allocation failures: 0

Packet Buffer(Software SupToSup) allocation failures: 0

IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total                5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

           113234211         2         0         1          0

Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Input Acl                    107839073         2         0         1          0

Output Acl                          46         0         0         0          0

SA Miss                        2063975         0         0         0          0

L2 Forward                          37         0         0         0          0

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Normal                        38271103         1         0         0          0

Medium                         5405345         0         0         0          0

High                          72786628         0         0         0          0

Crucial                         102215         0         0         0          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt                     26         0         0         0          0

L2DstDrop                           498         0         0         0          0

L2DstDropInAcl                      113         0         0         0          0

AclActionDrop                  63141355         1         0         0          0

NoDstPorts                          944         0         0         0          0

NoFloodPorts                   50091236         0         0         0          0

RouteLookupFailed                    39         0         0         0          0

Total packet queues 16

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Esmp                       31341641630       388       424       338        324

L2/L3Control                1911638290        10         8         7          7

Host Learning                  2433531         0         0         0          0

L3 Fwd High                      76459         0         0         0          0

L3 Fwd Medium                 13194150         0         0         0          0

L3 Fwd Low                    35932825         0         0         0          0

L2 Fwd High                       7349         0         0         0          0

L2 Fwd Medium                     2452         0         0         0          0

L2 Fwd Low                    62706217         0         0         0          0

L3 Rx High                      157444         0         0         0          0

L3 Rx Low                     74128765         1         0         0          0

RPF Failure                      78278         0         0         0          0

ACL sw processing            643672985         0         0         0          0

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

L2/L3Control                   2666463         0         0         0          0

Host Learning                 13458739         0         0         0          0

L3 Fwd Low                       61375         0         0         0          0

L3 Rx Low                           41         0         0         0          0

ACL sw processing              5509018         0         0         0          0

xxxx-xxx-xx#

/claus

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Claus,

We see in the logs that the CPU is high due to management processes, specifically in "KxAclPathMan update" and "K2L2 Address Table R" jobs:

xxxx-xxx-xx#sh proc cpu  sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  49  24659898802830920636          0 86.10% 85.93% 85.85%   0 Cat4k Mgmt LoPri

  48   1459019441878837282         77 11.74% 11.69% 11.71%   0 Cat4k Mgmt HiPri

xxxx-xxx-xx#sh platform health

                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

KxAclPathMan update    2.00  47.97     10     74  100  500   30  32   28  7308:38

K2AclMan-labeledFlat   1.00   3.15     10     37  100  500    1   6    5  400:26

K2AclCamMan stale en   1.00   9.41     10     93  100  500   21  18   16  2340:01

K2L2 Address Table R   2.00  17.48     12      7  100  500   14  14   10  4245:01

"K2L2 Address Table R" is related to L2 entries and adjacency updates. The learning of MAC addresses on this box is done in software, so if we keep on learning too many mac addresses you will see a spike of CPU on this process. This may happen in the following conditions:

     - constant receipt of spanning tree topology change BPDUs: on PVST that reduces cam aging and you will see entries being deleted too fast, on rapid-PVST/MST this flushes the entries. Overall a number of entries may need to be deleted/added again.

     - traffic hitting a static route which points to a broadcast multiaccess interface (such as a vlan interface) so that the box needs to ARP for end user addresses constantly

          - mac-address moves: where a user will move from one port to another , or a mac address may be looping in the network.

The "KxAclPathMan update" is related to path update that is expected to run whenever configuration changes, e.g. ACLs or QoS service-policy. It is normal for them to use high CPU for some time.

However, the utilization due to these processes should drop and stay low under steady conditions.

ACTION PLAN:

Such symptoms could also be an indication of a bug in IOS and the issue requires further investigation, for which we need additional information.

1. Could you send collect and attach to the thread the following outputs, please? Note some of them are collected 2nd time but it is required to compare statistics.

show tech          // in separate file

show platform health

show platform hardware acl statistics utilization brief

show platform software acl input path

show platform cpu packet statistics

show int

show log

2. Also, it would be good to see what kind of traffic is currently hitting the CPU using internal inband capture tool:

- debug platform packet all buffer

- show platform cpu packet buffered

It is low impact tool and does not affect CPU utilization much.

3. Finally, check if you can determine any change on the switch as a potential trigger for the issue and if there is some config change activity reoccurring on this device.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
Beginner

Re: Ask the Expert:Troubleshooting High CPU and Other Issues in

Hi

There has been no changes to the switch. We see the same problem on other 4500 switches with same IOS and hardware config also.

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Claus,

After checking the outputs I see 2 things increasing CPU consumption:

1. K2Fib Consistency Ch   1.00  18.13      5      6  100  500    0   1    2  36560:45

This is taking more CPU than expected due to the because of a background diagnostic job that runs every two minutes. This background job verifies the health of the L3 forwarding           hardware, verifying that things are functioning as intended.

This job runs at a low priority, and will not interfere with other more important software tasks, such as forwarding packets or computing spanning trees.  It shows high CPU in this case because the switch CPU is almost totally idle, so no other tasks are consuming that CPU.

Such spikes were fixed by a code enhancement CSCsy05491 (fix CPU spikes for K2 fib consistency checker), code fix was integrated into 12.2(53)SG, 12.2(52)SG and later releases. It smoothes the CPU usage by this job.

2. K2AclCamMan stale en   1.00  11.54     10     93  100  500   20  21   16  2351:58

The stale entry review is responsible for removing no-longer-needed entries from the hardware when classification features change.  The consolidation review moves entries to the top of the TCAM to help with updating any partially-programmed policies to use the newly-freed space.

Can you check if STP is stable and if any MACs are flapping between vlans? The 2nd (mac move) could be verified using the "mac address-table notification mac-move" command. It generates alarms every time mac move is detected:

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/46sg/configuration/guide/swadmin.html#wp1252082

3. I see a lot of TCP packets destined to the local physical interface fa3/1 in the CPU buffer, coming from 192.168.67.136 and 192.168.67.135:

Index 24:

1097 days 10:55:49:874231 - RxVlan: 1006, RxPort: Fa3/1

Priority: Normal, Tag: No Tag, Event: L3 Forward, Flags: 0x440, Size: 64

Eth: Src 00:13:C4:06:56:5D Dst 00:1C:58:2E:DB:BF Type/Len 0x0800

Ip: ver:IpVersion4 len:20 tos:0 totLen:40 id:17107 fragOffset:0 ttl:59 proto:tcp

    src: 192.168.67.136 dst: 192.168.89.100 firstFragment lastFragment

Remaining data:

0: 0xB2 0x69 0x0  0x16 0x12 0x4B 0xE5 0x7  0x4A 0x2B

10: 0x8D 0x34 0x50 0x10 0x1D 0x50 0xF3 0x14 0x0  0x0 

20: 0x0  0x0  0x0  0x0  0x0  0x0  0xB1 0x17 0xEF 0x22

Index 0:

1097 days 10:55:48:772480 - RxVlan: 1006, RxPort: Fa3/1

Priority: Normal, Tag: No Tag, Event: L3 Forward, Flags: 0x440, Size: 64

Eth: Src 00:13:C4:06:56:5D Dst 00:1C:58:2E:DB:BF Type/Len 0x0800

Ip: ver:IpVersion4 len:20 tos:0 totLen:40 id:2015 fragOffset:0 ttl:123 proto:tcp

    src: 192.168.67.135 dst: 192.168.89.100 firstFragment lastFragment

Remaining data:

0: 0x0  0x31 0x4C 0x51 0x6  0x36 0x7  0xCD 0xAE 0xB8

10: 0xA6 0x80 0x50 0x10 0xFA 0x63 0xE7 0x75 0x0  0x0 

20: 0x0  0x0  0x0  0x0  0x0  0x0  0x6D 0x48 0x17 0x16

interface FastEthernet3/1

no switchport

ip address 192.168.89.100 255.255.255.240

duplex full

!

Check that IP sources and reason for such traffic to be sent to the switch interface, stop or block it and check the CPU utilization after that.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

1) I will find out if it is posible to upgrade the IOS to a newer release

2) STP is stable and we dont see mac move

3) 192.168.67.135 is an internal ACS (tacacs+ server) and 192.168.67.136 is a internal jumphost, so it is not posible to block trafic from thos two hosts :-)

/claus

Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

What software version are you recommending 15.0.2sg5?

/claus

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Claus,

15.0(2)SG2 is  considered as a stable release and SG5 has mostly minor changes in the code to fix identified bugs. If you do not have any preference for specific features, that is a good release. I also checked DRAM and flash requirements and your HW is compatible.

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hello Nick and Ivan,

Good to see this session again.

Can you suggest all possible tools to troubleshoot high cpu due to interrupts ? and their impacts. I had tried CPU SPAN, but it seemed CPU intensive. Do we have NETDR in 4500 ?

Regards,

Akhtar

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Akhtar!

When seeing high CPU, use the following commands to check what jobs are consuming CPU and if there are specific packet types being dropped:

     - show platform health

     - show platform cpu stat

E.g.:

switch#show platform health

                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

K5CpuMan Review       30.00  70.81     30     17  100  500   91  66    9  19:17

switch#show platform cpu packet statistics

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Ip Option                     15167180    112673     84910     27502          0

As for capturing the CPU traffic, use the internal inband capture tool by configuring the following commands. It has low effect on the CPU utilization and can be used during high CPU situation:

switch#debug platform packet all buffer

platform packet debugging is on

switch#show platform cpu packet buffered

Total Received Packets Buffered: 1024

-------------------------------------

Index 0:

3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1

Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64

Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800

Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcp

    src: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragment

Remaining data:

0: 0x0  0x64 0x0  0x64 0x0  0x0  0x0  0x0  0x0  0x0

10: 0x0  0x0  0x50 0x0  0x0  0x0  0x8A 0x37 0x0  0x0

20: 0x0  0x1  0xB5 0x77 0x6A 0x7E

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Thanks for your response. If you could kindly respond to my IOS query as well.

Regards,

Akhtar

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Akhtar,

You are welcome! Replied to your question in the post below. Sorry for  delay with reply, somehow your notification for your post gone missing.

Kind Regards,

Ivan Shirshin

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi,

We have been currently using IOS 12.2(54)SG1 on the SUP6-E, can you recommend the next stable release from CPU utilization perspective.

Regards,

Akhtar

Beginner

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Ivan and Nick,

I have some questions, do switches have the same type of hardware structure as routers have?

I mean what is the main difference between hardware of routers and switches?

Thanks

Cisco Employee

Ask the Expert:Troubleshooting High CPU and Other Issues in the

Hi Fahad,

Different Cisco platforms have different hardware structure. Even between the routers, e.g. 7200 router is CPU based platform, while 7600 is based on hardware forwarding and has separate switch and router processors.

Nexus switches are even further with different HW architecture and SW based on Linux.

The extent of the difference depends on what you are comparing, e.g. 6500 and 7600 have the same foundation. Are you comparing some specific platforms?

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards, Ivan Shirshin **Please grade this post if you find it useful.
CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards