cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
747
Views
0
Helpful
5
Replies

Problem with CPU on Cat 4507

Dao Quynh
Level 1
Level 1

hi all,

Our Customer used Distribution switches are DistSW4507-01 and DistSW4507-02. These switches used HSRP with Dist-02 is active, Dist-01 is standby.

But, the past few days then the CPU of Dist-01 increase to 60%, you can see result as follow:

DistSW-4507-01#sh processes cpu hi    

     54515 5 5 5 5 5 545 5 5 5 515 5 515 5 5 5 5 5 5 515 5 515

    9414049494949494941494949494049494049494949494949404949404

100                                                          

90                                                          

80                                                          

70                                                          

60                                                          

50  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

40  *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

30  *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

20  *** * * * * * * *** * * * * * * * * * * * * * * * * * * *

10 ##########################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5   

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    5555555555555555555555555555555555555555554554444445555545444444444444

    5555454554444343443332233222221211211000009009999980000090999998877787

100                                                                      

90                                                                      

80                                                                      

70                                                                      

60 **** * **                                                            

50 **********************************************************************

40 **********************************************************************

30 **********************************************************************

20 **********************************************************************

10 ######################################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

DistSW-4507-01#sh processes cpu sorted

CPU utilization for five seconds: 54%/0%; one minute: 11%; five minutes: 9%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  41   263762272 258671478       1019 52.15%  8.30%  6.38%   0 Cat4k Mgmt LoPri

  40    68970600 388841616        177  2.07%  2.05%  2.05%   0 Cat4k Mgmt HiPri

  85     1575364  11474716        137  0.07%  0.02%  0.00%   0 Spanning Tree   

  69          40   7066609          0  0.07%  0.00%  0.00%   0 UDLD            

  82     2082608  61332016         33  0.07%  0.04%  0.03%   0 IP Input        

135     1170540  54087423         21  0.07%  0.05%  0.07%   0 HSRP (Standby)  

   7     5794312    955116       6066  0.00%  0.06%  0.05%   0 Check heaps     

   8           0         3          0  0.00%  0.00%  0.00%   0 Pool Manager  

Please help me resolve,

thank all,

Quynh

5 Replies 5

Dao Quynh
Level 1
Level 1

Normal,

CPU of the Dist-01 is very low as follow:

DistSW-4507-01#sh processes cpu sorted | exclude 0.00

CPU utilization for five seconds: 6%/0%; one minute: 7%; five minutes: 8%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  41   263764144 258675016       1019  3.43%  4.46%  5.60%   0 Cat4k Mgmt LoPri

  40    68971748 388848159        177  2.15%  2.07%  2.05%   0 Cat4k Mgmt HiPri

135     1170540  54088269         21  0.07%  0.05%  0.07%   0 HSRP (Standby)  

149        4660     64237         72  0.07%  0.11%  0.27%   1 Virtual Exec    

  77     1652764   3311459        499  0.07%  0.09%  0.08%   0 CDP Protocol

InayathUlla Sharieff
Cisco Employee
Cisco Employee

Hi Dao,

Could you please provide me with the following outputs:

show process cpu sorted | ex 0.00%

Show platform cpu packet statistics

Show platform health | ex 0.00

Also the below link has very good document on troubleshooting high cpu on cat 4500 kindly go through the same which help you in finding the root cause:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2

HTH

Regards

Inayath

*Plz rate all usefull posts.

Mgmt High & Low priority process'' processes control plane as well as data packets that need to be software-switched or processed.Try the following.

Hardcode the L3 interfaces with the commands no ip redirect, no ip unreachable and no ip proxy-arp. It should come to normal.

--Pls rate helpful posts--

hi InayathUlla Sharieff,

These are the result of the commands:

DistSW-4507-01#show platform cpu packet statistics

Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3

------------ --------------- --------------- --------------- ---------------

           0               0               0               0         287751

RkiosSysPacketMan:

Packet allocation falures: 0

Packet Buffer(Software Common) allocation falures: 0

Packet Buffer(Software ESMP) allocation falures: 0

Packet Buffer(Software EOBC) allocation falures: 0

IOS Packet Buffer Wrapper allocation falures: 0

Packets Dropped In Processing Overall

Total               5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

                 431         0         0         0         0

Packets Dropped In Processing by CPU event

Event             Total               5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Input Acl                         420         0         0         0         0

SA Miss                             11         0         0         0         0

Packets Dropped In Processing by Priority

Priority         Total               5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Medium                             11        0         0         0         0

High                               420         0         0         0         0

Packets Dropped In Processing by Reason

Reason             Total               5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt                     8         0         0         0         0

L2DstDrop                             3         0         0         0         0

NoFloodPorts                        420         0         0         0         0

Total packet queues 16

Packets Received by Packet Queue

Queue                 Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Esmp                         166023605       25       18       13         6

L2/L3Control                 99934266       15         7         3         0

Host Learning                     4353         0         0         0         0

L3 Fwd Low                           9         0         0         0         0

L2 Fwd Low                     331699         0         0         0         0

L3 Rx Low                     2333469         2         0         0         0

DistSW-4507-01#show platform health | exclude 0.00

                     %CPU   %CPU   RunTimeMax   Priority Average %CPU Total

                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour CPU

GalChassisVp-review   3.00   0.14     10     23 100 500   0   0   0 147:38

Lj-poll               1.00   0.02     2     0 100 500   0   0   0 27:18

StatValueMan Update   1.00   0.14     1     0 100 500   0   0   0 59:21

K2AclCamMan hw stats   3.00   0.55     10     5 100 500   0   0   0 437:25

K2AclPolicerTableMan   1.00   0.84     10     8 100 500   0   0   0 518:21

K2PortMan Review       2.00   0.87     15     10 100 500   0   1   0 1032:56

Gigaport9 Review       0.40   0.05     4     1 100 500   0   0   0 76:47

Gigaport11 Review     0.40   0.05     4     1 100 500   0   0   0 77:06

Gigaport16 Review     0.40   0.04     4     1 100 500   0   0   0 72:41

Gigaport18 Review     0.40   0.04     4     1 100 500   0   0   0 72:46

Gigaport25 Review     0.40   0.04     4     1 100 500   0   0   0 72:48

Gigaport39 Review     0.40   0.04     4     1 100 500   0   0   0 72:46

Gigaport48 Review     0.40   0.04     4     1 100 500   0   0   0 71:04

Gigaport49 Review     0.40   0.04     4     1 100 500   0   0   0 69:14

Gigaport50 Review     0.40   0.04     4     1 100 500   0   0   0 70:43

Gigaport51 Review     0.40   0.05     4     1 100 500   0   0   0 73:51

Gigaport52 Review     0.40   0.04     4      1 100 500   0   0   0 68:05

Gigaport53 Review     0.40   0.05     4     0 100 500   0   0   0 74:59

K2Fib Vrf Unpunt Rev   2.00   0.01     15     0 100 500   0   0   0 9:06

K2FibAdjMan Stats Re   2.00   0.50     10     8 100 500    0   0   0 213:07

K2FibMulticast Signa   2.00   0.04     10     1 100 500   0   0   0 39:10

K2FibPbr route map r   2.00   0.01     20     4 100 500   0   0   0 153:35

K2FibPbr consolidati   2.00   0.02     10     0 100 500   0   0   0 12:00

K2FibFlowCache flow   2.00   0.01     10     0 100 500   0   0   0 8:47

K2FibFlowCache adj r   2.00   0.01     10     0 100 500   0   0   0 8:47

K2MetStatsMan Review   2.00   0.07     5     2 100 500   0   0   0 55:00

K2QosDblMan Rate DBL   2.00   0.18     7     0 100 500   0   0   0 170:57

IrmFibThrottler Thro   2.00   0.01     7     0 100 500   0   0   0 9:00

K2 VlanStatsMan Revi   2.00   0.13     15     1 100 500   0   0   0 199:13

K2 Packet Memory Dia   2.00   0.04     15     8 100 500   0   0   0 754:50

K2 L2 Aging Table Re   2.00   0.03     20     2 100 500   0   0   0 138:37

K2NetflowMan - hw fl   5.00   0.03     20     1 100 500   0   0   0 29:32

K2PacketBufMonitor-P   1.00   2.58     10     1 100 500   2   2   1 2499:28

RkiosPortMan Port Re   2.00   0.05     12     8 100 500   0   0   0 115:32

Rkios Module State R   4.00   0.01     40     1 100 500  0   0   0 29:42

Rkios Online Diag Re   4.00   0.01     40     0 100 500   0   0   0 27:44

RkiosIpPbr IrmPort R   2.00   0.14     10     2 100 500   0   0   0 44:24

RkiosAclMan Review     3.00   0.03     30     0 100 500   0   0   0 56:38

EthHoleLinecardMan(1   1.66   0.01     10     24 100 500   0   0   0 31:04

EthHoleLinecardMan(3   1.66   0.13     10     4 100 500   0   0   0 235:10

                     -------------

%CPU Totals         239.93   9.50

                      Allocation ceiling       Current allocation

                       ------------------       ------------------

                       kbytes   % in use       kbytes   % in use

Linecard 1's Store       258.00       9%           24.63     100%

Linecard 3's Store       258.00       9%           24.79     100%

Linecard 4's Store       258.00     65%         168.57     100%

TSM objects           ------------------       ------------------

PacketBufRaw           25219.50     100%       25219.50     100%

PacketBufRawJumbo         732.81     25%         183.20     100%

Packet                   1158.75       2%           30.11       99%

PacketInfoItem           390.62       0%            0.01       0%

VbufNodes1600             55.75       0%           12.19       0%

VbufNodes400             292.00       0%           5.70       30%

VbufNodes64               62.00       0%           1.93       0%

MatEntryTableIterato       1.12      0%           0.03       0%

Rkios QoS ClassMaps       896.00       0%           0.02     100%

PimPhyports               882.81     25%         226.00     100%

PimPorts                 796.87     32%         255.00     100%

PimSlots                  5.00       2%           0.13     100%

PimChassis                 34.37       6%           2.14     100%

EbmPorts                 184.00     31%           57.32     100%

EbmIeNodes               536.00       0%           4.18     100%

EbmSortedHostTableIt       1.87       0%           0.05       0%

EbmSortedGroupTableI       1.75       0%           0.05       0%

IrmVrfs                   27.00       0%           0.10     100%

IrmFibAdjs               768.00       0%           0.46     100%

AclClassifier             768.00       0%           0.18     100%

AclFeature               1216.82       0%           0.42     100%

Acl                       384.00       0%           0.10     100%

AclListNode               256.00       0%           0.06     100%

AclClassifierActionL     512.00       0%           0.09     100%

CommandTables             48.00     23%           11.10     100%

K2FibVrfs                 152.00       0%           0.59     100%

K2TxPacket              384.00       0%           0.02       0%

K2TxPacketInfo           256.00       0%           0.03       0%

EbmVlanHostEntrys       3437.50       0%           2.37       94%

GalGbicEntrys             11.85       0%           0.09     100%

Event Nodes               128.00       0%           3.15       1%

Event Nodes               128.00       1%           7.15       22%

RkiosNetflowFlows       3072.00       0%           4.62       0%

                       ------------------       ------------------

TSM totals             194196.69     13%       26043.75       99%

Hi Dao,

Is the CPU high right now? When you took this output?

Regards

Inayath

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: