switch High cpu Catalyst 4500 Version 12.2(53)SG5

sdwan_mike · ‎03-10-2014

I need help trying to understand the following.

                     %CPU   %CPU    RunTimeMax   Priority Average %CPU Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour CPU
GalChassisVp-review    3.00   0.10     10     45 100 500    0   0    0 701:22
K2AclCamMan hw stats   3.00   1.48     10      5 100 500    0   0    0 3265:20
K2AclCamMan kx stats   1.00   0.10     10      5 100 500    0   0    0 1917:38
K2L2 Address Table R   2.00 72.93     12      5 100 500   95 87   66 458521:03
K2PortMan Review       3.00   4.73     15     11 100 500    5   5    4 28199:47
K2Fib Consistency Ch   1.00   0.19      5      2 100 500    0   0    0 1082:02
K2FibPbr route map r   2.00   0.21     20      5 100 500    0   0    0 2031:50
K2QosDblMan Rate DBL   2.00   0.11      7      1 100 500    0   0    0 662:26
K2 VlanStatsMan Revi   2.00   0.39     15      2 100 500    0   0    0 2062:11
K2PacketBufMonitor-P   3.00   2.76     10      1 100 500    2   2    2 13968:10
RkiosPortMan Port Re   2.00   0.30     12     29 100 500    0   0    0 1744:23
RkiosIpPbr IrmPort R   2.00   0.22     10      3 100 500    0   0    0 342:05
Grandprix 1-1 Stub R   2.00   0.17     15      5 100 500    0   0    0 1244:55
Grandprix 1-2 Stub R   2.00   0.14     15      5 100 500    0   0    0 866:57
Grandprix 1-3 Stub R   2.00   0.13     15      5 100 500    0   0    0 840:36
Grandprix 1-4 Stub R   2.00   0.16     15      5 100 500    0   0    0 846:55
Grandprix 1-5 Stub R   2.00   0.14     15      5 100 500    0   0    0 855:31
Grandprix 1-6 Stub R   2.00   0.15     15    124 100 500    0   0    0 1051:18
Grandprix 1-1 Stats    2.00   0.64      4      2 100 500    0   0    0 3151:58
Grandprix 1-2 Stats    2.00   0.48      4      2 100 500    0   0    0 675:30
Grandprix 1-3 Stats    2.00   0.20      4      2 100 500    0   0    0 567:34
Grandprix 1-6 Stats    2.00   0.35      4      2 100 500    0   0    0 2072:59
                     -------------
%CPU Totals          241.20 101.19

                       Allocation ceiling        Current allocation
                       ------------------        ------------------
                       kbytes    % in use        kbytes    % in use

Linecard 1's Store        258.00      62%          161.12      100%
TSM objects            ------------------        ------------------
PacketInfoItem            859.37       0%            0.64        0%
VbufNodes1600              55.50       0%            5.20        0%
VbufNodes400              288.00       0%            1.12       50%
PacketBufRaw            25219.50     100%        25219.50      100%
PacketBufRawJumbo         184.29     100%          184.29      100%
Packet                   2079.42       0%           54.10        0%
PimPhyports              1054.68       5%           54.84      100%
PimPorts                  906.25      11%          105.12      100%
PimModules                162.00       0%            0.63      100%
PimChassis                 38.37       6%            2.39      100%
EbmVlans                 5088.00       1%           67.07      100%
EbmPorts                  344.00      11%           38.63      100%
IrmVrfs                   315.00       0%            2.46       50%
IrmFibEntries           12288.00       0%            0.75        6%
IrmMfibEntryMemMan       6656.00       0%            0.10      100%
Acl                      1536.00       0%            1.87      100%
Ace24                    9215.85       0%            4.92      100%
Ace48                   15359.76       0%            0.35      100%
AclListNode               256.00       0%            0.30      100%
AclClassifierActionL     1152.00       0%            1.40       87%
CommandTables              48.00      22%           10.59      100%
K2FibVrfs                 152.00       0%            1.18       50%
K2TxPacketInfo            256.00       0%            0.25        0%
                       ------------------        ------------------

Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3
------------ --------------- --------------- --------------- ---------------
           1               0               0               0             822
           2               0         7538177               0               0

RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
881 0 0 0 0

Packets Dropped In Processing by CPU event

Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
SA Miss 874 0 0 0 0

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Normal                               7         0         0         0          0
Medium                             881         0         0         0          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
SrcAddrTableFilt                     87         0         0         0          0
STPDrop                             771         0         0         0          0
L2DstDrop                            21         0         0         0          0
NoDstPorts                            2         0         0         0          0

Total packet queues 16

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control                7269264609       236       268       214        205
Host Learning               3617387874        38        49        38         53
L3 Fwd Low                    21851501         0         0         0          0
L2 Fwd Low                     6786759         0         0         0          0
L3 Rx High                        1582         0         0         0          0
L3 Rx Low                        10306         0         0         0          0

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control                     73992         0         0         0          0
Host Learning                  7561249         0         0         0          0

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control 7269264609 236 268 214 205

PacketBufRaw 25219.50 100% 25219.50 100%
PacketBufRawJumbo 184.29 100% 184.29 100%

Both of these seem a bit high don't you think?

If you have any idea as to why this is happening. Or is this normal?

Thanks

Jose Solano · ‎03-10-2014

Hi,

This could be related to L2 issues like a loop, see that the K2L2 Address Table R process has a target of 2.00% and is currently running at 72.93%, which is way over the expected target, this can be an indication of host/mac flapping issues. Also the host learning queue seems very high and the L2/L3 control.

You can enable mac notification mac move command to see if on the logs are any flapping issues. Also below is a link which is very helpful when dealing with high cpu issues on 4500.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

Hope this helps.