Solved: High CPU utilization due to Cat4k Mgmt LoPri & Cat4k Mgmt HiPri in WS-C4507R-E

Athul Chandran · ‎10-14-2013

Hi,

We are getting the HIGH CPU process due to Cat4k Mgmt Lopri & Cat4k Mgmt HiPri with both core Switches installed at the Customer site.

CPU utilization for five seconds: 52%/4%; one minute: 53%; five minutes: 53%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
54     5409040 32696027        165 41.75% 42.57% 42.59%   0 Cat4k Mgmt LoPri
53      250860    543138        461 3.75% 3.74% 3.71%   0 Cat4k Mgmt HiPri
103       96976    799723        121 0.71% 0.61% 0.61%   0 IP Input
14       88016    879464        100 0.55% 0.52% 0.54%   0 ARP Input
109       46004    231990        198 0.31% 0.34% 0.34%   0 Spanning Tree
143         848     20439         41 0.15% 0.10% 0.08%   0 CEF: IPv4 proces
38       11748     13091        897 0.15% 0.12% 0.13%   0 IDB Work
96        4284      9200        465 0.07% 0.11% 0.11%   0 CDP Protocol
202         976       326       2993 0.07% 0.00% 0.03%   2 Virtual Exec
10         104         8      13000 0.00% 0.00% 0.00%   0 PrstVbl

Suggest me on this issue facing and confirm if there is any BUG impact or some other specific reason??

Regards,

AC

Carlo Poggiarelli · ‎10-14-2013

Hi Athul

As you can see K5CpuMan Review process shows a CPU about 50%.

This is usually caused by high traffic.

Which is the topology of customer's LAN.

Can you confirm that these are managing an high traffic?

Let me know

Regards

Carlo

Please rate all helpful posts

"The more you help the more you learn"

Please rate all helpful posts "The more you help the more you learn"

View solution in original post

Carlo Poggiarelli · ‎10-14-2013

Hi Athul.

Can you please post the output of show platform healt?

Thanks

Carlo

Please rate all helpful posts

"The more you help the more you learn"

Please rate all helpful posts "The more you help the more you learn"

Athul Chandran · ‎10-14-2013

Hi Carlo,

Thanks for your reply....

sh platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

RkiosObflMan 0.50 0.00 4 0 100 500 0 0 0 0:00

GalChassisVp-review 3.00 0.09 10 42 100 500 0 0 0 0:16

S2w-JobEventSchedule 10.00 0.00 10 0 100 500 0 0 0 0:00

Stub-JobEventSchedul 10.00 0.86 10 3 100 500 0 0 0 1:54

Lj-poll 1.00 0.01 2 0 100 500 0 0 0 0:03

StatValueMan Update 1.00 0.02 1 0 100 500 0 0 0 0:06

Pim-review 0.10 0.01 1 0 100 500 0 0 0 0:00

Ebm-host-review 1.00 0.01 8 4 100 500 0 0 0 0:10

Ebm-host-util-review 1.00 0.00 10 0 100 500 0 0 0 0:00

Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 0:00

Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00

EbmHostRedundancyMan 2.00 0.00 20 0 100 500 0 0 0 0:00

Acl-Flattener 1.00 0.00 10 0 100 500 0 0 0 0:00

IrmFibThrottler Thro 2.00 0.17 7 4 100 500 0 0 0 0:10

GalChassisVp Ondeman 2.00 0.00 2 0 100 500 0 0 0 0:00

KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 0:02

KxAclPathMan update 2.00 0.00 10 0 100 500 0 0 0 0:00

KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00

GalK5TatooineStatsMa 0.70 0.02 4 0 100 500 0 0 0 0:03

MOL FastDropReview 2.00 0.00 15 0 100 500 0 0 0 0:00

RkiosL3PortMan AclFe 2.00 0.00 15 2 100 500 0 0 0 0:00

K5L3FlcMan FwdEntry 2.00 2.33 15 10 100 500 2 2 1 5:14

K5L3FlcMan Consisten 2.00 1.05 15 7 100 500 1 1 0 2:16

FwdEntry Zombie Revi 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3FlcMan Cam Shuff 4.00 0.00 25 0 100 500 0 0 0 0:00

K5L3Unciast IFE Revi 2.00 2.25 15 4 100 500 2 2 1 4:35

K5L3UnicastRpf IFE R 2.00 2.23 15 1 100 500 2 2 1 4:27

K5L3Unicast Fwd Entr 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3McastMan IrmMfib 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3McastMan ImeSync 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3Unicast Fwd Entr 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3Unicast Adj Grou 2.00 0.00 15 0 100 500 0 0 0 0:00

K5L3Unicast Adj Chan 2.00 0.00 15 1 100 500 0 0 0 0:00

K5L3Unicast Adj Tabl 2.00 8.11 15 7 100 500 3 4 2 7:00

K5L3AdjStatsMan Revi 2.00 0.08 10 7 100 500 0 0 0 1:01

K5L3McastMan RetSync 1.00 0.00 8 0 100 500 0 0 0 0:00

K5FlcHitMan review 2.00 0.00 5 2 100 500 0 0 0 0:24

K5PortMan Regular Re 2.00 0.44 15 10 100 500 0 0 0 0:59

K5PortMan Ondemand L 3.00 0.52 30 0 100 500 0 0 0 1:10

K5PortMan Stats Revi 2.00 0.00 15 0 100 500 0 0 0 0:03

K5PortMan Tx Queue R 3.00 0.00 15 0 100 500 0 0 0 0:00

K5L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 0:02

K5 L2 Aging Table Re 2.00 0.04 20 4 100 500 0 0 0 0:21

K5 L2 Unicast Addres 2.00 0.00 20 10 100 500 0 0 0 0:00

K5 L2 Multicast Addr 2.00 0.00 20 0 100 500 0 0 0 0:00

K5 L2 Hardware Addre 2.00 0.14 20 10 100 500 0 0 0 1:32

K5 L2 Hardware Mac L 1.00 0.00 2 0 100 500 0 0 0 0:00

K5RetStatsMan Review 2.00 0.00 5 0 100 500 0 0 0 0:00

K5CpuMan Review 30.00 55.89 30 42 100 500 105 99 71 203:47

K5ForerunnerPacketMa 1.50 33.07 4 0 100 500 57 54 38 105:03

K5ForerunnerPacketMa 0.70 0.27 4 0 100 500 0 0 0 0:38

K5QosDhmMan Rate DBL 2.00 5.26 7 5 100 500 5 4 3 10:36

K5QosPolicerStatsMan 1.00 0.00 10 0 100 500 0 0 0 0:00

K5VlanStatsReview 2.00 0.68 10 4 100 500 1 1 0 2:13

K5AclMan-labeledFlat 1.00 0.00 10 0 100 500 0 0 0 0:00

K5AclLabelMan-punted 1.00 0.00 10 0 100 500 0 0 0 0:00

K5AclCamMan stale en 1.00 0.00 10 5 100 500 0 0 0 0:00

K5AclCamMan Audit re 1.00 2.38 10 5 100 500 2 2 1 5:33

K5AclCamStatsMan hw 3.00 0.29 10 5 100 500 0 0 0 1:04

K5Acl Action Update 2.00 0.00 15 7 100 500 0 0 0 0:00

VSI DMA Local Jawa R 1.00 0.06 6 0 100 500 0 0 0 0:12

VSI DMA Darkside Rev 1.00 0.00 6 0 100 500 0 0 0 0:00

VSI DMA Slot-02 Revi 1.00 0.00 6 0 100 500 0 0 0 0:00

VSI DMA Slot-03 Revi 1.00 0.00 6 0 100 500 0 0 0 0:00

VSI DMA Slot-04 Revi 1.00 0.00 6 0 100 500 0 0 0 0:00

VSI DMA Slot-05 Revi 1.00 0.00 6 0 100 500 0 0 0 0:00

VSI DMA Slot-06 Revi 1.00 0.28 6 1 100 500 0 0 0 0:43

GbmHostTable Review 2.00 0.00 8 0 100 500 0 0 0 0:00

GbmHostTableUtilizat 2.00 0.00 8 0 100 500 0 0 0 0:00

GbmHostTableAging Re 2.00 0.00 8 0 100 500 0 0 0 0:00

RkiosPortMan Port Re 2.00 0.23 12 6 100 500 0 0 0 0:29

Rkios Module State R 4.00 0.01 40 1 100 500 0 0 0 0:03

Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 0:03

RkiosIpPbr IrmPort R 2.00 0.02 10 1 100 500 0 0 0 0:05

RkiosIpWccp IrmWccpW 2.00 0.00 10 0 100 500 0 0 0 0:00

RkiosAclMan Review 3.00 0.03 30 0 100 500 0 0 0 0:07

MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00

LocalJawaVsiMan VsiR 0.20 0.00 2 0 100 500 0 0 0 0:00

QuackMan( 254 ) Revi 4.00 0.00 20 4 100 500 0 0 0 0:02

GlmBridgeMan(0) revi 0.50 0.00 2 0 100 500 0 0 0 0:00

QuackMan( 0 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00

Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 0:00

Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:00

Xgstub Stats Review 0.50 0.08 5 0 100 500 0 0 0 0:15

EpmPortGroup(0) stat 0.50 0.01 2 0 100 500 0 0 0 0:01

EpmPortGroup(0) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00

EpmPortGroup(0) on d 0.50 0.00 4 0 100 500 0 0 0 0:00

X2Controller(0) 0.50 0.00 0 0 100 500 0 0 0 0:00

PluggableLinecard(0) 0.50 0.00 1 0 100 500 0 0 0 0:00

EpmPluggableGroup(0) 0.50 0.00 2 0 100 500 0 0 0 0:00

FrysSeepromMan 0.50 0.00 2 0 100 500 0 0 0 0:00

GlmBridgeMan(5) revi 0.50 0.00 2 0 100 500 0 0 0 0:00

QuackMan( 5 ) Review 4.00 0.00 20 0 100 500 0 0 0 0:00

Stub periodic global 0.50 0.00 5 0 100 500 0 0 0 0:00

Stub ondemand global 0.50 0.00 5 0 100 500 0 0 0 0:00

Xgstub Stats Review 0.50 0.10 5 0 100 500 0 0 0 0:16

EthPhyControllerMan( 0.40 0.00 4 0 100 500 0 0 0 0:00

EthPhyControllerMan( 0.20 0.00 2 2 100 500 0 0 0 0:00

GalLinecardDiag Mana 1.00 0.00 10 0 100 500 0 0 0 0:00

EpmPortGroup(5) stat 0.50 0.00 2 0 100 500 0 0 0 0:01

EpmPortGroup(5) PimE 0.50 0.00 4 0 100 500 0 0 0 0:00

EpmPortGroup(5) on d 0.50 0.00 4 0 100 500 0 0 0 0:00

-------------

%CPU Totals 200.59 117.24

Allocation ceiling Current allocation

------------------ ------------------

kbytes % in use kbytes % in use

Linecard 1's Store 600.00 35% 213.42 100%

Linecard 2's Store 600.00 36% 219.47 100%

Linecard 3's Store 600.00 34% 209.72 100%

Linecard 4's Store 600.00 0% 0.00 0%

Linecard 5's Store 600.00 0% 0.00 0%

Linecard 6's Store 600.00 89% 537.19 100%

TSM objects ------------------ ------------------

PacketInfoItem 859.37 0% 1.54 0%

VbufNodes2400 80.50 0% 0.00 0%

VbufNodes1600 55.50 0% 12.14 0%

VbufNodes400 288.00 0% 9.56 23%

VbufNodes64 60.00 0% 3.28 0%

VbufNodes4200 68.37 12% 8.54 100%

PacketBufRaw 184.29 100% 184.29 100%

PacketBufRaw 4644.56 100% 4644.56 100%

RkiosSysPacketBuf 250.00 0% 0.71 4%

Packet 2322.42 0% 0.31 0%

IndexCache 800.78 0% 0.00 0%

K5AclOpDescNode 10752.00 0% 0.00 0%

K5AclRetMapEntryNode 21.00 0% 0.00 0%

K5AclLabelListNode 768.00 0% 0.00 0%

K5AclIpv6PackedAddrH 1024.00 0% 0.00 0%

K5AdjGroups 960.00 2% 26.19 99%

IrmFibUnicastRpfList 5120.00 0% 28.08 99%

K5L3FwdEntrys 27040.00 0% 193.27 93%

K5L3FwdEntryAvlTree2 12480.00 0% 44.74 92%

K5L3FwdTreeEntrys 21840.00 0% 156.59 92%

K5L3FwdTreeEntryAvlT 49920.00 0% 44.60 92%

IrmMfibFastDropFlowM 576.00 0% 0.00 0%

K5QosTxQueSelTableBl 14.00 2% 0.38 100%

K5QosPolicerBlockNod 2.00 0% 0.00 0%

K5QosPolicerBlockMem 69.00 0% 0.00 0%

K5QosPolicerMemAlloc 448.00 0% 0.00 0%

K5QosFeatureInfoList 1536.00 0% 0.00 0%

K5QosLabelToFeatureE 1792.00 0% 0.00 0%

K5QosPathFeatureInfo 384.00 0% 0.00 0%

K5CpuPacketInfoItem 859.37 0% 0.00 0%

K5TxPacketInfo 256.00 0% 0.57 0%

MatEntrys 7680.00 0% 11.71 98%

MatEntryTableIterato 1.00 0% 0.03 0%

RkiosAclMan NamedGal 43.75 0% 0.03 50%

EpmAceListNode 192.00 0% 0.00 0%

Rkios Acl VlanMaps 128.00 0% 0.00 0%

Rkios Acl VlanMapEnt 859.37 0% 0.00 0%

Rkios EVC EFPs 1280.00 0% 0.00 0%

Rkios GBM BridgeDoma 256.00 0% 0.00 0%

Rkios GBM PseudoPort 768.00 0% 0.00 0%

RkiosQosPortVlanToEf 767.98 0% 0.00 0%

Rkios QoS PolicyMaps 356.32 0% 0.00 0%

AclClassifierIdToCla 48.00 0% 0.00 0%

Rkios QoS ClassMaps 896.00 0% 0.02 100%

AclToIosFilterMapLis 384.00 0% 0.00 0%

Rkios QoS Policers 3500.00 0% 0.00 0%

RkiosTableMap Galios 3.00 0% 0.00 0%

PimPhyports 1054.68 25% 264.72 100%

PimPorts 906.25 31% 285.46 100%

PimModules 162.00 1% 2.53 100%

PimSlots 6.00 2% 0.14 100%

PimChassis 38.25 6% 2.39 100%

PimQuack 0.93 10% 0.09 100%

EbmVlans 5088.00 1% 69.56 100%

EbmVlanGroupEntrys 3840.00 0% 0.70 100%

EbmPorts 336.00 30% 102.04 100%

EbmPortHostEntrys 448.00 0% 0.00 0%

EbmIeNodes 540.00 1% 5.80 100%

EbmPortVlanAclFeatur 2560.00 0% 0.00 0%

Ebm PV Maps 192.00 0% 0.00 0%

EbmSortedHostTableIt 1.75 0% 0.00 0%

EbmSortedGroupTableI 1.75 0% 0.05 0%

EbmHostRedundancyMan 1082.81 0% 0.00 0%

IrmVrfs 315.00 0% 2.46 100%

IrmFibLoadBalances 1152.00 0% 0.21 66%

IrmFibAdjs 1920.00 2% 53.16 100%

IrmPortEtherAddrEntr 500.00 0% 0.03 50%

IrmFibEntries 12288.00 0% 89.20 93%

IrmMfibEntryMemMan 13312.00 0% 0.30 100%

IrmWccpMemMan 34.03 0% 0.00 0%

AclOp 2048.00 0% 0.00 0%

AclOpAceSet 2559.96 0% 0.00 0%

AclClassifier 1280.00 0% 1.87 97%

AclFeature 2570.04 0% 3.11 95%

Acl 1408.00 0% 2.06 97%

Ace24 9215.85 0% 5.62 98%

Ace48 15359.76 0% 0.35 100%

AceActionDescStorage 1023.98 0% 0.00 0%

AclListNode 256.00 0% 0.37 97%

AceListNode 25600.00 0% 0.12 25%

AclClassifierActionL 1152.00 0% 1.54 97%

AclLayerFeatureListN 512.00 0% 0.07 22%

AclClassifierListNod 256.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 116.00 0% 0.00 0%

FlatAcl 512.00 0% 0.12 25%

FlatAce24 10752.00 0% 0.82 40%

FlatAce48 16896.00 0% 0.00 0%

FlatAceActionListNod 281600.00 0% 0.47 36%

FlatAclOpSetStorage 3072.00 0% 0.00 0%

FlatAclCacheNode 4224.00 0% 0.25 50%

FlatAclListNode 256.00 0% 0.04 16%

QosFeatureClassifier 515.13 0% 0.00 0%

QosFeatureClassifier 1030.26 0% 0.00 0%

QosClassifierActionL 14423.71 0% 0.00 0%

QosPortVlanAclFeatur 18544.78 0% 0.00 0%

QosEvcEfpAllocator 4608.00 0% 0.00 0%

QoS Policers 37000.00 0% 0.00 0%

MOL EvcEfp 11776.00 0% 0.00 0%

EvcEfpEncapVlanRange 1024.00 0% 0.00 0%

MOL GBM BridgeDomain 2304.00 0% 0.00 0%

MOL GBM PseudoPort 5888.00 0% 0.00 0%

GbmSortedHostTableIt 1.62 0% 0.00 0%

GbmBridgeDomainHostE 16384.00 0% 0.00 0%

KxAclPath 1024.00 0% 0.06 50%

KxAclPathListNode 384.00 0% 0.00 0%

KxAclConstPathListNo 384.00 0% 0.05 14%

EpmPortGroups 0.71 28% 0.20 100%

EpmPortMans 61.75 21% 13.50 100%

KxAclLabeledFlatAcl 3584.00 0% 0.21 50%

KxAclLabeledFlatAclE 3072.00 0% 0.18 50%

EbmVlanHostEntrys 3222.65 2% 77.10 92%

K5L3FlcEntryAvlTree2 12902.39 0% 89.20 92%

VsiBuffers3072 300.00 0% 6.00 0%

VsiBuffers1024 1500.00 6% 101.00 99%

VsiBuffers128 250.00 0% 7.12 0%

VsiBuffers16 31.25 0% 4.68 4%

VsiTransactions(Max 123.04 0% 6.09 15%

VsiTransactions(Max 159.96 4% 15.38 51%

VsiTransactions(Max 15.42 0% 0.00 0%

VsiTransactions(Max 5.90 0% 0.00 0%

VsiTransactions(Max 4.83 0% 0.00 0%

VsiTransactionRespon 17.44 2% 0.85 55%

VsiS2wRequestMans 54.68 0% 1.25 0%

VsiVliRequestMans 147.65 5% 10.68 73%

VsiMdio22RequestMans 273.43 0% 22.50 0%

VsiMdio45RequestMans 30.07 0% 0.34 0%

VsiJtagRequestMans 5.46 0% 0.00 0%

GalGbicEntrys 11.57 0% 0.09 100%

EbmVlanBroadcastFloo 1408.00 1% 19.25 100%

Event Nodes 128.00 0% 0.00 0%

Event Nodes 128.00 2% 10.59 28%

RkisoIpPbrRouteMaps 97.65 0% 0.00 0%

Rkios QoS EVC EFPs 768.00 0% 0.00 0%

K5AclLabelSignatureM 10240.00 0% 0.62 50%

K5AclLabelMapEntryPa 1408.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 116.00 0% 0.00 0%

InpTosMarkTbl BlockA 1.75 12% 0.21 100%

InpCosMarkTbl BlockA 1.75 12% 0.21 100%

InpExpMarkTbl BlockA 1.75 12% 0.21 100%

OutTosMarkTbl BlockA 1.75 12% 0.21 100%

OutCosMarkTbl BlockA 1.75 12% 0.21 100%

OutExpMarkTbl BlockA 1.75 12% 0.21 100%

CommandTables 48.00 14% 6.75 100%

------------------ ------------------

TSM totals 744559.67 0% 6662.09 97%

Thanks And regards,

AC

Carlo Poggiarelli · ‎10-14-2013

Hi Athul

As you can see K5CpuMan Review process shows a CPU about 50%.

This is usually caused by high traffic.

Which is the topology of customer's LAN.

Can you confirm that these are managing an high traffic?

Let me know

Regards

Carlo

Please rate all helpful posts

"The more you help the more you learn"

Please rate all helpful posts "The more you help the more you learn"

Athul Chandran · ‎10-14-2013

HI Carlo,

Thanks again...

Yes, These are managing a high traffic...

So what ill the solution for same... please advise.

Thanks,

AC

Carlo Poggiarelli · ‎10-14-2013

Hi Athul

In this case you have to identify the root cause of high traffic and you can use the following useful link

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

HTH

Regards

Carlo

Please rate all helpful posts

"The more you help the more you learn"

Please rate all helpful posts "The more you help the more you learn"

Athul Chandran · ‎10-21-2013

Hi Carlo,

I have checked the traffic, and found all are valid. In this case what will the solution? is it CPU upgradation or something else?

Thanks,

AC

ahmad82pkn · ‎10-02-2014

Just want to share i resolved my issue by disabling ip redirects in my interface Vlans. there was one Vlan that was causing the issue, and CPU dropped from 80% to 30%. Happy days.

sushil.patel1 · ‎01-07-2019

Hi Budy,

can you please let me know is this command will impact in live environment, as we are having cisco catalyst 4507 and that is being used for LAN distribution with number of vlans, and presently the cpu utilisation is avg 55%.

Also if i apply no proxy command so is any other way to roll back on previously mode?

Regards,

SP

Athul Chandran · ‎10-31-2013

Hi all,

issue has been resolved...

it is becase of MAC address learning, disabled proxy-arp (no proxy-arp) in all VLAN interfaces. Suddenly cpu utilization went down from 70% to 16%

Thanks to all, who are supported me...

regards,

AC