High CPU on 3750X stack - how do I find the root cause ?

Shlomy Maron · ‎06-17-2015

Hi I have a stack of 3 members of 3750X -

WS-C3750X-48P-L (Version ID V03)
WS-C3750X-48P-L (Version ID V03)
WS-C3750X-48T-S (Version ID V05)

running version 15.0.2(SE7)

I have an issue with high CPU - and I can't seem to find the reason.

CPU utilization for five seconds: 82%/4%; one minute: 86%; five minutes: 85%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
278 74815021 1958023 38209 31.58% 32.91% 31.05% 0 VMATM Callback
184 105870616 11282890 9383 20.78% 19.55% 19.46% 0 Hulc LED Process
90 20996700 3830040 5482 3.01% 3.69% 3.69% 0 RedEarth Tx Mana
89 14369143 5776877 2487 2.69% 3.33% 3.28% 0 RedEarth I2C dri
394 4447481 162920 27298 2.37% 2.06% 1.95% 0 SpanTree Flush
111 5232820 6711632 779 1.58% 2.04% 2.12% 0 HRPC hlfm reques
136 8598337 493284 17430 1.42% 2.09% 2.11% 0 hpm counter proc

888888888999999999988888888888888888888777778888899999999999
555500000888884444477777222226666677777888887777722222555557
100 ***** ****
90 **** *************** ********** **************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

998999999999999999899998999999999999998899998999899999898998
999899380986254993803759892024997591969986899989410459697649
100 ** *** * *** * ** ** ** ***** ** **** *** ** * *
90 ##*#*****###****#****#**##******#*#*#***#**#*#*# *********
80 ##########################################################
70 ##########################################################
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

11 1 1 1 1 1 1
999989999999009909989999999099999999990999099999999996989999990979999990
999949599999009909918999999099699999990999099999999998913999990909999990
100 **** ************** ********************************* * ******* *****
90 **** ************** ********************************* * ******** *****
80 #*#*********##*###***##*#####****######**###########* ******###* ****#
70 ###****#***###*###***########**#*######*############**#**##*###***#*##
60 ###**#*#*#########***#########*######################*#**#######**#*##
50 ###**#*############**#########*######################*#**#######*#####
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

I dont see a lot of mac addresses or igmp groups.

any suggestions ?

Leo Laohoo · ‎06-17-2015

Post the output to the command "sh proc cpu sorted | ex 0.00".

Try IOS version 12.2(55)SE10.

Shlomy Maron · ‎06-17-2015

it would be quite difficult to change the IOS due to the down time.

is there any reference that shows that this is what needed to be done ?

#sh processes cpu sorted | ex 0.0
CPU utilization for five seconds: 76%/6%; one minute: 81%; five minutes: 81%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
278 75628443 1979485 38206 24.79% 27.48% 27.73% 0 VMATM Callback
184 106425123 11335619 9388 19.51% 19.89% 19.83% 0 Hulc LED Process
90 21101379 3849034 5482 4.00% 3.70% 3.68% 0 RedEarth Tx Mana
89 14461888 5806559 2490 2.71% 3.23% 3.23% 0 RedEarth I2C dri
136 8657329 495832 17460 2.23% 2.13% 2.09% 0 hpm counter proc
394 4498629 164453 27355 2.07% 1.83% 1.79% 0 SpanTree Flush
111 5297838 6778692 781 2.07% 2.05% 2.15% 0 HRPC hlfm reques
99 5613072 5034364 1114 1.91% 1.43% 1.57% 0 hrpc <- response
270 5882556 481458 12218 1.27% 1.19% 1.17% 0 PI MATM Aging Pr
334 1659692 1307458 1269 0.63% 0.43% 0.42% 0 IGMPSN
220 1389602 723225 1921 0.47% 0.46% 0.37% 0 CDP Protocol
252 2415226 927309 2604 0.47% 0.56% 0.59% 0 Spanning Tree
200 2103422 98488 21357 0.47% 0.39% 0.41% 0 HQM Stack Proces
112 2610422 10618499 245 0.47% 0.76% 0.79% 0 HLFM address lea
329 572630 1064729 537 0.15% 0.14% 0.16% 0 Marvell wk-a Pow
201 1795372 585522 3066 0.15% 0.32% 0.32% 0 HRPC qos request
46 449700 1079757 416 0.15% 0.14% 0.12% 0 Net Background
428 816769 1933432 422 0.15% 0.22% 0.23% 0 VLAN Manager
271 572141 4957129 115 0.15% 0.19% 0.20% 0 UDLD
132 984082 5831916 168 0.15% 0.28% 0.27% 0 hpm main process
333 722077 2528257 285 0.15% 0.13% 0.15% 0 IGMPSN MRD

Leo Laohoo · ‎06-17-2015

VMATM Callback

You get this message when the appliance is having failed memory allocation error.

Hulc LED Process

Process which is responsible for link status detection.

3750 series family is susceptible to high CPU once the appliance is running IOS higher than 12.2(55)SE.

What is the uptime of this appliance?

Madhukrishnan Gopinathan Nair · ‎06-17-2015

Hi,

VMATM callback - this process mainly takes care of removing the addresses learned on a
particular interface when the link goes down. Can you please check for any continuous link flap or spanning tree topology change in the network? If so try shutting down the port to see if it helps.

Also try enabling mac move notification

conf t

mac address-table notification mac-move

HTH,

Madhu.

Shlomy Maron · ‎06-25-2015

Hi,

Thanks for your help- and sorry for the late response.

I've used the command you've mentioned and I dont think that this is the issue -

SW-4070-STACK#sh mac address-table move update
Switch-ID : 01d3.f16b.7b80
Dst mac-address : 0180.c200.0010
Vlans/Macs supported : 1023/6400
Default/Current settings: Rcv Off/Off, Xmt Off/Off
Max packets per min : Rcv 40, Xmt 60

Rcv packet count : 0
Rcv conforming packet count : 0
Rcv invalid packet count : 0
Rcv packet count this min : 0
Rcv threshold exceed count : 0
Rcv last sequence# this min : 0
Rcv last interface : None
Rcv last src-mac-address : 0000.0000.0000
Rcv last switch-ID : 0000.0000.0000

Xmt packet count : 0
Xmt packet count this min : 0
Xmt threshold exceed count : 0
Xmt pak buf unavail cnt : 0
Xmt last interface : None

and I dont see anything on the log regarding mac.

# sh processes cpu sorted | ex 0.0
CPU utilization for five seconds: 82%/31%; one minute: 85%; five minutes: 87%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
184 257909322 27583048 9350 24.48% 26.63% 26.75% 0 Hulc LED Process
234 645804 2995995 215 4.48% 4.87% 4.91% 0 IP Input
89 35565556 13836324 2570 4.16% 3.41% 3.40% 0 RedEarth I2C dri
136 23538451 1181067 19929 2.71% 2.35% 2.25% 0 hpm counter proc
90 48978905 9179470 5335 2.71% 3.59% 3.57% 0 RedEarth Tx Mana
270 14731238 1148267 12829 1.75% 1.55% 1.58% 0 PI MATM Aging Pr
388 3851 447 8615 0.95% 0.17% 0.15% 1 Virtual Exec
334 4129434 3124891 1321 0.79% 0.73% 0.88% 0 IGMPSN
200 5024387 234615 21415 0.47% 0.40% 0.41% 0 HQM Stack Proces
252 6054015 2306636 2624 0.47% 0.41% 0.41% 0 Spanning Tree
112 5195924 26115894 198 0.31% 0.21% 0.18% 0 HLFM address lea
137 2100818 2652591 791 0.31% 0.16% 0.16% 0 HRPC pm-counters
132 2129936 14213569 149 0.15% 0.17% 0.20% 0 hpm main process
99 9727369 9840062 988 0.15% 0.10% 0.10% 0 hrpc <- response
329 1268818 2545877 498 0.15% 0.13% 0.15% 0 Marvell wk-a Pow

any other suggestions please ?