Help > High CPU Utilization on switch 2960

axel.tadoy · ‎02-17-2012

Hi guys I am a new network engineer on a retail company and I am having a problem on the High utilization of my 3-stack switch 2960 only during Peak Hours of the Store Sales.

Help me guys how to start the troubleshooting or just finding the root cause of the issue.

Note: on Peak hours, it is peaking at 90-95 %

here is the utilization after Peak hours:

4444333334444433333333333333333333333333333333333333333333

0000777772222277777666664444499999777775555555555777777777

100

90

80

70

60

50

40 ************************ *****************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

5444477444444444444444444444444444444444449944474494448884

2613722302521251534141155214351414281521729996297532323006

100 **

90 ** *

80 ** * * ***

70 ** ** * * ***

60 #* ** * * ***

50 ** *#* * * * ** * * * * ##** ***# #***

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

9888888877778778887887787878888888887878778878878787887877778889778777

9018162299883990028209808094001641268290884183080908289199984044980898

100 *

90 * * * * * * *

80 **********************************************************************

70 *###**#############################################*##################

60 *####*################################################################

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

rsimoni · ‎02-18-2012

Hi Axel,

please refer to this guide to troubleshoot high CPU condition on DSBU switches (2960 has similar architecture of 3750 so this guide applies to your switch too).

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Riccardo

glen.grant · ‎02-18-2012

Also look for bugs in the version you are using the bug toolkit . There have code issues in the past with similar units and it's been a code issue . Do a show proc cpu sort and see what the biggest user is .

axel.tadoy · ‎02-21-2012

Hi Guys,

Here is the cpu level during Peak Hours.

@Riccardo thanks for the reference but I could not see the root cause of the issue on that docu..

3333377777777777777777777777777777766667777766666666666666

3333300000222220000000000111112222299991111188888555559999

100

90

80

70 *****************************************************

60 *****************************************************

50 *****************************************************

40 *****************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

7777777777777777777787777777777777777777777777777777777777

2648345456554653365508665445554456534664383566543655556436

100

90

80 * * * **** ** ******** *** *** ** * **** ****** *

70 ##########################################################

60 ##########################################################

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

8887877777887787887887778888888887777887777787887789877888778887797787

0328389899019737207368980220175426887278887828838700787722772137948639

100

90 * ** * * ** * *

80 **********************************************************************

70 ######################################################################

60 ######################################################################

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

-----------------------------------------------------------------------------------------------------------------------------------------------

CPU utilization for five seconds: 35%/10%; one minute: 51%; five minutes: 65%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

136 3145029520 278301852 11300 17.09% 17.61% 17.48% 0 Hulc LED Process

105 168018946 30730056 5467 1.19% 0.97% 0.95% 0 hpm counter proc

190 287984354 84721905 3399 0.99% 1.35% 1.43% 0 Spanning Tree

146 29735054 17852521 1665 0.29% 0.18% 0.17% 0 HRPC qos request

145 39258409 2978472 13180 0.19% 0.22% 0.21% 0 HQM Stack Proces

106 11764285 63445692 185 0.19% 0.10% 0.09% 0 HRPC pm-counters

37 25174121 14854487 1694 0.09% 0.12% 0.11% 0 Per-Second Jobs

140 346 168 2059 0.09% 0.17% 0.07% 1 SSH Process

74 17867785 73877403 241 0.09% 0.08% 0.09% 0 hrpc <- response

43 5373851 2979714 1803 0.09% 0.03% 0.03% 0 Compute load avg

204 5070516 14854389 341 0.09% 0.02% 0.00% 0 PI MATM Aging Pr

101 19749930 138153106 142 0.09% 0.09% 0.09% 0 hpm main process

13 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

12 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

15 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage

11 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

14 132 6 22000 0.00% 0.00% 0.00% 0 Entity MIB API

18 238166 14854590 16 0.00% 0.00% 0.00% 0 IPC Periodic Tim

16 4778 248525 19 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

20 213487 14854593 14 0.00% 0.00% 0.00% 0 IPC Deferred Por

21 27 48 562 0.00% 0.00% 0.00% 0 IPC Seat Manager

axel.tadoy · ‎02-21-2012

Also,

Here are the Highest talkers on the lan:

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

-------------------------------------------------------------------------

* Vlan1 0 0 0 0 2000 2 2000 2 0

* GigabitEthernet1/0/1 0 0 0 3 30000 26 61000 46 0

* GigabitEthernet1/0/2 0 0 0 1 19000 17 6000 6 0

* GigabitEthernet1/0/3 0 0 0 0 3000 3 5000 6 0

* GigabitEthernet1/0/5 0 0 0 337 4000 3 5000 5 0

* GigabitEthernet1/0/6 0 0 0 0 3000 2 4000 5 0

* GigabitEthernet1/0/7 0 0 0 390 4000 3 5000 6 0

* GigabitEthernet1/0/9 0 0 0 353 4000 3 5000 6 0

* GigabitEthernet1/0/10 0 0 0 328 3000 2 5000 6 0

* GigabitEthernet1/0/11 0 0 0 0 4000 3 5000 5 0

* GigabitEthernet1/0/22 0 0 0 177 0 0 92000 53 0

* GigabitEthernet1/0/29 0 0 0 0 0 0 93000 54 0

* GigabitEthernet1/0/30 0 0 0 0 13000 9 257000 21 0

* GigabitEthernet1/0/38 0 0 0 0 0 0 20000 16 0

* GigabitEthernet1/0/39 0 0 0 0 4000 3 25000 27 0

* GigabitEthernet1/0/40 0 0 0 0 15000 16 127000 37 0

* GigabitEthernet1/0/41 0 0 0 0 2000 0 28000 27 0

* GigabitEthernet1/0/42 0 0 0 0 1000 0 28000 28 0

* GigabitEthernet1/0/43 0 0 0 2157413 13000 13 69000 39 0

* GigabitEthernet1/0/44 0 0 0 0 2000 1 28000 26 0

* GigabitEthernet1/0/45 0 0 0 0 0 0 28000 27 0

* GigabitEthernet1/0/46 0 0 0 0 2000 3 27000 28 0

* GigabitEthernet1/0/47 0 0 0 0 89000 51 0 0 0

* GigabitEthernet1/0/48 0 0 0 888 3213000 313 2686000 314 0

* GigabitEthernet2/0/7 0 0 0 8 0 0 64000 47 0

* GigabitEthernet2/0/14 0 0 0 408 5000 3 16000 5 0

* GigabitEthernet2/0/38 0 0 0 0 2492000 201 0 0 0

* GigabitEthernet2/0/39 0 0 0 0 1000 0 35000 26 0

* GigabitEthernet2/0/41 0 0 0 0 6000 3 34000 38 0

* GigabitEthernet2/0/42 0 0 0 0 0 0 34000 37 0

* GigabitEthernet2/0/43 0 0 0 0 18000 0 57000 32 0

* GigabitEthernet2/0/44 0 0 0 0 0 0 37000 30 0

* GigabitEthernet2/0/45 0 0 0 0 0 0 36000 21 0

* GigabitEthernet3/0/4 0 0 0 0 0 0 18000 1 0

* GigabitEthernet3/0/5 0 0 0 0 0 0 79000 24 0

* GigabitEthernet3/0/13 0 0 0 0 0 0 18000 1 0

* GigabitEthernet3/0/14 0 0 0 782 0 0 18000 1 0

* GigabitEthernet3/0/16 0 0 0 0 1000 1 18000 1 0

* GigabitEthernet3/0/21 0 0 0 0 1574000 181 35000 63 0

* GigabitEthernet3/0/26 0 0 0 0 0 0 107000 48 0

* GigabitEthernet3/0/28 0 0 0 0 60824000 5017 403000 741 0

* GigabitEthernet3/0/29 0 0 0 0 0 0 2533000 207 0

* GigabitEthernet3/0/33 0 0 0 0 0 0 1604000 181 0

* GigabitEthernet3/0/34 0 0 0 0 58000 100 2535000 220 0

* GigabitEthernet3/0/36 0 0 0 3 401000 740 63358000 5238 0