High CPU usage on 3750X - IOS Version 15.0(2)SE

Luis Ruiz Lopez · ‎02-27-2013

Hi all,

I have a 2 Cisco 3750X stacked working as CORE switches but not in production yet showing a high CPU utilization and I would like to see with you whether you think this is normal or a IOS upgrade/downgrade should be performed.

I have very low traffic coming through as this is not yet in production:

122222333331111122222222222222222222222221111122222222222222
      933333999999999922222777772222277777333339999922222111111111
100
   90
   80
   70
   60
   50
   40       *****
   30       *****          *****     *****
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

      333353333334333323333333334344334333332333233633333333433633
      513408436624643485444599150406970864877332906542922150079867
100
   90
   80
   70                                              *           *
   60                                              *           *
   50     *                        *               *           *
   40 *   ** ** **    *   *** ** ******* **      ** *   * ****
   30 ****#****#****#************************#*****#***********#
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      865676675874764689949779996465674565756879967585777565676656659656566578
      847000908247569559862378775817559844618889912250393503931130571190370443
100                  **    ***               **
   90 *               *** * ***             * **   *               *
   80 *        * *   *** * ****     *    * ****   * *            *
   70 *   * ** ** ** **** *******   **    * ***#* * * ***   **    * *    *
   60 *********** ** **** ******* **** ** * ***#*** * ***** **** ******* **
   50 *****************************************#****************************
   40 *****************************************#****************************
   30 *******#*********#***####******#*********##***#********#*******#######
   20 ######################################################################
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Part of the sorted process are:

CPU utilization for five seconds: 23%/0%; one minute: 26%; five minutes: 25%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

180 116840262 42936753 2721 7.02% 6.14% 6.14% 0 Hulc LED Process

88 52293417 8497839 6153 3.19% 2.93% 2.90% 0 RedEarth Tx Mana

424 24527333 96889765 253 1.43% 0.87% 0.82% 0 HSRP Common

112 18459496 46660575 395 0.95% 0.70% 0.65% 0 HLFM address ret

366 4883845 53837164 90 0.63% 0.21% 0.20% 0 MMON MENG

87 4691193 12761808 367 0.63% 0.26% 0.29% 0 RedEarth I2C dri

110 32978123 46660435 706 0.47% 1.14% 1.13% 0 HLFM address lea

227 17415346 46660576 373 0.31% 0.68% 0.61% 0 IP ARP Retry Age

228 4224616 19985195 211 0.31% 0.67% 0.40% 0 IP Input

428 3034117 11296136 268 0.31% 0.18% 0.16% 0 IP SNMP

99 1520217 8622262 176 0.31% 0.05% 0.02% 0 yeti2_emac_proce

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

247 3866133 6217902 621 0.31% 0.25% 0.23% 0 Spanning Tree

194 6033206 344073 17534 0.31% 0.33% 0.31% 0 HQM Stack Proces

195 3785520 1373988 2755 0.15% 0.16% 0.19% 0 HRPC qos request

336 672837 3326141 202 0.15% 0.02% 0.00% 0 Timer Library

55 355897 345782 1029 0.15% 0.03% 0.00% 0 Compute load avg

430 8579240 6636326 1292 0.15% 0.50% 0.60% 0 SNMP ENGINE

184 92412 691515 133 0.15% 0.02% 0.00% 0 HRPC hl3u reques

333 1944121 17145973 113 0.15% 0.04% 0.05% 0 MDFS RP process

263 1161488 1711240 678 0.15% 0.07% 0.06% 0 PI MATM Aging Pr

234 2888805 16999306 169 0.15% 0.09% 0.09% 0 MDFS MFIB Proces

425 378624 5834851 64 0.15% 0.05% 0.01% 0 HSRP IPv4

391 2361 2851 828 0.15% 1.40% 0.53% 2 Virtual Exec

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

388 4511538 11777258 383 0.15% 0.16% 0.15% 0 PM Callback

24 513977 1674911 306 0.00% 0.00% 0.00% 0 IPC Periodic Tim

25 475712 1674911 284 0.00% 0.01% 0.00% 0 IPC Deferred Por

23 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

28 18049 98744 182 0.00% 0.00% 0.00% 0 IPC Check Queue

26 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve

27 19364 229718 84 0.00% 0.00% 0.00% 0 IPC Seat Manager

22 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv

32 16341 344963 47 0.00% 0.00% 0.00% 0 IPC Loadometer

33 33 4 8250 0.00% 0.00% 0.00% 0 PrstVbl

34 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

35 0 1 0 0.00% 0.00% 0.00% 0 Exception contro

It is a 48port switch but I am only using 10 ports.

It is configured with few ip prefix list and 3 route-maps but not a heavy config.

Any recommendation?

Thanks

Luis Ruiz Lopez · ‎02-27-2013

Just wanted to add that when I get into details following the troubleshooting procedure, everything seem normal but in the cpu history I can see it is between 80 and 100% most of the time.

Thanks

Giuseppe Larosa · ‎02-27-2013

Hello Luis,

I would try an IOS upgrade specially because they are not already in production.

I have tried to make a bug search but I didn't find a match for C3750-X, 15.0(2)SE, high cpu

However, this looks like a SW bug and if I'm not wrong somebody else reported similar issues in the forums.

Hope to help

Giuseppe

Leo Laohoo · ‎03-02-2013

Hi Giuseppe,

I'm going to wait until Wilson's finished with his troubleshooting before putting my opinion on board.

Wilson Bonilla · ‎03-02-2013

Hello

Luis you're seeing only spikes on CPU, it's not real CPU utilization, and we don't even know the process that is spiking. First of all do the following to find out what is process spiking and keep an eye in the show log.

SUMMARY STEPS
1. enable

2. configure terminal

3. process cpu threshold type {total | process | interrupt} rising percentage interval seconds [falling percentage interval seconds]

Sent from Cisco Technical Support iPhone App

Sent from Cisco Technical Support iPhone App