3750G CPU Use

[eric-owens]

First off, I know there was another topic somewhere in the discussion boards about 3750 high CPU utilzation which I've referenced.  Just looking for some feed back on my scenario here.

We're experiencing some intermittent network issues and I'm starting to think it's stemming from our core 3750 stack.  I have (2) WS-C3750G-48PS switches stacked.  They are SW Image C3750-IPSERVICESk9-M and running 12.2(58)SE2. 

When we experience network "slowness" I've noticed CPU spikes on the switch stack.  I don't know what tends to be average vs. unhealthy.  I've pasted the output for "show proc cpu history" even though it doesn't paste in too well, you can kind of get the picture.  

Core-1-3750Stack#sh processes cpu history

      222222111113333333333444445555544444222221111111111111111111
      144444777777777766666444441111100000888887777777777999998888
  100
   90
   80
   70
   60
   50                           *****
   40            *************************
   30            ******************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

      552355455625555656688898999899999899899986687566855656565467
      115495777085367281159907011601221901812068456710343090429201
  100
   90                    *##**##***#####*#*#*#*  *
   80                    *################*###*  **   *
   70                    ######################* **   *
   60     ** *** * **#***#######################*#***#*  *** **
   50 **  *#*#** ****##########################################
   40 **  ###### #*############################################*
   30 ##*#######*###############################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      999999998977676986766867666677687666667976676767666666766767666966667999
      991222220032875014986366444523273857631109937787879745047739986154952012
  100 **
   90 ******** *     *               *       *                       *     *
   80 **********   * ** *  * *       *       *     * *         * *   *     *
   70 ***************** ******   *** ****** ************** ** ** ****** ****
   60 ******#***************************************************************
   50 #*****#***************************************************************
   40 ##*#*##***************************************************************
   30 ########*************************************************************#
   20 ######################################################################
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0

Additionally, here is cpu process sorted:

CofR-Core-1-3750Stack#show processes cpu sorted
CPU utilization for five seconds: 57%/29%; one minute: 47%; five minutes: 43%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 161   108718387   107210878       1014 10.32%  9.09%  7.80%   0 Hulc LED Process 
 102    34027334      357495      95184  5.08%  1.07%  0.89%   0 HULC Tcam Memory 
  96    52608505   102581449        512  2.31%  2.39%  2.19%   0 HLFM address lea 
  82    48842354   183628533        265  1.23%  1.35%  1.34%   0 Fifo Error Detec 
 121    40852019     7533520       5422  0.92%  1.15%  1.09%   0 hpm counter proc 
  76    22245248    98530442        225  0.46%  0.46%  0.49%   0 RedEarth Tx Mana 
 238    26135643    40775562        640  0.46%  0.70%  0.78%   0 Spanning Tree    
 171    14668729      715024      20515  0.30%  0.38%  0.38%   0 HQM Stack Proces 
 305     1276077   112200200         11  0.30%  0.05%  0.01%   0 MMON MENG        
 224     7786230    14162648        549  0.30%  0.44%  0.36%   0 IP Input         
 172     5682987     2842030       1999  0.15%  0.16%  0.16%   0 HRPC qos request 

Does any of this out of the ordinary?  I want to make sure my core switch isn't going to die on me anytime soon :)  

Thank you!

[Leo Laohoo]

Use 12.2(55)SE9.

[eric-owens]

Thanks.  I think I'll give 12.2(55)SE9 a try.  Appears to be the recommended version.  

[InayathUlla Sharieff]

Few links to ts the high cpu:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

HTH

Regards

Inayath

[InayathUlla Sharieff]

also I have created the document for the same;

https://supportforums.cisco.com/document/12298401/troubleshooting-high-cpu-3750

[jubair151]

dear eric-owens,

Your high cpu is because of interrupt traffic. As per my experience, myself first check whether any end systems are generating any high multicast/broadcast. You may please check all access port input traffic  ( show interface | include input traffic ) size, you may get one system will keep broadcasting/multicasting the traffic which may be around 3 to 5 Mb constantly. 

Regards,

Jubair.S

[eric-owens]

Thanks for the suggestion Jubair.  I may not be following you're suggestion or maybe this isn't the issue, but when I do a (show interface | inc input traffic) I don't see any results.

I'm not sure where to look for specifically for any interrupt traffic.  Maybe it's a good thing if I don't see the output?  I'm not sure.

Core-1-3750Stack#show interfaces | inc input traffic
Core-1-3750Stack#

Thanks.

[jubair151]

Dear eric-owens,

apologies, it is show interface |  inc input rate, kindly check what are the highly utilized interfaces and trace it.

Regards

Jubair.S