High cpu question catalyst 3750

[Nicolas Maton]

I have a very strange high cpu issue on my stacked CAT3750 switch. 

Since yesterday night I have a minimuum of 60 % cpu, when i look at the sorted cpu and calculate the total i don't get even near that 60%.

We are running : 

System image file is "flash:/c3750-ipservicesk9-mz.122-55.SE5.bin"

Switch/Stack Mac Address : 0025.838e.5680

                                           H/W   Current

Switch#  Role   Mac Address     Priority Version  State 

----------------------------------------------------------

 1       Member a8b1.d468.1f00     1      0       Ready               

*2       Master 0025.838e.5680     1      0       Ready               

sh proc cpu sorted 

CPU utilization for five seconds: 65%/29%; one minute: 63%; five minutes: 63%

 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 

 220  29958456812321814474          0 11.66% 12.38% 12.26%   0 Spanning Tree    

  82    929376871711373890         54  8.62%  5.77%  5.72%   0 HLFM address lea 

   9  19556589673689911047          0  6.22%  4.12%  3.80%   0 ARP Input        

 167  2329339277  67374109      34573  3.03%  3.25%  3.29%   0 PI MATM Aging Pr 

 145   4730481931438578737        328  0.95%  1.10%  1.03%   0 Hulc LED Process 

 106   211001781 101058207       2087  0.31%  0.25%  0.26%   0 hpm counter proc 

  83    76489137  67377055       1135  0.15%  0.08%  0.09%   0 HLFM aging proce 

  44    29076540  19101433       1522  0.15%  0.09%  0.10%   0 HC Counter Timer 

 155   109184164  56068206       1947  0.15%  0.12%  0.12%   0 HRPC qos request 

  40    48637882 914493775         53  0.15%  0.12%  0.07%   0 Net Input        

 154    55412319  14098946       3930  0.15%  0.10%  0.10%   0 HQM Stack Proces 

  41    93925763  15921101       5899  0.15%  0.15%  0.15%   0 Compute load avg 

  13           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager   

  14         847        25      33880  0.00%  0.00%  0.00%   0 Entity MIB API   

  12           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit 

  16       16627   1177216         14  0.00%  0.00%  0.00%   0 IPC Dynamic Cach 

  17           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager 

  15           0         1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage 

  11           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT 

  20     2218318  67377658         32  0.00%  0.00%  0.00%   0 IPC Deferred Por 

  21      502497   9400340         53  0.00%  0.00%  0.00%   0 IPC Seat Manager 

  22           0         1          0  0.00%  0.00%  0.00%   0 IPC Session Serv 

  23         109         4      27250  0.00%  0.00%  0.00%   0 PrstVbl          

  24     3051858  67377680         45  0.00%  0.00%  0.00%   0 Dynamic ARP Insp 

  18     2311613  67377650         34  0.00%  0.00%  0.00%   0 IPC Periodic Tim 

  19           0         6          0  0.00%  0.00%  0.00%   0 IPC Managed Time 

  10           0         1          0  0.00%  0.00%  0.00%   0 CEF MIB API      

  28     2318324  67377395         34  0.00%  0.00%  0.00%   0 GraphIt          

  29           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You realize your total CPU includes process CPU and interrupt CPU?  In your case, your last 5 sec process CPU is 36% (the 65% - 29%), and that looks to reflect about the sum of the CPU used by the listed processes.

[Nicolas Maton]

Thank you for this.. but this doesn't explain why my CPU goes to 60+ cpu since yesterday. 

In the sorted list there's also no culprit to be found .. I need to get this down again.. Any idea? 

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The major "culprit" is your 36% interrupt CPU.

Identifying the cause of high interrupt CPU can be difficult, but, on switches, it usually concerns packet forwarding being done by the CPU instead of the 3750's ASICs.  It might be something like using a non-optimal SDM template and you've exceeded one of your TCAM limits. It can also be caused, I believe, by certain interactions between some configuration statements and packet contents (i.e. a new flow of traffic might cause it).  (What changed yesterday?)

Again, some of these can be difficult to troubleshoot.  I would suggest searching Cisco's main site for troubleshooting CPU issues on 3750s.

PS:

I doubt it will help, by you might also consider upgrading to the last variant of your IOS, i.e. 55SE9; considered by many the most stable version of the 3750 IOS.

You might also consider, if your CPU average stays under 75%, and your traffic forwarding performance isn't being particularly harmed, live with it.

[Nicolas Maton]

Thank you for the info. The problem is that we didn't change anything in the topology yesterday. 

Before yesterday evening we had an avg of 50% cpu. This changed overnight with no specific increase in traffic or broadcasts..

I'll go with the firmware upgrade and see where it takes me!