Strange Issue on multiple 2960s Stacks

[neilobrien]

Hi,

I have 2x 4507's in the core with a L2 either channel between them.  I have multiple 2960s stacks, each a seperate vlan/subnet connected back to the core over dual fibre links (one to each core).  I have increased the STP cost on one etherchannel port for the stack vlans to allow me utilise both fibre links and I'm using GLBP on the core as a FHRP.

On each stack, POE running and IP phones and PC's.  I'm getting average 80% processor utilisation while idle (at night).  Stange thing is that when I look at the processes, I'm not seeing 80% worth of processes.  Also, as soon as I log onto the switch via telnet, the processor utilisation drops.  I'm a bit stumped??

Just wondering if anyone has any pointers.  I am running port security, bpduguard and qos on each port but my issue is that I can't see what process keeping the processor up?  I understand that HULC LED PROCESS is notorious for high proc utilisation but taking all into account, it doesn't add up to 80%.

I'd appreciate any comments,

Thanks,

Neil

See output below:

STACK#sh proc cpu sort 5min 
CPU utilization for five seconds: 45%/11%; one minute: 67%; five minutes: 72%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 137   226554617  13639394      16610 25.38% 25.09% 25.61%   0 Hulc LED Process 
 106     7931883   1571318       5047  0.79%  0.83%  0.84%   0 hpm counter proc 
   4     8144723    440392      18494  0.00%  0.70%  0.83%   0 Check heaps      
 191     3960953   2467043       1605  0.69%  0.57%  0.56%   0 Spanning Tree    
 146     2226638    160857      13842  0.29%  0.24%  0.24%   0 HQM Stack Proces 
 147     1433304    893190       1604  0.19%  0.17%  0.16%   0 HRPC qos request 
 176     1433124  15808031         90  0.49%  0.20%  0.16%   0 IP Input         
   8     1918180     13333     143867  0.00%  0.11%  0.15%   0 Licensing Auto U 
 166     1184376   1582817        748  0.00%  0.12%  0.12%   0 CDP Protocol     
  37     1060429    796877       1330  0.09%  0.10%  0.11%   0 Per-Second Jobs  
  75      719698   3003127        239  0.00%  0.13%  0.11%   0 hrpc <- response 
  67      929675   7545394        123  0.19%  0.11%  0.11%   0 RedEarth Tx Mana 
  68     1181846  27246067         43  0.00%  0.12%  0.10%   0 RedEarth Rx Mana 
 102      949546   8224391        115  0.00%  0.08%  0.09%   0 hpm main process 
 107      527361   2781387        189  0.00%  0.05%  0.05%   0 HRPC pm-counters 
  33      599657    155171       3864  0.00%  0.05%  0.05%   0 Net Background   
 206      366343    855322        428  0.09%  0.03%  0.04%   0 UDLD             
 249      378371   1761360        214  0.00%  0.05%  0.04%   0 Marvell wk-a Pow 
  43      409690    159858       2562  0.09%  0.04%  0.04%   0 Compute load avg 
 248      272902   1432302        190  0.00%  0.03%  0.03%   0 Inline Power     
  61      390718  26993171         14  0.19%  0.05%  0.02%   0 Draught link sta 
 305          80        63       1269  0.00%  0.07%  0.01%   1 Virtual Exec     
  47      137816      5379      25621  0.00%  0.03%  0.01%   0 SFF8472          
 299      253895  20669585         12  0.00%  0.01%  0.00%   0 RADIUS           
 231      160726   1590638        101  0.00%  0.01%  0.00%   0 DHCPD Receive    
  52      324584  13638872         23  0.09%  0.03%  0.00%   0 DownWhenLooped   

[Leo Laohoo]

What IOS are you running?  Are you doing routing on the 2960S?

[venkata dandu · Accepted solution]

Hi;

This is a cosmetic Bug (CSCtgth24278) on versions 12.2(53)SE and 12.2(53)SE2 and cisco recently announed that it is been fixed in version 12.2(55)SE and 12.2(58)SE1.

BugID: CSCth24278

What Changed: Fixed in version, Bug Details

Title: High CPU when no Console/VTY activity

Technology: nav

Version: 12.2(53)SE2

Fixed-in: 12.2(55)SE3

Severity: 5

State: Verified

Bug Details:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth24278

Hope to help.Please rate if this is of any help.

Regards;

Rams

[neilobrien]

Hi Guys,

Sorry for the delay, I wasn't getting notifications of replies.

However, I did do some more digging and refining my search on here and realised there has been a lot of discussion on this particular issue.  https://supportforums.cisco.com/message/3170210

I'm running 12.2(55)SE2.  I haven't noticed any network performance and if it's just cosmetic as some say, then that's ok.  However I will go down the upgrade route if I do notice any issues.  The network is still quite new so is not being fully utilised yet so I'll keep a close eye.

THanks for the replies.

Neil

[Leo Laohoo]

The network is still quite new so is not being fully utilised yet so I'll keep a close eye.

On the contrary, I'd upgrade as soon as I can before anyone starts to notice it.

[neilobrien]

yeah - i'm kind of torn,  it's not fully utilised but is in production none-the-less.  So any outage is a big deal at this stage.  I guess I can propose it and see what reaction I get

[Leo Laohoo]

Please don't take any offense to what I am about to say because it's not meant for anyone (including yourself).

The organization I currently work for has a very "open" attitude about upgrades of IOS.  When I found out later that the 12.2(58)SE IOS has a number of critical bugs I let my supervisors know about it.  "And what are you going to do about it," they demanded.  When the 12.2(58)SE1 came out, I created a change window and rolled out the IOS.   I personally don't want to be confronted one day and asked the two questions, "How long have you known about this?" and "And why didn't you do anything about it?".

I don't know the size of your network.  Ours is around 90 sites, >600 switches and between 6K to 9K users.  No laughing matter.  I rolled out the IOS (in less than a week) without using any forms of automation.

[neilobrien]

no offence taken, I take your point.

thanks....