Cisco 3750X 24SE-12.2(58)SE2 CPU Utilization High

[faamin011]

We have just purchased Cisco 3750X 24 S switches (QTY 2), last week we put them in prodcution in simple Layer 2 mode. No Core, just act as Access switch in order to only check the performace and availibility of switches.

But strange thing I have noticed the highest cpu utilization, cpu sits on 13 to 14% and from historical logs it hit above 60%. I am confused what is happening. Remember to terminate copper connection we have inserted Copper SFP Modules on both switches.

Topology like as

Both switches connected on Layer 2 mode via trunk, in LAN environment we have here 7 3Com 2824 Management switches which are terminated on Copper SFP modules at Cisco 3750X 24 S. All 3Comm Switches uplink are terminated on Cisco 3750 Switch 1, another switch is idle. Whole setup is running on VLAN1 .

Logs are attached for your reference,

Please advice why CPU sits on high peak..

Also refer no port in TCP and UDP packets in show ip traffic. any abnormal condition you predict..

Basically, In near days we plan to make these switches as Core layer and run intervlan routing, HSRP/VRRP and OSPF. But in this cpu usage how we can go ahead until this resolved..

[ROBERTO TACCON]

Hello Fahad,

you can contact directly the Cisco TAC and ask for an update software as all the Cisco 3750x switch are coverd by

Cisco® Enhanced Limited Lifetime Warranty and lifetime software update policy

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps10745/product_bulletin_c25-607000.html

Cisco Enhanced Limited Lifetime Hardware Warranty:

The Cisco Catalyst 3750-X, Catalyst 3560-X, and  Catalyst 2960-S Series Switches come with an enhanced limited lifetime  hardware warranty (E-LLW) that includes next-business-day delivery of  replacement hardware where available and 90 days of 8 x 5 Cisco  Technical Assistance Center (TAC) support.

Cisco reserves the right to refund the purchase price as its exclusive warranty remedy.

Software Policy for Cisco Catalyst 3750-X, Catalyst 3560-X, and Catalyst 2960-S Series Switches:

Customers  with Cisco Catalyst LAN Lite, LAN Base, and IP Base software feature  sets will be provided with maintenance updates and bug fixes designed to  maintain the compliance of the software with published specifications,  release notes, and industry-standards compliance as long as the original  end user continues to own or use the product or up to 1 year from the  end-of-sale date for this product, whichever occurs earlier.

[faamin011]

thanks, please make correction does this bug exists with switch hardware or software, as per given link cisco is not entitiled IP Services in maintenance updates other than we have shared or smartnet. Advice please in this case...

Or do we need to replace hardware, will this replacement cover in E-LLW.

[faamin011]

Please respond so we may move forward, I have seen the bug which exists as michelpe said but no where mention the said bug is assosiated with current IOS.

Waiting for kind response in this regard

[ROBERTO TACCON]

http://tools.cisco.com/Support/BugToolKit/search/knownAffectedVersions.do?method=fetchKnownAffectedVersions&bugId=CSCsh62565

[faamin011]

Go through bug toolkit, total 455 iOS are effected but 12.2 (58) se2 is not a

Part of that list. Pls elobrate

[faamin011]

Any feedback, advice please

[michelpe]

There are 2 things here. 1 ) the high cpu when regenerating the keys when SSH is enabled. This is a spike you will

see in the utilization that shouldnt last long. This is not changed regardless of if that bug is or is not fixed. The

regeneration is per RFC 4253 and can only be prevented by disabling SSH server

2) in older IOS releases that regenaration of the keys caused an issue with some protocols. This is adressed with the bug.

So in other words, the short spikes can be ignored as those are due to the keys being regenerated as per the RFC.

To disable SSH server

 - removing RSA keys ("crypto key zeroize rsa").
   or 
 - pointing SSH server to non-existing RSA keypair ("ip ssh rsa keypair-name FAKE").  This method will allow to disable SSH server but keep RSA keys for other purposes (IPsec, SSL, etc).

[faamin011]

thanks will manage rsa key generation by disabling or can be ignored as the spike stayed b/w 0.5 to 5 sec. simply say ignorable, but i have concern about the spike stayed on regular interval and kept cpu above 30% and some times 45%. Like mentioned below

Switch2#sh processes cpu sorted | ex 0.0

CPU utilization for five seconds: 30%/0%; one minute: 23%; five minutes: 18%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

125   123700870   108400355       1141  9.84%  8.45%  7.76%   0 hpm main process

347        7693         790       9737  8.55%  5.67%  2.00%   1 Virtual Exec  

169     4751330    40115682        118  2.09%  1.27%  0.64%   0 Hulc LED Process

  10     2881365       26749     107718  1.93%  0.24%  0.14%   0 Licensing Auto U

  85     6148727     5915324       1039  0.96%  0.42%  0.36%   0 RedEarth Tx Mana

129    14330885     1607810       8913  0.80%  0.82%  0.83%   0 hpm counter proc

  84     4106263     7925621        518  0.16%  0.37%  0.33%   0 RedEarth I2C dri

181     3225396      320762      10055  0.16%  0.21%  0.17%   0 HQM Stack Proces

On other end, we have around 400LAN users + 3 WAN + 2ISP environment and all will be pointed this switch as default gateway. Once this comes under production then wouldn't be any issue of slow response.

[Vineesh Viswanath]

fahad , have you managed to find a fix for this ?

[Leo Laohoo]

Hi Vineesh,

Unless you have a very important requirement to run this very specific version, i.  e.  12.2(58)SE and later, try 12.2(55)SE6 or you could try 15.0(2)SE. 

[michelpe]

The baseline CPU utilization on the 3750x is expected to be a bit higher then older 3750 platforms.

You do have to take into account with that though that the CPU is not involved in traffic forwarding

itself. So if you increase the traffic load through the switch that wouldnt impact the CPU.

Neither will a high CPU impact traffic flow through the switch.

[faamin011]

Thanks for your response, you are right there will be no issue while on increase load but as corporate organization we have adopt some step to resolve this.

our vendor has recomended following IOS, please advice and comment. So I will go to switch over 

c3750e-universalk9npe-tar.150-2.SE.tar

So far things are running well, that is why i don't want to come under trouble to upgrade blindly.

[faamin011]

Pls comment who have knowledge about this

[Leo Laohoo]

We've rolled back the IOS for your fleet of 3750E/X and 3560E/X from 15.0(1) and 15.0(2) to the more stable 12.2(55)SE6.

[daleynicholson]

We also expeirenced a similiar issue to yours Fahad.  After upgrading to the 15.0(2)SE the SSH issue has stabilized to about 40% utilization, which is normal for the device.