experiencing high CPU on both the core switch-6509 .

[sameermunj]

I am experiencing high CPU on both the core switch and that’s causing HSRP and OSPF to reconverge frequently. Also  if I am  trying to reach the HSRP IP’s on the core from the LAN ICMP reply vary from 1 Ms to 4000Ms every 40 secs( ideally it should be <1Ms) but interesting thing is inter VLAN ICMP replies are <1 Ms, same thing is experienced when connecting laptop directly to core switch.

attached is my show tech .

[Giuseppe Larosa]

Hello Sameer,

check STP activity on affected Vlans it is likely that there troubles or even temporary loops on them

use

show spanning-tree vlan 5

show spanning-tre vlan 10

show spanning-tree vlan 201

see if with the detail you can see last time STP calculation happened it is likely that STP activity is huge on affected Vlans.

use

show proc cpu sorted

to see what processes are using most resources

it is also possible that an access layer switch where the affected vlans are on the uplink is not working correctly causing issues

the high cpu usage can be a result of these L2 instabilities.

Hope to help

Giuseppe

[sameermunj]

Hi  Giuseppe,

Thanks for the reply, I have attached the o/p of the spann-tree for vlan.. seems to be fine, aslo sh-tech of one of my L2 device.

[gephelps]

You are running Modular IOS and 12.2(18)SXF7 is impacted by CSCsh33937. This bug was marked as a duplicate of CSCsh06605 which fixes several Modular IOS bugs. This was first put into 12.2(33)SXH.

In the show tech you sent we can see for the RP most of the CPU is being used by the kernel:

CPU utilization for five seconds: 100%; one minute: 68%; five minutes: 38%
PID/TID   5Sec    1Min     5Min Process             Prio  STATE        CPU
1       101.2%   57.7%    27.9% kernel                                30d11h

[Giuseppe Larosa]

Hello Samir,

George is right I hadn't seen that it is modular IOS

System image file is "sup-bootdisk:s72033-ipservices_wan-vz.122-18.SXF7.bin"

wan_vz means modular

Hope to help

Giuseppe

[sameermunj]

Hi  giuslar/ George

Thanks for the reply..so just to confirm this abnormal behaviour is observed because of the bug in the current ios...now as we are aware presently the switch is running modular ios i have 2 questions in mind..

1--do we have to upgrade the switch with modular ios only or can we go ahead with regulat ios ..( i have seen couple of commands output not available in modular ios like show proc cpu sorted..)

2--if we have to upgarde with Modular/Regular ios which would be best option

Presently 1 switching is running Version 12.2(18)SXF8 while the other is running Version 12.2(18)SXF7

current hw config

1   48  CEF720 48 port 10/100/1000mb Ethernet                 WS-X6748-GE-TX    
  2    0  2 port adapter Enhanced FlexWAN                       WS-X6582-2PA      
  3   48  48-port 10/100 mb RJ45                                     WS-X6148-45AF     
  5    2  Supervisor Engine 720 (Active)                             WS-SUP720-3B 

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-IPSERVICES_WAN-VM), Version 12.2(18)SXF7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 23-Nov-06 07:41 by kellythw
Image text-base: 0x01020150, data-base: 0x01021000

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
BOOTLDR:
Core A uptime is 2 years, 11 weeks, 2 days, 23 hours, 52 minutes
Time since Core A switched to active is 2 years, 11 weeks, 2 days, 23 hours, 52 minutes
System returned to ROM by  power cycle (SP by power on)
System restarted at 13:15:37 IST Sun Aug 3 2008
System image file is "sup-bootdisk:s72033-ipservices_wan-vz.122-18.SXF7.bin"

cisco WS-C6509-E (R7000) processor (revision 1.2) with 491520K/32768K bytes of memory.
Processor board ID SMG1044N3A4
SR71000 CPU at 600Mhz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
Bridging software.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
TN3270 Emulation software.
1 Enhanced FlexWAN controller (1 ATM).
45 Virtual Ethernet/IEEE 802.3 interfaces
48 FastEthernet/IEEE 802.3 interfaces
50 Gigabit Ethernet/IEEE 802.3 interfaces
1 ATM network interface
1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

Patching is not available since the system is not running from an installed image. To install please use the "install file" command

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-IPSERVICES_WAN-VM), Version 12.2(18)SXF8, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Sat 03-Mar-07 01:58 by tinhuang
Image text-base: 0x01020150, data-base: 0x01021000

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
BOOTLDR

Core B uptime is 7 weeks, 2 days, 19 hours, 54 minutes
Time since Core B switched to active is 7 weeks, 2 days, 19 hours, 54 minutes
System returned to ROM by  power cycle (SP by power on)
System restarted at 17:24:14 IST Tue Aug 31 2010
System image file is "sup-bootdisk:s72033-ipservices_wan-vz.122-18.SXF8.bin"

cisco WS-C6509-E (R7000) processor (revision 1.3) with 491520K/32768K bytes of memory.
Processor board ID SMG1115N6D7
SR71000 CPU at 600Mhz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
Bridging software.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
TN3270 Emulation software.
1 Enhanced FlexWAN controller (1 ATM).
46 Virtual Ethernet/IEEE 802.3 interfaces
48 FastEthernet/IEEE 802.3 interfaces
50 Gigabit Ethernet/IEEE 802.3 interfaces
1 ATM network interface
1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

Patching is not available since the system is not running from an installed image. To install please use the "install file" command

[sameermunj]

Hi

Please revert on the same so that i can do the upgrade and test the operation..

[Naveen Kumar Rajaharia]

Pls check -

1) utilization on interfaces

2) Timers for HSRP

3) Broadcast/Multicast Traffic

if above doesn't help then use "Debug Platform CPU buffer" to zero down the cause

[sameermunj]

Hi

Thanks for the reply..

I think as mentioned in the previous thread it is a bug in the modular ios which is CSCsh33937 observed in the current sw train 12.2(18)SXF7.Same has been resolved in 12.2(33 SXH)..now my concern is can i directly upgarde to this ios or any intermediate upgarde is required....can i upgarde the same with ip servcies sw or i need to upgarde with ip servcies (modular) which is currently running...which is the recommended sw in 12.2(33 sxh)..I have traied finding teh same with software advisor by adding all the existing hardware the result i received is 12.2(33)SXH7
s72033-ipservices_wan-vz.12.2-33.SXH7 ...can anyone please confirm the same...cards used are

  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX    
  2    0  2 port adapter Enhanced FlexWAN        WS-X6582-2PA      
  3   48  48-port 10/100 mb RJ45                 WS-X6148-45AF    
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B     

PS   Type              
---- ------------------ ------- ------ ------ ------ -----
1    WS-CAC-3000W     
2    WS-CAC-3000W

Fan  Type              
---- ------------------ ------- ------ -----
1    WS-C6509-E-FAN

[Giuseppe Larosa]

Hello Sameer,

I would go for 12.2(33)SXI non modular IOS

the upgrade should be possible with the usual procedure.

Hope to help

Giuseppe

[sameermunj]

Hi Giuslar

Thanks for the reply

Using software Adviosr i have checked the IOS version for the same looking at the current hardware.

details are

12.2(33)SXH6
s72033-ipservices_wan-mz.12.2-33.SXH6

12.2(33)SXI4
s72033-ipservices_wan-mz.12.2-33.SXI4

As you are suggesting SXI train and non modular ios i will go ahead with s72033-ipservices_wan-mz.12.2-33.SXI4 ....

just to confirm can i upgarde directly from 12.2.18 to directly 12.2.33  or any intermediate upgrade required...

you have already confirmed that even though current ios is modular we can go ahed with regular ip services software which is mentioned above..

please confirmabout the direct upgarde...

[Puneet Bansal]

Hi

You can directly upgrade.

Just check the memory requirements.

Also as per the Show tech (Show Logging) you HSRP is not working properly - fluctuating)

Just upgrade the IOS and then check.

Regards

Puneet Bansal

[sameermunj]

Thanks for teh reply..

I think the BUG is resolved in 12.2.33 SXH release but just to verify i think SXI train is latest than SXH train so it must be covering any open caveats in SXH series so i am proceeding with SXI series..

The Software advisor suggested the sw  12.2(33)SXI4 for the current hw so we are proceeding with the same...

Please confirm if anything is missing as we will do the upgarde in live enviornment..we will upgarde current ip services(modular ios) with 12.2.33 IP services IOS (not modular..)

[Giuseppe Larosa]

Hello Sameer,

point of attention:

the old 6148 linecard may be not supported correctly in 12.2SXI there has been a recent thread where two colleagues have reported problems with ports of 6148 linecards with 12.2SXI

Have you provided to software advisor the complete list of your linecards?

Edit:

you are fine 6148-AF are listed as supported under the 10/100 linecards category

WS-X6148-45AF

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/hardware.html#wp4631272

Hope to help

Giuseppe

[sameermunj]

Hi Giuseppe

Again i have tried with the software advisor the supported IOS for the line cards in my chassis and i am getting 12.2(33)SXI4
s72033-ipservices_wan-mz.12.2-33.SXI4 software..it seems   WS-X6148-45AF  line card is supported.

i have given all below details in advisor and got teh result above

Mod Ports Card Type                              Model            

--- ----- -------------------------------------- ------------------ -----------
  1   48  CEF720 48 port 10/100/1000mb Ethernet       WS-X6748-GE-TX    
  2    0  2 port adapter Enhanced FlexWAN                  WS-X6582-2PA     
  3   48  48-port 10/100 mb RJ45                                WS-X6148-45AF    
  5    2  Supervisor Engine 720 (Active)                        WS-SUP720-3B     

S   Type               Watts  
---- ------------------ ------- ------ ------ ------ -----
1    WS-CAC-3000W      
2    WS-CAC-3000W      
                       
Fan  Type             

---- ------------------ ------- ------ -----
1    WS-C6509-E-FAN