Catalyst 3560-X IOS 15.0(2) IP Base

kerstin-534 · ‎11-30-2012

Hi,

we have observed increase of CPU utilization of 50% after upgrade from 15.0(1)SE3 to 15.0(2).

The SDM template is "desktop default" template.

The increase of CPU utilization in Switches with LAN Base after upgrade is about 10%.

Has somebody an idea to find the cause for the 50% increase ?

This is command is the output of 15.0(2). The calculated sum for 5min process-utilization is 21% plus

1% interrupt is 22 %. Where is the rest ?

switch#sh proc cpu sorted 5min | exc 0.00

CPU utilization for five seconds: 30%/1%; one minute: 29%; five minutes: 28%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

169 135084171 33273787 4059 9.92% 8.94% 8.73% 0 Hulc LED Process

83 80075989 13546109 5911 5.60% 5.39% 5.38% 0 RedEarth Tx Mana

82 35961521 22044350 1631 2.40% 2.58% 2.57% 0 RedEarth I2C dri

4 23049116 1025093 22485 2.87% 1.16% 1.52% 0 Check heaps

126 12684154 1334361 9505 0.79% 0.91% 0.89% 0 hpm counter proc

233 14289030 36476685 391 0.95% 0.90% 0.88% 0 Spanning Tree

181 4784768 265620 18013 0.31% 0.35% 0.32% 0 HQM Stack Proces

401 3112704 4956783 627 0.15% 0.21% 0.21% 0 SNMP ENGINE

54 1528521 1334812 1145 0.31% 0.13% 0.11% 0 Per-Second Jobs

182 1521435 531267 2863 0.15% 0.11% 0.10% 0 HRPC qos request

302 1463582 2928584 499 0.15% 0.10% 0.10% 0 Marvell wk-a Pow

249 1373699 1322338 1038 0.15% 0.08% 0.10% 0 PI MATM Aging Pr

107 1522052 37139967 40 0.31% 0.06% 0.09% 0 HLFM address ret

400 886100 4956777 178 0.15% 0.04% 0.04% 0 PDU DISPATCHER

regards,

Herbert

Arumugam Muthaiah · ‎11-30-2012

Hi Herbert,

When you check the cpu utilization, it will show the three different values like 5 secs, 1 min, 5 mins average CPU utilization. These are the values which combine process which utilize the CPU respectively.

In our case, The first number, 30%, tells how busy the CPU was in the past 5 seconds. This number is the total CPU utilization for all the active system processes, including the percentage of time at the interrupt level.

The second number, 1%, shows the percentage of time at the interrupt level in the past 5 seconds. The interrupt percentage is the CPU time spent receiving packets from the switch hardware. The percentage of time at interrupt level is always less than or equal to the total CPU utilization

I could see that the process Hulc LED Process utilized more CPU percentage. There is some known Bug matching this issue

CSCtn42790 - 3560X/3750X: Elevated CPU usage due to Hulc LED process

Symptom:

Hulc LED Process uses 15-30% CPU on Catalyst 3560X/3750X platforms.

Conditions:

The is seen in 12.2(53)SE releases or later.

Workaround:

This is an expected behavior and there is no workaround.

Further Problem Description:

The "Hulc LED" process does following tasks:

Check Link status on every port
If the switch supports POE, it checks to see if there is a Power Device (PD) detected
Check the status of the transceiver
Update Fan status
Set Main LED and ports LEDs
Update both Power Supplies and RPS
Check on system temperature status

Please observe the average CPU utilization and share the result

# show proc cpu history

# show platform port-asic stats drop

# show controllers cpu-interface

Refer:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

kerstin-534 · ‎12-03-2012

Hi Aru,

I have uploaded the 3 show commands.

Back to my question:

Where is the difference between the sum of all processes and the output of

"show proc cpu" ?

The interrupt level is constant with 1%. (I have seen some trouble if there is a lot of

zeroconf service).

In previous releases it was sufficient to disable all unused interfaces to decrease the cpu utilization.

This is useless in this case.

regards,

Herbert

Arumugam Muthaiah · ‎12-03-2012

Hi Herbert,

Sorry for the delay. i verified the capture and understood that your average CPU utilization is 30%

455555544554455554544454455454455454455444444554555454454545545445454555

910600377036914038378916741839901908701899999109402718829082172970918300

100

90

80

70

60 *

50 **********************************************************************

40 **********************************************************************

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

I tried to verify the same calculation in another switch and i tried to add the 5 mins utilization and found that there is small mismatch with total CPU percentage still.
i hope the switch is pulling up avergage data vs instant process consumption

Switch#sh proc cpu sort 5min
CPU utilization for five seconds: 3%/2%; one minute: 2%; five minutes: 2%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
24     6243252 75514754         82 0.87% 0.33% 0.35%   0 IPC Seat Manager
   9     4651464    305202      15240 0.00% 0.29% 0.27%   0 Check heaps
258     1064396    302344       3520 0.00% 0.09% 0.09%   0 CDP Protocol
285        2980 208738526          0 0.07% 0.09% 0.07%   0 Ethernet Msec Ti
51       13516   1648917          8 0.00% 0.05% 0.07%   0 Per-Second Jobs
342      525216   2418370        217 0.15% 0.05% 0.05%   0 CEF: IPv4 proces
462        4112 102767277          0 0.07% 0.05% 0.05%   0 HSRP Common
52      988728     29231      33824 0.00% 0.03% 0.05%   0 Per-minute Jobs
363      139800    824337        169 0.00% 0.03% 0.02%   0 HIDDEN VLAN Proc
465      467156   2800171        166 0.00% 0.02% 0.01%   0 Port manager per

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

Leo Laohoo · ‎11-30-2012

we have observed increase of CPU utilization of 50% after upgrade from 15.0(1)SE3 to 15.0(2).

If you have 3560/G/E/X and/or 3750/G/E/X, then stay away from using IOS version 15.0(1)SE (and later) and 15.0(2)SE.

Instead, look into 12.2(55)SE6.

kerstin-534 · ‎12-03-2012

Hi,

please can you comment your findings why not use 15.0(x).

thank you,

Herbert

Leo Laohoo · ‎12-03-2012

please can you comment your findings why not use 15.0(x).

Sure, to sum it all up: If you enjoy watching your fleet of 3560/G/E/X and/or 3750/G/E/X continuously having CPU spikes and crashing (as a result), then 15.0(1)SE and 15.0(2)SE is your bet.

Unless there are specific functions you need with the IOS versions you need, then 12.2(55)SE6 is one of THE most stable versions of IOS.

All of my 3560/G/E/X and/or 3750/G/E/X are running this version. Search through the forums of users having issues with their Catalyst switches and running these two versions. You'll find a lot of them.

kerstin-534 · ‎12-04-2012

From my side:

there is a feature that starts with 15.0(2) - disable physical sleep mode.

There are people that are happy because some Broadcom NICs did not work with C3750-X. (they got no link)

IPv6 First Hop Security causes c3750-X to crash in a round robin manner in 15.0(2).

Leo Laohoo · ‎12-04-2012

there is a feature that starts with 15.0(2) - disable physical sleep mode.

There are people that are happy because some Broadcom NICs did not work with C3750-X. (they got no link)

IPv6 First Hop Security causes c3750-X to crash in a round robin manner in 15.0(2).

That's one reason and I won't talk you out of using 15.0(2)SE.

May I be able to provide you with an advice?

I like to call IOS, versons without any numbers at the end, as version "0". It's like Windows 7 RTM. What I'm saying is this: Keep a sharp eye on any further/later releases to 15.0(2)SE, or versions with numbers at the end like 15.0(2)SE1 or 15.0(2)SE2, etc. This means that there were bug fixes. I am not sure if you are aware but Cisco there are two "types" of bugs. One is public-knowledge bugs and the other is internal bugs.

kerstin-534 · ‎12-05-2012

Hi,

I always thought this is constructive forum. Things as explaining version numbers is something of a really odd thing or comparisons with Windows 7.

best regards,

Herbert

Leo Laohoo · ‎12-05-2012

I always thought this is constructive forum. Things as explaining version numbers is something of a really odd thing or comparisons with Windows 7.

Sorry, wasn't meant to be offensive or insulting.

But the relevance is there. When MS released the OS to PC/Laptop manufacturer, it's called RTM or "Release To Mananufacturer". You can call this version "0".

Same with IOS versions. IOS versions without a number at the very end is simply known as version "0".

lcd_shouldit · ‎12-10-2012

Hi Leo，

how about 2960，still 122-55.SE6?

Sent from Cisco Technical Support iPhone App

Leo Laohoo · ‎12-11-2012

how about 2960，still 122-55.SE6?

2960/G/S? No problem with 15.0(2)SE.

Just don't do any TDR. You'll get a heart attack!

lcd_shouldit · ‎12-13-2012

I have change 3750X to

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 54 WS-C3750X-48P 12.2(55)SE6 C3750E-UNIVERSALK9-M

But，the cpu usage

CPU utilization for five seconds: 29%/0%; one minute: 29%; five minutes: 29%

a little high , is it nornal.......?

Arumugam Muthaiah · ‎12-13-2012

Hi,

Yes, This is an expected behavior, in this platform.

The Catalyst 3750-X switches have a CPU utilization level that is higher than the previous models of the Catalyst 3750 switches. This is normal behavior. One can expect that the overall CPU utilization will be between 25 – 35%.

Since almost all of the packet forwarding is done in hardware and not by the CPU a CPU utilization of 25 – 35% should not be cause for concern.

Please check the show proc cpu sort | ex 0.00% and share the detail.

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***